Developer

Description

• Implement complex transformations, aggregations, and joins ensuring performance, and reliability
• Develop and manage dataflows, and semantic models to support specific analytics related business requirements
• Implement and apply robust data validations, cleansing, and profiling techniques to ensure data accuracy and consistency across datasets
• Create, maintain, and update documentation and internal knowledge repository
• Implement role-based access, data masking, and compliance protocols
• Work collaboratively with analysts and business stakeholders to translate requirements into technical solutions
• Design, build, and optimise scalable data pipelines for batch and streaming workloads

Requirements

• Experience of programming under Microsoft Fabric Platform
• Familiar with the Agile. Well communication.
• Spark Streaming/batch processing
• Experience of programming under Microsoft Azure Cloud Platform
• Spark application performance tuning
• Good English listening and speaking for communicating requirements and development tasks/issues
• Java programming language, OOP knowledge
• Hands-on experience with lake houses, dataflows, pipelines, and semantic models
• Performance tune and optimise jobs and workloads to reduce latency
• Experience with One Lake, Azure Data Lake, and distributed computing environments
• Python / Notebook programming
• Ability to prepare and process datasets for Power BI usage
• Experience of using the tools: Gitlab, Python unit test, CICD pipeline.
• Knowledge of Docker / Kubernetes
• Good skill of troubleshooting
• Database knowledge, including Relational Database and Non-SQL database.
• Familiarity with time-series data, market feeds, transactional records, and risk metrics
• Delta table Optimization
• Familiarity with Git, DevOps pipelines, and automated deployment
• Ability to build ETL workflows
• Understanding of financial regulations such as GDPR, SOX etc.
• Strong communication skills with a collaborative mindset to work with and manage stakeholders
• Have knowledge of Spark Programming Ability to write Spark code for large scale data processing, including RDDs, Data Frames, and Spark SQL

Benefits

• PySpark programming

Training + Development

Interview process

Visa Sponsorship

Security clearance