Senior Data Engineer

Description

• Driving the adoption of CI/CD and DevOps practices for data engineering workflows.
• Champion data quality, lineage, and observability through automated testing, monitoring, and documentation.
• Building and managing orchestrations.
• Stay ahead of emerging technologies and Databricks platform updates, evaluating their relevance and impact.
• Collaborate with cross-functional teams to define data requirements, governance standards, and integration strategies.
• Design and implement scalable, high-performance data pipelines.
• Work with the lead cloud architect on the design of data lakehouse solutions leveraging Delta Lake and Unity Catalog.

Requirements

• Understanding of OLTP and OLAP and what scenarios to deploy them in.
• Experience with Delta Lake and Unity Catalog in production environments.
• Experience of working on machine learning workflows and integration with data pipelines.
• Excellent oral and written communication, both within the team and with our stakeholders.
• Databricks Certified Data Engineer Professional or equivalent certification.
• Experience with CI/CD tools and version control systems (e.g., Git, GitHub Actions, Azure DevOps[RP4], Databricks Asset Bundles).
• Strong knowledge of data modelling, warehousing, and governance principles.
• Knowledge of data privacy and compliance standards (e.g., GDPR, HIPAA).
• Experience leading data engineering projects with distributed teams, ideally in a cross functional environment.
• Strong proficiency in Python and SQL. Experience of working with Scala would be beneficial.
• Deep understanding of distributed data processing, data lakehouse architecture, and cloud-native data platforms.
• Experience with real-time data processing, both batch and streaming.
• Proven ability to design and optimize large-scale ETL/ELT pipelines.
• Understanding of incremental processing patterns.
• 5+ years of experience in data engineering, with at least 2 years working extensively with Databricks and orchestrated pipelines” such as DBT, DLT, or workflows using jobs.

Benefits

• Mentoring and guidance of junior data engineers. Using you passion for data engineering to foster a culture of technical excellence and continuous learning.

Training + Development

Interview process

Visa Sponsorship

Security clearance