Data Engineer (No Sponsorship Available)

Overview

Develop and maintain data pipelines on Databricks, integrating Azure services and CI/CD.

Key Responsibilities

• data pipelines
• databricks
• unity catalog
• cluster optimisation
• delta lake
• ci/cd automation

Tasks

-Develop and support data pipelines within our Cloud Data Platform Databricks -Collaborate with cross-functional teams to support data governance using Databricks Unity Catalog -Monitor and optimize Databricks cluster performance, ensuring cost-effective scaling and resource utilization -Design, implement, and maintain Delta Lake tables and pipelines to ensure optimized storage, data reliability, performance, and versioning -Design and implement monitoring solutions, leveraging Databricks monitoring tools, Azure Log Analytics, or custom dashboards for cluster and job performance -Integrate Databricks with other Azure services, such as Azure Data Factory and ADLS -Automate CI/CD pipelines for data workflows using Azure DevOps

Requirements

• python
• sql
• databricks
• snowflake
• azure
• ci/cd

What You Bring

-Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field required -Proficient in Microsoft Office -Experience with Test Drive Development (TDD) -Python application development -Experience with programming languages such as Python, Java, R, or SQL -Strong analytical and problem-solving skills -Experience with Databricks or Snowflake Cloud Data Platforms -Hands-on experience with Databricks or Snowflake Cloud Data Platforms -Knowledge of statistical analysis and data visualization tools, such as PowerBI or Tableau -Database Schema Design and Optimization -Expertise with programming languages such as Python and/or SQL -Linux / Bash / Docker -Experience working in a cloud environment (Azure preferred) with strong understanding of cloud data architecture -Automated CI/CD Pipelines -Experience with workflow orchestration (e.g., Databricks Jobs, or Azure Data Factory pipelines) -Experience with Python Data Science Libraries (PySpark, Pandas, etc.)

People Also Searched For

About Heritage Construction + Materials

-Known for delivering large-scale projects across multiple sectors, from residential to industrial. -Expert in heavy civil, commercial, and residential developments, often working on complex projects. -Has a strong presence in industrial and infrastructure sectors, specializing in material supply and construction. -Engages in notable projects in commercial and residential developments, enhancing local communities. -The company thrives on innovation and craftsmanship, blending traditional expertise with modern technology. -Renowned for handling large, multi-phase projects with intricate logistics and a focus on sustainability.

Sector Specialisms

Visa Sponsorship

-no sponsorship available; requires u.s. work authorization.