Principal Digital Solutions Consultant

Description

• Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
• Collaborate with data architects to design and implement data models that support business requirements.
• Create and maintain ETL processes using Airflow, Python and PySpark to move and transform data between different systems.
• Design Operations architecture and conduct performance engineering for large scale data lake in production environment.
• Participate in Client design workshops and provide tradeoffs and recommendations towards building solutions.
• Optimize and troubleshoot AWS Glue jobs for performance and reliability.
• Design and Optimize data models on AWS Cloud using AWS data stores such as Redshift, DynamoDB, RDS, S3 etc.
• Implement monitoring solutions to track data pipeline performance and proactively identify and address issues.
• Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
• Create and maintain optimal data pipeline architecture by designing and implementing data ingestion solutions on AWS using AWS native services.
• Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
• Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.

Requirements

• Familiarity with Infrastructure as Code (IaC) tools like Terraform, AWS CDK and others.
• Proficiency in eventdriven integrations, batchbased and APIled data integrations.
• Proficiency in CICD pipelines such as Azure DevOps, AWS pipelines or Github Actions.

Benefits

Training + Development

Interview process

Visa Sponsorship

Security clearance