Description

You will build and manage data pipelines and ETL processes to move data from various sources into a centralized platform.

Responsibilities

• Develop and implement data pipelines for ingesting data from multiple sources into centralized platforms.
• Build and maintain ETL jobs using AWS Glue, Python, and PySpark to process data at scale.
• Create and manage ETL processes using Airflow to move and transform data between systems.
• Optimize and troubleshoot AWS Glue jobs, databases, and monitoring solutions to ensure pipeline reliability.
• Collaborate with architects to design data models that support business requirements.

Required Skills

• Over 15 years of professional experience.
• Proficiency in Python, PySpark, and SQL for data manipulation and pipeline development.
• Hands-on experience with AWS Glue, Airflow, DynamoDB, Redshift, S3 buckets, and Event-Grid.
• Experience with event-driven, batch-based, and API-led data integrations.
• Proficiency in CI/CD pipelines including Azure DevOps, AWS pipelines, or GitHub Actions.
• Knowledge of Infrastructure as Code (IaC) tools such as Terraform or AWS CDK.
• Experience with Swagger, JSON, XML, SOAP, and REST-based web services.
• Ability to implement data testing practices within CI/CD workflows.
• Bachelor's degree.

Key Skills

Education

Bachelor's degree