Description

Build and manage scalable data pipelines and ETL processes to ingest data from multiple sources into centralized platforms.

This role is on-site.

Responsibilities

• Develop and implement data pipelines for ingesting data from various sources into centralized systems.
• Build and maintain ETL jobs using AWS Glue, Python, and PySpark to process data at scale.
• Create and manage ETL processes using Airflow to move and transform data between systems.
• Optimize and troubleshoot AWS Glue jobs, databases, and monitoring solutions to ensure pipeline reliability.
• Collaborate with architects to design data models that support business requirements.

Required Skills

• Over 15 years of professional experience in data engineering or integration.
• Proficiency in Python, PySpark, and SQL for data manipulation and pipeline development.
• Hands-on experience with AWS Glue, Airflow, DynamoDB, Redshift, S3 buckets, and Event-Grid.
• Experience with event-driven, batch-based, and API-led data integrations.
• Proficiency in CI/CD pipelines including Azure DevOps, AWS pipelines, or GitHub Actions.
• Knowledge of Infrastructure as Code (IaC) tools such as Terraform or AWS CDK.
• Experience with Swagger, JSON, XML, SOAP, and REST-based web services.
• Ability to implement data testing practices within CI/CD workflows.

Preferred Skills

• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree