Description

Design and maintain scalable data pipelines and ETL/ELT processes on AWS.

This role is hybrid.

Responsibilities

• Develop batch and real-time data processing solutions using Python and PySpark.
• Build and optimize Data Lakes and Data Warehouse architectures.
• Perform data quality validation, monitoring, and troubleshooting.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Support production deployments and resolve critical incidents.

Required Skills

• 6+ years of overall IT experience with 4+ years in Python and PySpark.
• Hands-on expertise with AWS services: S3, EMR, Glue, Lambda, IAM, Athena, Redshift.
• Strong SQL and Data Warehousing concepts.
• Experience building ETL/ELT pipelines and Data Lake implementations.
• Git/GitHub version control.
• Performance tuning of Spark applications.
• Understanding of Agile methodologies.

Preferred Skills

• Experience with Databricks, Apache Airflow, dbt, Snowflake, or Kafka.
• Familiarity with CI/CD pipelines, Docker, Kubernetes, and Terraform.

Key Skills

Education

Bachelor's degree