Description

You will build and optimize end-to-end data pipelines and processing jobs within an AWS environment.

Responsibilities

• Develop and optimize PySpark jobs for complex data transformations and aggregations.
• Design and implement scalable data pipelines and warehouses using AWS services including EC2 and S3.
• Manage SQL database schema design, query optimization, and performance tuning.
• Orchestrate and automate data workflows using Apache Airflow.
• Maintain Databricks clusters and Spark jobs to ensure high data quality and accessibility.

Required Skills

• 5+ years of experience in Data Engineering.
• Extensive expertise with PySpark and Python.
• Strong proficiency in SQL and data warehouse design.
• Hands-on experience with AWS cloud services.
• Experience building and maintaining end-to-end data pipelines.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or Information Technology.
• Ability to troubleshoot PySpark scripts, Databricks notebooks, and Airflow DAGs.

Preferred Skills

• Experience with Snowflake and Apache Airflow.
• Familiarity with CI/CD processes using GitHub.

Key Skills

Education

Bachelor’s or Master’s degree in Computer Science