Description

You will design, build, and maintain scalable data platforms.

Responsibilities

• Develop and maintain batch and streaming data pipelines using PySpark.
• Implement complex business logic using Spark SQL, DataFrame, and RDD APIs.
• Design and optimize ETL/ELT processes for data transformation and quality.
• Utilize workflow orchestration tools to manage data pipeline execution.
• Work with AWS services to manage data lakes and Spark workloads.

Required Skills

• 5+ years of professional experience in data engineering.
• Expert proficiency in PySpark and Spark SQL.
• Strong programming skills in Python, including data structures and algorithms.
• Proven experience designing and maintaining data pipelines.
• Familiarity with ETL/ELT processes and performance tuning.
• Experience with AWS services (S3, Glue/EMR, Lambda).
• Hands-on experience with workflow orchestration tools like Apache Airflow.
• Experience with data transformation frameworks like dbt.

Key Skills

Education

Any Graduate