Description

You will build and optimize large-scale data pipelines using PySpark.

This role is on-site.

Responsibilities

• Design and optimize ETL/ELT pipelines for structured, semi-structured, and unstructured data.
• Ingest and process data from Data Lake, RDBMS, and API sources.
• Implement monitoring, logging, and error-handling mechanisms for pipeline reliability.
• Clean, transform, and deliver business-ready datasets to data teams.

Required Skills

• 10+ years of data engineering experience with strong PySpark expertise.
• Proficiency in Python for data manipulation and pipeline development.
• Deep knowledge of Spark SQL, DataFrames, and RDDs.
• Experience with big data ecosystems including Hadoop or Databricks.
• Familiarity with AWS cloud platforms.
• Competency with relational databases such as PostgreSQL, MySQL, or Oracle.
• Experience with version control using Git.
• Experience with containerization using Docker and Kubernetes.

Key Skills

Education

Bachelor's degree