Description

You will build and manage data pipelines for large-scale structured, semi-structured, and unstructured data processing.

Responsibilities

• Develop, test, and deploy ETL pipelines for batch and real-time stream processing.
• Design data pipelines using Python, SQL, PySpark, Scala, Kafka, and Java.
• Deploy data programs using cloud storage, data warehouses, and orchestration utilities on AWS, Azure, or GCP.
• Build CI/CD pipelines using Jenkins, CircleCI, or Git, and implement data versioning with tools like DVC or Pachyderm.
• Apply data quality checks using Great Expectations or Deequ, and document high-level and low-level designs.

Required Skills

• 3+ years of experience in data engineering.
• Proficiency in Python, SQL, PySpark, and Scala.
• Experience with Kafka, Java, and Docker.
• Hands-on experience with Airflow for orchestration.
• Cloud platform experience with AWS or Azure.
• Bachelor's degree in Computer Science.

Preferred Skills

• Knowledge of lakehouse architectures such as Delta Lake or Apache Iceberg.
• Ability to perform feasibility studies and travel to client locations within the US.

Key Skills

Education

Bachelor's degree in Computer Science