Description

You will design and maintain scalable data pipelines for batch and real-time processing across cloud platforms.

This role is remote.

Responsibilities

• Build and optimize ETL/ELT workflows using Python, SQL, and Spark.
• Develop automated testing, monitoring, and deployment processes for data pipelines.
• Implement scalable data models to support business intelligence and analytics.
• Troubleshoot production issues and improve data platform performance and cost efficiency.
• Collaborate with data scientists and stakeholders to ensure data quality and governance.

Required Skills

• 6-10 years of experience in Data Engineering.
• Strong programming experience in Python and SQL.
• Hands-on experience with Apache Spark, PySpark, and Databricks.
• Experience with Azure Data Factory (ADF), AWS Glue, or similar orchestration tools.
• Proficiency with Azure Data Lake Storage (ADLS), Amazon S3, or similar storage.
• Knowledge of Azure Synapse Analytics, Snowflake, or Redshift.
• Experience with Git, CI/CD pipelines, and version control.
• Strong understanding of ETL/ELT design patterns and data warehousing concepts.

Preferred Skills

• Docker containerization experience.

Key Skills

Education

Bachelor's degree