Description

You build and maintain data pipelines, ELT processes, and workflow orchestration to ensure data delivery.

Responsibilities

• Design and implement custom connectors to ingest structured and unstructured data from various sources.
• Implement DataOps principles and design data CI/CD pipelines for automated data processes.
• Monitor and troubleshoot data pipelines, resolving ingestion, transformation, and loading issues.
• Conduct data validation and testing to ensure data accuracy and compliance.
• Document data workflows, processes, and technical specifications.

Required Skills

• 8 years experience in data engineering, ELT development, and data modeling.
• Proficiency in Apache Airflow and PySpark for data transformation and integration.
• Demonstrated experience developing custom connectors for data ingestion.
• Strong understanding of SQL and database concepts, including query optimization.
• Experience implementing DataOps principles and data CI/CD pipelines.
• Knowledge of Python, Git, and CI/CD practices.
• Understanding of distributed systems and large-scale datasets.
• Experience implementing workflow orchestration using tools like Apache Airflow.
• Strong software development principles, including version control and code review.

Key Skills

Education

Bachelor's degree