Description

Own the design, maintenance, and optimization of scalable data pipelines and workflows across Cloudera, Databricks, and Starburst environments.

This role is on-site.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines using PySpark and advanced Python.
• Orchestrate, schedule, and monitor complex data workflows using Apache Airflow.
• Optimize and tune Spark jobs for performance, efficiency, and resource utilization.
• Implement data quality checks, governance policies, and ensure data integrity across all pipelines.
• Design data models for optimal storage and retrieval of business data.

Required Skills

• 6+ years of professional experience in data engineering.
• Extensive hands-on experience with PySpark and Python.
• Proficiency with Big Data platforms: Cloudera, Databricks, and Starburst.
• Strong SQL skills and deep experience with ETL/ELT processes.
• Experience with data modeling and data warehousing concepts.
• Ability to implement data governance and security policies.

Preferred Skills

• Experience with Trino/Presto.
• Familiarity with Linux/Unix environments, GitHub, and CI/CD pipelines.
• Experience mentoring junior engineers and promoting best practices.

Key Skills

Education

Any Graduate