Description

Own the design and maintenance of scalable data pipelines using PySpark and Python. You will orchestrate complex workflows, optimize Spark performance, and ensure data integrity across Cloudera, Databricks, and Starburst environments.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines using PySpark and advanced Python.
• Develop, schedule, and monitor data workflows using Apache Airflow.
• Optimize and tune Spark jobs for performance and efficiency.
• Implement data quality checks and ensure data integrity across all pipelines.
• Design data models for optimal storage and retrieval of business data.

Required Skills

• 6+ years of professional experience in data engineering.
• Extensive hands-on experience with PySpark and Python.
• Proficiency with Big Data platforms: Cloudera, Databricks, and Starburst.
• Strong SQL skills and experience with ETL/ELT processes.
• Experience with data modeling and data warehousing concepts.
• Ability to implement data governance and security policies.

Preferred Skills

• Experience with Trino/Presto.
• Familiarity with Linux/Unix environments, GitHub, and CI/CD pipelines.
• Experience mentoring junior engineers and promoting best practices.

Key Skills

Education

Any Graduate