Description

You will design and implement scalable data pipelines and ETL processes using Databricks and PySpark. This role is hybrid.

Responsibilities

• Design and implement data pipelines and ETL processes using Databricks and PySpark.
• Develop and optimize SQL queries for data transformation and analytics.
• Ensure data quality, governance, and compliance across all data workflows.
• Integrate data from multiple sources and manage large-scale datasets efficiently.
• Troubleshoot and optimize performance for data workflows and queries.

Required Skills

• 5+ years of experience building and managing data pipelines in cloud environments.
• Expertise in Databricks, including Delta Lake and ML capabilities.
• Strong proficiency in PySpark for distributed data processing.
• Advanced SQL skills for data manipulation and analytics.
• Experience with Azure Services (ADF, Synapse) or AWS Services (Glue, Lambda, Step Functions).
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.

Preferred Skills

• Experience with multi-cloud environments (Azure, AWS, GCP).
• Familiarity with workflow orchestration tools like Airflow or data transformation tools like DBT.
• Knowledge of data governance frameworks and streaming technologies like Kafka.

Key Skills

Education

Bachelor's or Master's degrees