Description

You will design and implement scalable data pipelines and ETL processes using Databricks and PySpark. You own the development and optimization of SQL queries for data transformation and analytics, ensuring high performance for large-scale datasets. You collaborate with data architects and business stakeholders to deliver robust data solutions, enforcing strict data quality, governance, and compliance standards. You troubleshoot workflow bottlenecks and integrate data from multiple sources efficiently.

Responsibilities

• Design and implement data pipelines and ETL processes using Databricks and PySpark.
• Develop and optimize SQL queries for data transformation and analytics.
• Ensure data quality, governance, and compliance across all data workflows.
• Integrate data from multiple sources and manage large-scale datasets efficiently.
• Troubleshoot and optimize performance for data workflows and queries.

Required Skills

• 5+ years of experience in building and managing data pipelines in cloud environments.
• Expertise in Databricks, including Delta Lake and ML capabilities.
• Strong proficiency in PySpark for distributed data processing.
• Advanced SQL skills for data manipulation and analytics.
• Experience with Azure Services (ADF, Synapse) or AWS Services (Glue, Lambda, Step Functions).
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
• Ability to troubleshoot complex data integration issues.

Preferred Skills

• Experience with multi-cloud environments (Azure, AWS, GCP).
• Familiarity with workflow orchestration tools like Airflow or data transformation tools like DBT.
• Knowledge of data governance frameworks and streaming technologies like Kafka.

Key Skills

Education

Bachelor's or Master's degrees