Description

You design, build, and optimize big data ETL/ELT pipelines using PySpark, SQL, and Delta Lake in cloud environments.

This role is on-site.

Responsibilities

• Design and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
• Process structured and unstructured data, leveraging Delta Lake for efficient storage and management.
• Perform performance tuning and optimization on Databricks clusters and Spark jobs for cost-efficiency.
• Orchestrate workflows using Azure Data Factory (ADF) or Airflow.
• Integrate Databricks with cloud data sources (Azure Data Lake Gen2, S3) and BI tools.

Required Skills

• Expert-level knowledge of PySpark, Apache Spark, and SQL.
• Hands-on experience with Azure Databricks, AWS Databricks, or GCP.
• Strong understanding of Data Modeling, ETL/ELT patterns, and Data Warehousing concepts.
• Proficiency in Python and SQL.
• Experience with Azure Data Factory (ADF) or Airflow for workflow orchestration.
• Familiarity with Git and Azure DevOps.
• 6-8 years of experience in data engineering or software development.

Preferred Skills

• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree