Description

You will design and implement scalable data pipelines using Azure Databricks, Apache Spark, and Azure Data Factory. You will build, optimize, and maintain ETL/ELT workflows for large datasets, ensuring high performance and reliability.

This role is remote.

Responsibilities

• Develop and maintain robust ETL/ELT workflows using Apache Spark and PySpark DataFrames.
• Design and support modern data architecture using Lakehouse principles and distributed processing.
• Implement data quality frameworks and maintain data lineage across enterprise data platforms.
• Utilize data governance tools to manage access, security, compliance, and lifecycle.
• Apply Master Data Management (MDM) concepts to ensure data consistency and integrity.

Required Skills

• 5+ years of experience with Azure Databricks and Apache Spark.
• Proficiency in PySpark DataFrames and SQL-based data transformations.
• Experience building data pipelines and ETL/ELT processes using Azure Data Factory.
• Strong understanding of Lakehouse Architecture and distributed data processing.
• Practical experience with Data Governance, data lineage, and MDM concepts.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree