Description

You will design, develop, and maintain scalable ETL/ELT pipelines using Databricks to ingest and process structured, semi-structured, and unstructured data.

This role is hybrid.

Responsibilities

• Build and optimize data ingestion frameworks and transformation workflows using PySpark and Spark SQL.
• Manage large-scale datasets using Delta Lake, Unity Catalog, and Databricks ecosystem tools.
• Implement data quality checks, monitoring, and performance optimization for cloud-based data initiatives.
• Collaborate with stakeholders and data architects to deliver secure data solutions within Clean Room environments.
• Maintain technical documentation, data lineage, and resolve pipeline issues.

Required Skills

• 5+ years of overall experience in Data Engineering.
• Strong hands-on experience with Databricks architecture and Delta Lake.
• Proficiency in PySpark, Python, and Spark SQL.
• Strong understanding of ETL/ELT processes and Data Warehousing concepts.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Strong SQL skills and experience with relational and distributed databases.
• Familiarity with workflow orchestration tools like Databricks Workflows or Airflow.
• Experience with Git, CI/CD practices, and deployment automation.

Preferred Skills

• Experience working in Data Clean Room environments and knowledge of privacy-preserving analytics.
• Experience with Azure Data Factory, Azure Synapse, Snowflake, or similar technologies.
• Databricks Certification (Associate/Professional) or exposure to DevOps and Infrastructure-as-Code tools.

Key Skills

Education

Bachelor's or Master's degrees