Description

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks.
• Build and optimize data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop data transformation and processing workflows using PySpark and Spark SQL.
• Work with Delta Lake, Unity Catalog, and Databricks ecosystem tools to manage large-scale datasets.
• Collaborate with business stakeholders, data architects, and analytics teams to deliver data solutions.
• Implement data quality checks, monitoring, and performance optimization techniques.
• Ensure adherence to data governance, security, and privacy requirements in a Clean Room environment.
• Troubleshoot and resolve issues related to data pipelines and platform performance.
• Maintain technical documentation, data lineage, and process workflows.
• Support cloud-based data engineering initiatives and modernization programs.

Qualifications:

• 4-5 years of overall experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Proficiency in PySpark, Python, and Spark SQL.
• Experience working with Delta Lake and Databricks architecture.
• Strong understanding of ETL/ELT processes and Data Warehousing concepts.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Strong SQL skills and experience with relational and distributed databases.
• Familiarity with workflow orchestration tools such as Databricks Workflows, Airflow, etc.
• Experience with Git, CI/CD practices, and deployment automation.
• Good understanding of data security, governance, and compliance requirements.

Preferred Skills

• Experience working in Data Clean Room environments.
• Knowledge of privacy-preserving analytics and secure data-sharing frameworks.
• Experience with Azure Data Factory, Azure Synapse, Snowflake, or similar technologies.
• Databricks Certification (Associate/Professional) is a plus.
• Exposure to DevOps and Infrastructure-as-Code tools.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline

Key Skills

Education

Bachelor's or Master's degrees