Description

You will build and maintain data ingestion and transformation pipelines within a Databricks environment, ensuring data governance and performance monitoring.

This role is on-site.

Responsibilities

• Build and maintain data ingestion and transformation pipelines based on existing data models.
• Design and maintain a framework for pipeline observation, monitoring, and performance tuning.
• Implement data governance, security, and compliance best practices.
• Troubleshoot data integration errors related to processing timing, performance, and integrity.
• Participate in Agile/Scrum ceremonies including sprint planning and reviews.

Required Skills

• 5+ years of experience building and maintaining Databricks ecosystems.
• Expertise with Delta Live Tables (DLT), Unity Catalog, and Medallion Architecture.
• Hands-on experience with Apache Kafka and streaming data technologies.
• Proficiency in Python, specifically using Pandas, NumPy, Boto3, and SimpleSalesforce.
• Development experience with PySpark, PySQL, and Apache Spark.
• Infrastructure management using Terraform.
• Experience building data-lake style infrastructures.
• Knowledge of data security and compliance frameworks such as GDPR or HIPAA.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate