Description

You will design, build, and maintain scalable data pipelines using Databricks and Python.

Responsibilities

• Develop and manage ETL/ELT workflows using Databricks Notebooks, Jobs, and Workflows.
• Build scalable data pipelines with Python and Apache Spark, optimizing for performance and cost.
• Design data models, schemas, and tables across SQL and NoSQL databases.
• Implement data quality, validation, and monitoring frameworks to ensure reliability.
• Troubleshoot pipeline failures and continuously improve system stability.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Python for data processing and automation.
• Expertise in Database Engineering: SQL, query tuning, and schema design.
• Hands-on experience with Apache Spark (RDDs/DataFrames, optimization, partitioning).
• Mandatory experience with Databricks components: Delta Lake, Spark SQL, Notebooks, Jobs, and Workflows.
• Experience with at least one cloud platform: AWS, Azure, or GCP.
• Familiarity with ETL/ELT tools (ADF, Glue, Dataflow, or equivalent).
• Experience implementing CI/CD for data pipelines using Git, DevOps, or Jenkins.

Key Skills

Education

Any Gradute