Description

Design, build, and optimize scalable data pipelines using Databricks, Python, and database engineering.

This role is on-site.

Responsibilities

• Design, build, and maintain ETL/ELT workflows on Databricks.
• Develop scalable data pipelines using Python and Spark for structured and unstructured data.
• Optimize data transformation logic for performance and cost efficiency.
• Develop and manage data models, schemas, and tables across SQL/NoSQL databases.
• Implement data quality, validation, and monitoring frameworks.

Required Skills

• 5+ years of experience in data engineering.
• Mandatory expertise in Databricks (Delta Lake, Notebooks, Spark SQL, Jobs, Workflows).
• Strong proficiency in Python for data processing, Spark APIs, and automation.
• Deep knowledge of Apache Spark (RDDs/DataFrames, optimization, partitioning).
• Strong SQL skills including query tuning, schema design, and stored procedures.
• Experience with cloud platforms: AWS, Azure, or GCP.
• Familiarity with ETL/ELT tools such as ADF, Glue, or Dataflow.
• Experience with CI/CD for data pipelines (Git, DevOps, Jenkins).

Preferred Skills

• Experience integrating data from APIs and streaming sources.
• Knowledge of data security, governance, and compliance.

Key Skills

Education

Any Graduate