Description

You will design, build, and optimize scalable data pipelines using Databricks, Python, and Spark to support analytics and business intelligence initiatives.

This role is on-site.

Responsibilities

• Design, build, and maintain ETL/ELT workflows on Databricks.
• Develop scalable data pipelines using Python and Spark for structured and unstructured data.
• Manage data models, schemas, and tables across SQL and NoSQL databases.
• Implement data quality, validation, and monitoring frameworks.
• Troubleshoot pipeline failures and integrate data from APIs and streaming sources.

Required Skills

• Databricks: Delta Lake, Notebooks, Spark SQL, Jobs, Workflows.
• Python: Data processing, Spark APIs, automation, and optimization.
• Apache Spark: RDDs/DataFrames, optimization, and partitioning.
• Database Engineering: Strong SQL, query tuning, schema design, stored procedures.
• Cloud Platforms: AWS, Azure, or GCP.
• ETL/ELT Tools: ADF, Glue, Dataflow, or equivalent.
• CI/CD for data pipelines using Git, DevOps, or Jenkins.
• 5+ years of experience with large-scale datasets.

Preferred Skills

• Experience collaborating with Data Scientists and ML engineers.
• Knowledge of data security, governance, and compliance standards.

Key Skills

Education

Any Graduate