Description

Key Responsibilities:

• Design, build, and maintain ETL/ELT workflows on Databricks.
• Develop scalable data pipelines using Python and Spark.
• Optimize data transformation logic for performance and cost efficiency.
• Work with structured and unstructured data from various sources.
• Develop and manage data models, schemas, and tables across databases (SQL/NoSQL).
• Implement data quality, validation, and monitoring frameworks.
• Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads.
• Ensure data security, governance, and compliance in all solutions.
• Troubleshoot pipeline failures and continuously improve pipeline reliability.
• Integrate data from APIs, streaming sources, and cloud storage.

Required Skills:
Core Technical Skills:-

• Databricks (mandatory) – Delta Lake, Notebooks, Spark SQL, Jobs, Workflows
• Python – Data processing, Spark APIs, automation, optimization
• Database Engineering – Strong SQL, query tuning, schema design, stored procedures
• Apache Spark – RDDs/DataFrames, optimisation, partitioning
• Cloud Platforms (one or more): AWS / Azure / GCP
• ETL/ELT tools: ADF, Glue, Dataflow, or equivalent
• Experience working with large-scale datasets
• CI/CD for data pipelines (Git, DevOps, Jenkins, etc.)

Key Skills

Education

Bachelor's degree