Description

You will design, build, and maintain scalable ETL/ELT workflows on Databricks, optimizing data transformation logic for performance and cost efficiency. You will develop robust data pipelines using Python and Spark, integrating data from APIs, streaming sources, and cloud storage while ensuring data security and governance.

Responsibilities

• Design and maintain ETL/ELT workflows on Databricks (Delta Lake, Notebooks, Spark SQL, Jobs, Workflows).
• Develop scalable data pipelines using Python and Spark for large-scale datasets.
• Optimize data transformation logic, query tuning, and partitioning for performance and cost efficiency.
• Implement data quality, validation, and monitoring frameworks to ensure reliability.
• Collaborate with Data Scientists and Cloud Engineers to support analytics and ML workloads.

Required Skills

• Databricks (mandatory): Delta Lake, Notebooks, Spark SQL, Jobs, Workflows.
• Python: Data processing, Spark APIs, automation, and optimization.
• Apache Spark: RDDs/DataFrames, optimization, and partitioning.
• Database Engineering: Strong SQL, query tuning, schema design, stored procedures (SQL/NoSQL).
• Cloud Platforms: Experience with AWS, Azure, or GCP.
• ETL/ELT Tools: ADF, Glue, Dataflow, or equivalent.
• CI/CD for data pipelines: Git, DevOps, Jenkins.

Preferred Skills

• Experience working with structured and unstructured data from various sources.
• Background in data modeling, schema design, and table management.

Key Skills

Education

Bachelor's degree