Description

You will design, develop, and maintain scalable data pipelines using Databricks. You will build and optimize ETL/ELT workflows for structured and unstructured data while ensuring data quality and reliability. You will work with large-scale datasets, implement data modeling best practices, and troubleshoot production issues during the evening shift (4 PM – 1 AM).

Responsibilities

• Design and maintain scalable data pipelines using Databricks.
• Develop data transformations using PySpark and Spark SQL.
• Optimize ETL/ELT workflows for structured and unstructured data.
• Monitor workflows, troubleshoot production issues, and perform performance tuning.
• Collaborate with data scientists, analysts, and stakeholders on data requirements.

Required Skills

• 10+ years of experience in data engineering roles.
• Strong hands-on expertise with Databricks and Apache Spark (PySpark / Spark SQL).
• Solid experience building data pipelines, data lakes, and data modeling.
• Proficient in SQL and workflow orchestration tools (Airflow, ADF).
• Experience with cloud platforms: AWS, Azure, or GCP.
• Strong debugging and performance tuning skills for large-scale datasets.
• BE / BTech / MCA / MSc in Computer Science, Data Engineering, or related field.

Preferred Skills

• Experience with Delta Lake and Databricks optimization techniques.
• Knowledge of CI/CD for data pipelines.
• Exposure to real-time or streaming data processing.

Key Skills

Education

Any Gradute