Description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize ETL/ELT frameworks for processing large volumes of structured and unstructured data.
• Develop data ingestion solutions from multiple source systems into cloud data lakes and warehouses.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Implement data quality, validation, and monitoring frameworks.
• Perform data modeling and transformation for reporting and analytics use cases.
• Troubleshoot production issues and provide timely resolutions.
• Follow best practices for coding, testing, version control, and deployment.

Required Skills

• 8+ years of experience in Data Engineering.
• Strong hands on experience with Databricks.
• Expertise in PySpark, Python, and SQL.
• Experience with Spark optimization techniques and performance tuning.
• Strong understanding of data warehousing concepts and dimensional modeling.
• Experience building ETL/ELT pipelines.
• Knowledge of Delta Lake, Spark SQL, and Databricks workflows.
• Experience with Git, CI/CD, and Agile methodologies.
• Strong analytical and problem solving skills.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of Azure Data Factory (ADF), Airflow, or other orchestration tools.
• Experience with streaming technologies such as Kafka or Structured Streaming.
• Exposure to Power BI, Tableau, or other analytics tools.
• Databricks or cloud certifications are a plus.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field

Key Skills

Education

Bachelor's degree