Description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, SQL, and Databricks.
• Build and optimize ETL/ELT workflows for batch and large-scale data processing.
• Develop data transformation logic using PySpark and advanced SQL.
• Work with Databricks notebooks, jobs, clusters, and Delta Lake tables.
• Ingest, cleanse, transform, validate, and integrate data from multiple sources.
• Optimize data processing performance, query execution, and resource utilization.
• Implement data quality checks, error handling, logging, and monitoring.
• Collaborate with data architects, analysts, data scientists, and business stakeholders to understand data requirements.
• Develop and maintain data models and curated datasets for reporting and analytics.
• Implement data governance, security, access controls, and data integrity standards.
• Troubleshoot pipeline failures and resolve data processing issues.
• Participate in code reviews, technical design discussions, testing, and Agile ceremonies.

Required Qualifications

• 7–8 years of experience in data engineering, data development, or related roles.
• Strong hands-on programming experience with Python and PySpark.
• Advanced SQL skills, including complex queries, joins, aggregations, CTEs, and performance tuning.
• Hands-on experience with Databricks for data engineering and distributed data processing.
• Strong understanding of ETL/ELT concepts and data pipeline architecture.
• Experience working with large datasets and distributed computing frameworks.
• Knowledge of data warehousing, dimensional modeling, and data transformation techniques.
• Experience with Delta Lake, Spark SQL, and Databricks Jobs.
• Familiarity with Git, version control, testing, and deployment practices.
• Strong analytical, troubleshooting, and problem-solving skills

Key Skills

Education

Not specified