Description

What You'll Do: 

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize enterprise-grade ETL and ELT frameworks for large-scale data processing.
• Investigate and resolve data quality issues, pipeline failures, and production incidents.
• Perform root cause analysis for framework failures, processing issues, and application defects.
• Monitor, tune, and optimize Spark workloads for performance, scalability, and reliability.
• Develop and optimize SQL queries for data extraction, transformation, validation, and reporting.
• Collaborate with business users, architects, and engineering teams to implement robust data solutions.
• Support production deployments and provide L2/L3 support for critical data platforms and applications.
• Troubleshoot complex data processing, workflow, and platform-related issues.
• Implement monitoring, alerting, and operational support processes to improve platform stability.
• Ensure data quality, accuracy, consistency, and availability across systems.
• Drive adoption of engineering best practices, coding standards, and operational governance.
• Participate in code reviews and contribute to continuous improvement initiatives.
• Support cloud-native data solutions using Azure and AWS platforms.
• Create and maintain technical documentation, operational runbooks, and support procedures.
• Contribute to Agile ceremonies including sprint planning, stand-ups, reviews, and retrospectives.

Expertise You'll Bring:

• 6 to 8 years of experience in Data Engineering, Big Data, and cloud-based data platforms.
• Minimum 5+ years of hands-on experience with Databricks and PySpark.
• Strong expertise in designing and developing distributed data processing solutions.
• Extensive experience in Production Support, incident management, and root cause analysis.
• Strong SQL development skills with experience in query optimization and performance tuning.
• Experience building and supporting complex ETL/ELT pipelines and large-scale data frameworks.
• Strong debugging, troubleshooting, and performance optimization capabilities.
• Hands-on experience with Azure or AWS cloud platforms.
• Experience working with Spark optimization techniques, job performance tuning, and cluster management.
• Knowledge of Delta Lake architecture and modern Lakehouse implementations.
• Understanding of data governance, data quality, and data security best practices.
• Familiarity with CI/CD pipelines, DevOps practices, and deployment automation.
• Experience with monitoring, observability, and operational support tools.
• Strong analytical and problem-solving skills with the ability to resolve complex technical issues.
• Excellent communication and stakeholder management capabilities.
• Bachelor's degree in Engineering, Computer Science, Information Technology, or equivalent experience

Key Skills

Education

Any Graduate