Description

Design, develop, and maintain scalable batch and stream data pipelines using Python and PySpark in distributed environments.

This role is on-site.

Responsibilities

• Implement efficient transformations, aggregations, and joins on large datasets while ensuring performance and cost optimization.
• Write optimized SQL for data extraction, validation, and reconciliation across multiple sources.
• Build reusable, testable modules and follow engineering best practices including code reviews, unit testing, and documentation.
• Troubleshoot production issues, perform root-cause analysis, and implement long-term fixes and monitoring improvements.
• Collaborate with stakeholders to translate requirements into technical designs, delivery plans, and measurable outcomes.

Required Skills

• 5–9 years of hands-on experience in software development and/or data engineering roles.
• Strong proficiency in Python with experience building production-grade applications or data workflows.
• Strong proficiency in PySpark, including DataFrame APIs, optimization techniques, and distributed processing concepts.
• Working knowledge of SQL for complex queries, data analysis, and validation.
• Experience delivering reliable solutions with attention to performance, scalability, and maintainability.
• Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).

Key Skills

Education

Bachelor’s degree