Description

Key Skills: PySpark, Palantir Foundry, Data Engineering, SQL, Data Modeling, Distributed Computing, ETL, Data Pipelines, Python, Big Data

Good to Have Skills: Experience with version control systems, testing frameworks, data validation techniques, performance tuning, shuffle optimization, partitioning strategies, caching mechanisms, collaborative development practices, hybrid work environment experience, cross-functional team collaboration, code review processes, troubleshooting and root cause analysis, data lineage documentation, secure coding practices.

Roles & Responsibilities:

• Design robust PySpark data pipelines that reliably process large scale structured and unstructured datasets to enable accurate reporting and analytics for business stakeholders.
• Develop optimized transformations in PySpark that improve runtime performance and resource utilization while maintaining high standards of data quality and consistency.
• Implement modular data workflows in Palantir Foundry that integrate diverse enterprise data sources and provide curated datasets for downstream applications.
• Configure and manage datasets transformations and schedules in Palantir Foundry to ensure that critical data assets remain fresh traceable and well documented.
• Collaborate with product owners data analysts and other developers to translate business requirements into clear technical specifications and reusable data components.
• Conduct detailed code reviews for PySpark and Foundry transformation logic to uphold coding standards improve maintainability and reduce production issues.
• Troubleshoot complex data pipeline incidents by performing root cause analysis and implementing sustainable fixes that prevent recurrence and protect service reliability.
• Optimize data models and query patterns so that analytical and operational dashboards perform efficiently and deliver timely insights to decision makers.
• Document data lineage business rules and transformation logic in a clear and accessible manner so that teams across the organization can confidently reuse shared data assets.
• Partner with platform and infrastructure teams to ensure that Spark cluster configurations job schedules and resource allocations align with performance and cost objectives.
• Apply secure coding and data handling practices to safeguard sensitive information and comply with internal policies and external regulatory expectations.
• Provide mentoring and guidance to less experienced developers on PySpark patterns testing approaches and best practices for building reliable data solutions.

Experience Required: 6 to 9 years of hands-on experience in designing and implementing data engineering solutions using PySpark in large scale enterprise environments

Key Skills

Education

Any Graduate