PySpark/Palantir Developer

Design and optimize large-scale data pipelines using PySpark and Palantir Foundry for enterprise analytics.

This role is on-site.

Responsibilities

• Build and maintain PySpark data pipelines to process large-scale structured and unstructured datasets.
• Develop modular data workflows in Palantir Foundry, integrating diverse enterprise sources and managing dataset transformations.
• Optimize Spark transformations, query patterns, and cluster configurations to improve runtime performance and resource utilization.
• Collaborate with product owners and analysts to translate business requirements into reusable technical specifications.
• Conduct code reviews, troubleshoot pipeline incidents, and document data lineage and business rules.

Required Skills

• 6+ years of hands-on experience in data engineering.
• Strong proficiency in PySpark and distributed computing.
• Experience with Palantir Foundry for data workflow management.
• Proficiency in SQL and Python.
• Deep understanding of ETL processes and data modeling.
• Knowledge of data pipelines, big data technologies, and performance tuning.

Preferred Skills

• Experience with version control systems, testing frameworks, and data validation techniques.
• Knowledge of shuffle optimization, partitioning strategies, and caching mechanisms.
• Familiarity with secure coding practices and root cause analysis.