Description

You will design and build large-scale data pipelines for batch and streaming workloads using PySpark and Python.

This role is remote.

Responsibilities

• Develop production-grade PySpark applications for complex transformations, window functions, and multi-dataset joins.
• Build Structured Streaming pipelines with Kafka, Event Hubs, or Pub/Sub, applying watermarking and stateful processing.
• Optimize Spark performance through partition tuning, skew mitigation, AQE, and executor sizing.
• Create Python-based data services, REST API integrations, and automation scripts supporting the enterprise platform.
• Implement lakehouse architectures on Azure, GCP, or AWS using Delta Lake, Iceberg, or Hudi.

Required Skills

• 5+ years of experience with PySpark and Python development.
• Strong expertise in PySpark DataFrame operations, UDFs, and Pandas UDFs.
• Experience with cloud platforms: Azure, GCP, or AWS.
• Hands-on experience with Databricks, Delta Lake, or similar lakehouse technologies.
• Proficiency in building REST APIs and integrating with third-party systems.
• Knowledge of data quality tools like Great Expectations or Pandera.
• Familiarity with orchestration tools such as Airflow or Databricks Workflows.
• Understanding of software engineering practices including unit testing and modular design.

Preferred Skills

• Experience with SAP S/4HANA, Salesforce, or other FMCG data sources.
• Knowledge of Unity Catalog for governance and lineage.

Key Skills

Education

Any Graduate