Description

You will design, build, and optimize large-scale data pipelines and analytics solutions within the GWD platform.

This role is hybrid.

Responsibilities

• Develop and implement data applications across the full SDLC, from design and build to test and deployment.
• Design and implement data pipelines using PySpark to ingest, process, and deliver large datasets.
• Lead the migration of legacy database modules onto the big data stack while ensuring data integrity and security.
• Optimize and troubleshoot Spark jobs for performance, focusing on partitioning strategies and execution plans.
• Analyze deep data patterns to identify gaps and support platform enhancements.

Required Skills

• 10+ years of experience in data architecture, database design, or data engineering with a focus on big data.
• Deep expertise in PySpark development, including RDDs, DataFrames, transformations, and actions.
• Strong experience with Oracle SQL programming and development.
• Hands-on knowledge of Hadoop ecosystem components (HDFS, YARN) and Hive.
• Strong grasp of Object-Oriented Programming (OOP) principles applied within PySpark for maintainable code.
• Proven experience working with large datasets (e.g., ~100 million records) and optimizing for scalability.
• Familiarity with the full SDLC process.

Key Skills

Education

Any Graduate