Description

Design, develop, test, and deploy scalable ETL/ELT pipelines for large-scale data processing using PySpark and Apache Spark.

Responsibilities

• Build and maintain complex logical and physical databases to ensure high performance, scalability, and security.
• Provide technical guidance and serve as an escalation point for complex data issues within the team.
• Collaborate with Scrum masters, product owners, and business partners to deliver project objectives using Agile methodologies.
• Ensure all data solutions adhere to strict financial industry regulations, data governance policies, and security standards.

Required Skills

• 5+ years of experience with compute engines, specifically Apache Spark and PySpark.
• Strong knowledge of data layers including Lakehouse/Objectstore, Iceberg, and Delta Lake.
• Proficiency in SQL for data manipulation and querying.
• Experience with streaming technologies, particularly Kafka.
• Ability to work effectively in a financial industry environment with strict compliance requirements.

Preferred Skills

• Familiarity with agentic AI tools such as LangGraph, LangChain, and A2A.
• Previous experience working in the financial sector.

Key Skills

Education

Any Graduate