Description

Design and build scalable data pipeline platforms and automated Spark-based frameworks for data ingestion, transformation, and consumption.

Responsibilities

• Design and implement automated Spark frameworks to facilitate data ingestion and consumption.
• Build high-performing data pipeline platforms using Hadoop, Apache Spark, MongoDB, Kafka, and object storage.
• Implement security protocols including Kerberos authentication, data encryption at rest, and Apache Ranger for role-based access control.
• Develop automated testing frameworks for data validation and support troubleshooting for pipeline issues.
• Collaborate with infrastructure engineers, architects, and data analysts to design big-data infrastructure and scalable solutions.

Required Skills

• 5+ years of software engineering experience delivering complex enterprise-wide IT solutions.
• 5+ years of experience with ETL, data warehousing, and data analytics on Hadoop architecture.
• 5+ years of Apache Spark design and development using Scala, Java, Python, or Data Frames with RDDs, Parquet, or ORC formats.
• 6+ years of ETL programming experience.
• 5+ years of experience with complex SQL and performance tuning.
• 2+ years of experience with Kafka or equivalent streaming technologies.
• 2+ years of experience with NoSQL databases such as MongoDB or Couchbase.
• Experience working in hybrid environments where legacy ETL and big-data applications co-exist.

Preferred Skills

• 3+ years of Agile experience.
• 2+ years of experience with RESTful APIs or CI/CD tools.
• 2+ years of experience in reporting, analytics, or operational/credit risk and compliance domains.

Key Skills

Education

Any Graduate