Description

You will build and maintain large-scale distributed computing applications and data processing systems for operational and analytical needs.

This role is on-site.

Responsibilities

• Design, develop, test, and tune data ingestion pipelines and data products using the Hadoop ecosystem.
• Build Spark jobs in Scala for data processing and implement Spark Streaming with Kafka for real-time ingestion.
• Optimize HiveQL queries through partitioning and bucketing to ensure high throughput.
• Develop ETL applications to extract data from DB2 sources and manage secure transfers to third-party vendors.
• Participate in sprint planning, design reviews, and code reviews to support platform-level governance.

Required Skills

• 5+ years of experience in Big Data development.
• Proficiency with Hadoop, Hive, Spark, SparkSQL, Sqoop, Kafka, and ZooKeeper.
• Hands-on experience with Scala for Spark job development.
• Experience managing file formats including PARQUET, ORC, Avro, CSV, and JSON.
• Knowledge of NoSQL databases, specifically HBase.
• Experience working with Hortonworks Distribution and Amazon Web Services S3.
• Proficiency with GIT for version control and code reviews.
• Ability to configure Control-M for batch and streaming job scheduling.

Preferred Skills

• Bachelor's degree in Computer Science, IT, or a related field.

Key Skills

Education

Bachelor's Degree