Description

You will build and maintain data pipelines and storage solutions using Spark, Scala, and AWS services.

This role is on-site.

Responsibilities

• Implement data pipelines using Spark and Scala to ingest, transform, and load data from various sources.
• Develop SDKs in Java and Scala to parse data from REST APIs and expose functionality as dependencies.
• Build applications that ingest data from multiple sources and publish events to Amazon SNS topics.
• Build and maintain the data lake, making data available in formats like Parquet, Avro, and JSON.
• Configure, run, and monitor Spark jobs and data storage using various AWS services.

Required Skills

• 5+ years of professional software development experience.
• Strong proficiency in Java and Scala.
• Experience with Spark for data transformation and pipeline development.
• Working knowledge of AWS services including EC2, S3, EMR, Lambda, SNS, CloudWatch, and Elastic Search.
• Experience using Hadoop ecosystem tools such as Hive and Sqoop.
• Familiarity with SQL for querying data in Hive and Spark SQL.
• Experience with data formats like JSON, Parquet, and Avro.
• Database experience with MySQL and Oracle.

Key Skills

Education

Any Gradute