Description

You will implement data pipelines using Spark and Scala to ingest, transform, and load data from various sources.

Responsibilities

• Develop Software Development Kits (SDKs) in Java and Scala to parse data from REST APIs and expose functionality as dependencies.
• Create applications that ingest data from multiple sources and publish events to Amazon SNS topics on AWS.
• Build and maintain the data lake, making data available in formats like Parquet, Avro, RC, and JSON.
• Configure, run, and monitor Spark jobs and data storage using various AWS services.
• Tune Hive queries by parallelizing and optimizing operations like 'cluster by' and 'distribute by'.

Required Skills

• 5+ years of professional experience.
• Strong proficiency in Java and Scala.
• Experience with Spark for data transformation.
• Working knowledge of AWS services including EC2, S3 Buckets, EMR, Lambda, SNS, CloudWatch, and Elastic Search.
• Experience using Hadoop ecosystem tools such as Hive and Sqoop.
• Familiarity with SQL for querying data in Hive and Spark SQL.
• Experience with data formats like JSON, Parquet, and Avro.
• Database experience with MySQL and Oracle.

Key Skills

Education

Any Gradute