Description

Transform raw data into actionable information for data science and product teams by building and optimizing large-scale data pipelines.

This role is on-site.

Responsibilities

• Extract data from Hadoop databases using Sqoop and manage Linux-based systems.
• Stage real-time data from gateways into AWS S3 or Azure Blob storage.
• Analyze and transform data through preprocessing, standardization, and filtering stages using SQL.
• Implement Spark processing using Scala, DataFrames, and the Spark SQL API.
• Build ETL pipelines in Azure Data Factory (ADF) to move data between Azure SQL, Blob storage, and Azure SQL Data Warehouse.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency with Java and Scala.
• Experience with Spark SQL API and Spark DataFrames.
• Hands-on work with Hadoop and Sqoop.
• Experience managing data in AWS (S3) or Azure (Blob storage).
• Strong SQL querying and data transformation skills.
• Working knowledge of Linux systems.
• Experience with Azure SQL and Azure SQL Data Warehouse.
• Ability to travel or relocate to client sites within the USA as required.

Preferred Skills

• Optimize Spark session performance through effective partitioning.
• Develop Spark streaming pipelines in Java.

Key Skills

Education

Any Graduate