Description

You will transform raw data into actionable information for data science and product teams. You own the end-to-end data engineering lifecycle, from ingestion to optimization.

Responsibilities

• Extract data from Hadoop databases using Sqoop and manage Linux-based systems.
• Stage real-time data from gateways into AWS S3 or Azure Blob storage.
• Develop Spark streaming pipelines in Java and implement Spark using Scala.
• Build ETL pipelines in ADF using Linked Services, Datasets, and Pipelines to move data between Azure SQL, Blob storage, and Azure SQL Data Warehouse.
• Optimize data processing performance by utilizing Spark partitions and Spark SQL API.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency with Spark and Spark SQL API.
• Experience programming in Scala and Java.
• Strong SQL skills for data analysis and transformation.
• Hands-on experience with Hadoop and Linux systems.
• Knowledge of AWS S3 or Azure Blob storage.
• Experience with Sqoop for data ingestion.
• Ability to travel or relocate to client sites throughout the USA as required.

Preferred Skills

• Experience with Azure Data Factory (ADF) and Azure SQL Data Warehouse.

Key Skills

Education

Any Graduate