Description

You will build and maintain data pipelines that transform raw data into actionable information for data science and product teams.

Responsibilities

• Extract data from Hadoop databases using Sqoop and manage Linux-based systems.
• Stage real-time data from gateways into AWS S3 or Azure Blob storage.
• Implement Spark processing using Scala, DataFrames, and the Spark SQL API.
• Optimize Spark session performance through effective partitioning.
• Develop ETL pipelines in ADF using Linked Services and Datasets to move data between Azure SQL, Blob storage, and Data Warehouses.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency with Linux and Hadoop environments.
• Strong SQL skills for data analysis and transformation.
• Experience with AWS (S3) or Azure Blob storage.
• Hands-on experience with Spark SQL API.
• Programming proficiency in Java.
• Experience with Scala for Spark implementations.
• Knowledge of ADF (Azure Data Factory) for ETL processes.

Preferred Skills

• Experience with Sqoop for data ingestion.

Key Skills

Education

Any Graduate