Description

You will design and manage large-scale data pipelines and architectures to extract value from disconnected datasets.

Responsibilities

• Build and optimize big data pipelines, data structures, and metadata management processes.
• Integrate traditional databases like Oracle, SQL Server, and MySQL into Hadoop Data Lakes.
• Develop enterprise data lake applications using Spark Streaming, Kafka, and Sqoop.
• Automate manual reporting jobs to support business data analysis.
• Perform root cause analysis on internal and external data to identify process improvements.

Required Skills

• 10+ years of experience in big data engineering.
• Advanced SQL knowledge and experience with relational databases.
• Proficiency with Hadoop Ecosystem, Hive, and Sqoop.
• Hands-on experience with Spark, PySpark, and Scala.
• Strong programming skills in Python and Unix/Shell scripting.
• Experience with AWS big data technologies.
• Working knowledge of message queuing, stream processing, and scalable data stores.
• Experience with Kafka, Oozie, HDFS, and Streamsets.
• Ability to work with unstructured datasets and perform complex data transformations.

Preferred Skills

• Experience using Tidal for workload management.

Key Skills

Education

Any Graduate