Description

Build and maintain automated data pipelines that cleanse, organize, and transform structured and unstructured data from diverse sources.

Responsibilities

• Design and implement data extraction, transformation, and loading (ETL) processes using Spark and Python.
• Manage data integrity and storage within Azure, Greenplum, and Teradata environments.
• Develop Unix shell scripts to automate data operations and system maintenance.
• Process real-time data streams using Kafka to support actionable business insights.
• Analyze database performance to identify optimization opportunities and recommend improvements.

Required Skills

• 5+ years of experience in data engineering or related technical roles.
• Proficiency in Python for data processing and tool development.
• Hands-on experience with Apache Spark for large-scale data transformation.
• Strong knowledge of Azure cloud services for data architecture and deployment.
• Experience with message streaming platforms, specifically Kafka.
• Familiarity with relational and columnar databases such as Greenplum and Teradata.
• Ability to write and maintain Unix shell scripts for automation tasks.
• Any graduate degree.

Key Skills

Education

Any Gradute