Description

Build and maintain automated data pipelines that cleanse, organize, and transform structured and unstructured data from diverse sources.

This role is on-site.

Responsibilities

• Design and implement ETL processes using Spark and Python.
• Manage data integrity and storage within Azure, Greenplum, and Teradata.
• Develop Unix shell scripts to automate data operations and system maintenance.
• Process real-time data streams using Kafka to support business insights.
• Analyze database performance to identify optimization opportunities.

Required Skills

• 5+ years of experience in data engineering or related technical roles.
• Proficiency in Python for data processing and tool development.
• Hands-on experience with Apache Spark for large-scale data transformation.
• Strong knowledge of Azure cloud services for data architecture and deployment.
• Experience with message streaming platforms, specifically Kafka.
• Familiarity with relational and columnar databases such as Greenplum and Teradata.
• Ability to write and maintain Unix shell scripts for automation tasks.

Preferred Skills

• Any graduate degree.

Key Skills

Education

Any Gradute