Description

You will build and maintain near-real-time and batch data pipelines.

This role is on-site.

Responsibilities

• Build and maintain data pipelines using Python, SQL, and Kafka for near-real-time and batch processing.
• Ingest and process data from APIs, message queues, databases, and file systems (JSON, CSV, XML).
• Optimize Python code and SQL queries for high performance.
• Manage deployments and CI/CD using Git, Jenkins, and Agile practices.
• Troubleshoot and resolve production data issues in a Linux environment.

Required Skills

• 5+ years of experience in data engineering.
• Strong SQL skills with MS SQL, MySQL, Hive, Impala, or Spark SQL.
• Experience with Python, including object-oriented programming.
• Experience with Kafka for streaming data.
• Experience with distributed data systems such as Hadoop, Spark, and Hive.
• Proficiency with data formats including JSON, CSV, and XML.
• Experience working within an Agile development process.
• Ability to work with REST APIs and relational databases.

Key Skills

Education

Any Graduate