Description

You will design and manage end-to-end data solutions and processing pipelines for large-scale, varied datasets.

Responsibilities

• Build and manage optimal data processing pipelines and architectures for large volumes of data.
• Develop efficient pre-processing and data manipulation tasks using PySpark and Pandas.
• Create RESTful web services and API platforms.
• Implement machine learning solutions and production pipelines in collaboration with data science and infrastructure teams.
• Design and maintain event-driven architectures using stream-processing systems.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python with strong knowledge of OOP and design patterns.
• Hands-on experience with PySpark, Dataframes, and Pandas Dataframes.
• Experience with SQL and NoSQL databases such as Postgres, MongoDB, or Elasticsearch.
• Working knowledge of stream-processing systems like Spark-Streaming and Kafka.
• Experience with event-driven architectures.
• Ability to build RESTful web services.
• Any Graduate degree.

Key Skills

Education

Any Graduate