Description

You will build and maintain scalable data pipelines and processing systems using Python, PySpark, and Databricks.

This role is on-site.

Responsibilities

• Develop and optimize large-scale data pipelines using Python, PySpark, and Apache Spark.
• Manage data processing and storage using Databricks and Delta Lake.
• Design and maintain SQL and NoSQL database integrations, including PostgreSQL, MongoDB, and Redis.
• Implement real-time streaming solutions with Kafka and optimize data partitioning strategies.
• Support cloud deployments on Azure and AWS, including CI/CD pipelines and production monitoring.

Required Skills

• 5+ years of experience in Data Engineering.
• Proficiency in Python and PySpark for data processing.
• Hands-on experience with Apache Spark and Databricks.
• Strong knowledge of Delta Lake for data management.
• Experience with SQL and NoSQL databases (PostgreSQL, MongoDB, Redis).
• Familiarity with Kafka for real-time data streaming.
• Understanding of cloud platforms (Azure, AWS) and CI/CD practices.

Preferred Skills

• Experience with GenAI, LLM workflows, and prompt engineering.
• Knowledge of system design and REST API integration.

Key Skills

Education

Any Gradute