Description

You will design, build, and maintain scalable data pipelines and data processing systems.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Spark and PySpark within the Databricks ecosystem.
• Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data needs and deliver data solutions.
• Implement and manage DataOps practices to streamline data development and deployment processes.
• Ensure data quality and governance through rigorous testing, validation, and monitoring of data pipelines.
• Develop ETL/ELT workflows and scripts using Python and SQL.

Required Skills

• Minimum 3 years of experience in a Data Engineering role.
• Strong expertise in Apache Spark and PySpark.
• Solid proficiency in Python programming for data processing tasks.
• Advanced SQL skills for writing and optimizing complex queries.
• Practical knowledge of DataOps principles, tools, and automation.
• Experience with version control tools (e.g., Git) and CI/CD pipelines.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Experience with ETL/ELT concepts.
• Familiarity with Databricks.

Key Skills

Education

Any Graduate