Description

You will design, develop, and optimize large-scale data pipelines using PySpark for structured, semi-structured, and unstructured data, while leading the build of ML pipelines for training and deployment.

This role is on-site.

Responsibilities

• Design and optimize large-scale data pipelines using PySpark for batch and streaming workflows.
• Lead the building of ML pipelines for model training, validation, and deployment in streaming/batch modes.
• Write high-quality, efficient Python code for data transformation, cleaning, and feature engineering using Pandas and NumPy.
• Collaborate with data scientists and stakeholders to define data requirements and deliver actionable insights.
• Monitor pipeline performance, troubleshoot issues, and implement improvements for robustness and scalability.

Required Skills

• 7+ years of experience in data engineering, analytics, or data science roles with significant hands-on big data processing.
• Expert-level proficiency in Python programming, emphasizing clean, maintainable, and scalable code.
• Deep knowledge of PySpark, including Spark SQL, DataFrames, and Spark MLlib.
• Experience with data storage solutions such as SQL (MySQL, SQL Server) and NoSQL (Cassandra, MongoDB, Hive).
• Hands-on experience with ETL tools like Apache Airflow, Jenkins, or GitHub Actions for scheduling and automation.
• Proficiency with cloud environments, specifically Azure Data Factory, Azure Synapse, AWS Glue, or S3.
• Bachelor’s or Master’s degree in Computer Science, Data Science, Mathematics, or a related field.

Preferred Skills

• Experience with containerization and orchestration (Docker, Kubernetes).
• Exposure to distributed storage solutions like Hadoop HDFS or Azure Data Lake.
• Previous leadership or mentorship experience in a technical team.

Key Skills

Education

Bachelor's or Master's degrees