Description

You will design, develop, and maintain data infrastructure and pipelines to support business requirements and system performance.

Responsibilities

• Build and optimize data processing pipelines using Apache Spark, Python, and Scala.
• Develop data ingestion and extraction processes, including Kafka streaming pipelines and batch workflows.
• Containerize data applications with Docker and orchestrate deployments using Kubernetes on NetApp S3.
• Implement performance tuning for Spark jobs and SQL queries to ensure scalability.
• Write and maintain unit tests using pytest or junit and execute BDD scenarios with Cucumber and Lettuce.

Required Skills

• 5+ years of experience designing and building scalable data pipelines.
• Proficiency in Apache Spark, Python, and Scala.
• Experience with Kafka for streaming data pipelines.
• Hands-on experience with Docker and Kubernetes orchestration.
• Knowledge of NetApp S3 or AWS S3 environments.
• Strong understanding of data warehousing, ETL processes, and data modelling.
• Experience with testing frameworks including pytest, junit, Cucumber, or Lettuce.
• Ability to implement data governance, security, and compliance best practices.

Preferred Skills

• Experience with NetApp S3 specifically.

Key Skills

Education

ANY GRADUATE