Description

You design, implement, and optimize scalable data processing solutions for attribution and forecasting engines.

Responsibilities

• Design and maintain fault-tolerant data pipelines with end-to-end monitoring and alerting.
• Build, schedule, and manage DAG workflows and infrastructure modules using Python.
• Optimize Spark job performance by tuning resource usage, shuffling, partitioning, and caching.
• Collaborate with stakeholders to define software objectives, business requirements, and technical specifications.
• Mentor junior staff and document technical processes and system architectures.

Required Skills

• 5-8 years of experience in Scala programming and Apache Spark for Big Data processing.
• In-depth knowledge of the Hadoop ecosystem, including HDFS, YARN, and MapReduce.
• Hands-on experience with Python for developing infrastructure modules.
• Proficiency in SQL and database systems (RDBMS/Warehouse) to handle terabytes of data.
• Experience deploying big data solutions on AWS and GCP cloud environments.
• Strong understanding of data modeling, partitioning strategies, and sharding for distributed systems.
• Experience with Git and unit/integration testing frameworks.
• Solid grasp of Agile/SCRUM methodologies and ability to work across multiple time zones.

Preferred Skills

• Hands-on experience with Databricks, Delta Lake, and the ELK stack.
• Experience with Docker, Kubernetes, Tableau, Power BI, or Grafana.

Key Skills

Education

Any Graduate