Description

You will design, develop, and optimize data generation pipelines as a distributed systems specialist.

Responsibilities

• Design and optimize data processing pipelines using DAGs, graphs, and distributed workers.
• Implement solutions to handle large fan-out loads and ensure system scalability.
• Collaborate with data scientists and engineers to integrate and scale system solutions.
• Identify and implement optimizations to improve performance and reduce resource utilization.

Required Skills

• 5+ years of software development experience with a focus on distributed systems.
• Experience running compute clusters or building data infrastructure.
• Proficiency in Python, Rust, C++, or Hack (PHP).
• Experience with cloud-based infrastructure such as AWS or GCP.
• Hands-on experience with large fan-out loads and distributed workers/jobs.
• Bachelor's degree in a technical field.

Preferred Skills

• Experience with companies operating at large data scales.
• Experience with machine learning frameworks like TensorFlow or PyTorch.
• Experience with Digital Content Creation (DCC) tools such as Blender.

Key Skills

Education

Not specified