Description

Lead the architecture and development of large-scale, distributed data processing pipelines and enterprise-grade software in Waltham, MA, USA (Remote).

Responsibilities

• Architect distributed data processing pipelines using Apache Spark, Apache Beam, and Apache Airflow.
• Design efficient ingestion, transformation, and storage solutions for structured and unstructured data.
• Build and optimize real-time and batch processing systems for high availability and scalability.
• Mentor junior engineers and participate in technical decision-making and code reviews.
• Implement best practices for data governance, quality, and security across the data lifecycle.

Required Skills

• 10+ years of experience in backend software development with a focus on data engineering.
• Expertise in Apache Spark, Apache Beam, and Apache Airflow.
• Proficiency in Java, Scala, and SQL.
• Experience building software in cloud-native environments (GCP or AWS) using services like GCS/S3, Dataflow/Glue, Dataproc/EMR, Cloud Function/Lambda, BigQuery/Athena, and BigTable/Dynamo.
• Hands-on experience with Kafka, HBase, and containerization tools like Docker and Kubernetes.
• Experience with SQL performance optimization.
• Knowledge of data modeling, warehousing, and integration best practices.
• Experience designing CI/CD pipelines using Jenkins or GitHub Actions.
• Background in Graph and Vector databases or processing frameworks.

Preferred Skills

• Experience with machine learning pipelines and real-time analytics.
• Familiarity with streaming data processing architectures.

Key Skills

Education

ANY GRADUATE