Description

Architect and build scalable data integration solutions on AWS using Databricks, owning the full development lifecycle from technical design through testing and deployment.

This role is on-site.

Responsibilities

• Design and develop data pipelines using Databricks clusters, AWS S3, RDS, and EC2.
• Migrate data from on-prem Hadoop environments to cloud-native Databricks and AWS infrastructure.
• Implement CI/CD pipelines using Jenkins and manage workflow orchestrations with Airflow.
• Execute unit test automation and produce technical and functional design documents.
• Coordinate with analysts and developers to integrate with existing SS environment vendors.

Required Skills

• 12+ years of software development experience with an Engineering degree in Computer Science or equivalent.
• Hands-on expertise with Databricks, Unity Catalog, and Databricks modules.
• Strong proficiency in Python, PySpark, Scala, and Shell scripts.
• Deep experience with AWS services: EC2, S3, ELB, Auto Scaling, ECS, and RDS.
• Proficiency in HIVE, Bigdata/Hadoop, and Airflow.
• Solid understanding of CI/CD practices and Jenkins.
• Ability to work onsite in Boston, MA from day one.

Preferred Skills

• Experience migrating legacy on-prem Hadoop systems to cloud-native Databricks.
• Proven track record in complex data integration projects.

Key Skills

Education

Not specified