Description

Design and build data pipelines on AWS using Databricks, integrating with existing SS environment vendors. Own the full development lifecycle from technical design to testing and deployment.

Responsibilities

• Architect and develop data integration solutions coordinating with analysts and developers.
• Design data pipelines using Databricks clusters, AWS S3, RDS, and EC2.
• Execute unit test automation and prepare technical and functional design documents.
• Migrate data from on-prem Hadoop to Databricks/AWS infrastructure.
• Implement CI/CD pipelines using Jenkins and manage Airflow orchestrations.

Required Skills

• 12+ years of software development experience.
• Hands-on expertise with Databricks, Unity Catalog, and Databricks modules.
• Strong proficiency in Python, PySpark, Scala, and Shell scripts.
• Experience with AWS services: EC2, S3, ELB, Auto Scaling, ECS, RDS.
• Proficiency in HIVE, Bigdata/Hadoop, and Airflow.
• Solid understanding of CI/CD practices and Jenkins.
• Engineering degree in Computer Science or equivalent.
• Ability to work onsite in Boston, MA from day one.

Preferred Skills

• Experience migrating legacy on-prem Hadoop systems to cloud-native Databricks.
• Proven track record in data integration projects.

Key Skills

Education

Not specified