Description

Design, architect, and implement data-intensive product features and large-scale processing frameworks.

Responsibilities

• Build high-scale data frameworks using Apache Spark and data cataloging tools like Apache Hive and AWS Glue.
• Develop ETL pipelines using Apache Airflow to integrate diverse data sources and sinks.
• Design and architect new product features, serving as a technical lead for tool adoption.
• Contribute code across multiple product domains to support feature engineering for AI modules.
• Translate data requirements into detailed functional and technical designs for development and testing.

Required Skills

• 5+ years of data engineering experience with a Graduate degree, or 7+ years with a Bachelor's degree.
• Proficiency in Java, Scala, or C++.
• Experience with Apache Spark (batch and stream-processing), Kafka, and Hive.
• Hands-on experience with AWS services including Glue, Athena, Lambda, EMR, and RDS.
• Strong SQL and NoSQL skills using MySQL, Postgres, or DynamoDB.
• Experience building ETL pipelines with Apache Airflow and managing metadata.
• Proficiency in Python and Unix/Linux shells.
• Experience designing and consuming RESTful API services.
• Ability to communicate technical risks to both QA and non-technical stakeholders.

Preferred Skills

• Knowledge of healthcare data and insurance claims.
• Experience using Apache Zeppelin or Presto for data validation.

Key Skills

Education

Bachelor's degree in Computer Science