Description

You will build and manage an enterprise data lake and cloud-based processing pipelines to support data science and business intelligence.

Responsibilities

• Build and manage automated data pipelines to capture IoT, healthcare, and third-party data.
• Develop cloud-based processing pipelines using Spark, SQL, Python, AWS Lambda, and REST APIs.
• Create data models for different classification layers used in analytics.
• Collaborate with application developers to prioritize and deploy data solutions.
• Execute verification testing to ensure data accuracy and quality.

Required Skills

• 5+ years of experience as a data engineer.
• Extensive experience with SQL, Python, and Apache Spark/PySpark.
• Proficiency with AWS services including S3, IAM, Lambda, Secrets Manager, and Event Bridge.
• Experience building data integrations via REST APIs, JDBC databases, and cloud storage buckets.
• Hands-on experience with Terraform for deploying cloud infrastructure and pipelines.
• Experience building stream processing pipelines using Kafka and Structured Spark.
• Knowledge of data modeling and modern data architecture.
• Experience with cloud-based platforms such as Databricks or Snowflake.
• Solid foundations in software engineering and quality engineering practices.

Preferred Skills

• Experience working with IoT or healthcare data.
• Experience operating within an onsite/offshore model.

Key Skills

Education

Any Graduate