Description

You will design and oversee complex data architectures, engineering pipelines, and analytics infrastructure.

Responsibilities

• Design and implement real-time data processing pipelines for streaming analytics.
• Configure data infrastructure, including pipeline integration and performance tuning.
• Manage data ingestion, streaming, and troubleshooting within production environments.
• Collaborate with cross-functional teams and stakeholders to resolve complex engineering challenges.

Required Skills

• 7+ years of experience in Data Architecture, data engineering, and analytics.
• 5+ years of consulting experience working with external clients.
• Databricks Certified Machine Learning Professional certification (required).
• Expertise in Databricks components: Unity Catalog, Delta Lake, Delta Live Tables, and Apache Spark.
• In-depth knowledge of AWS services: Redshift, Glue, S3, AuroraDB, MWAA, Lambda, and SageMaker.
• Advanced SQL skills and experience with ETL testing.
• Proficiency in Python, R, Spark, or Scala, including NumPy and pandas.
• Hands-on experience with streaming technologies like Apache Kafka or Apache Flink.
• Experience with Terraform, Git, CI/CD, and automation/integration testing.

Preferred Skills

• Experience in the healthcare insurance domain.
• Experience with dbt and other related data technologies.

Key Skills

Education

Bachelor's degree