Description

You will design and develop data pipelines for ingestion and transformation using Spark within large-scale enterprise deployments.

Responsibilities

• Design and develop data pipelines for data ingestion and transformation using Spark.
• Implement data lake ETL processes using AWS Glue and Databricks.
• Troubleshoot performance bottlenecks and data skew issues in distributed systems.
• Provide recommendations and design optimal configurations for large-scale deployments.
• Manage data integration patterns including streaming, batch, error, and replay.

Required Skills

• 5+ years of experience with Python, PySpark, and Apache Spark.
• 5+ years of experience with AWS capabilities including Glue, DynamoDB, Lambda, Redshift, and Elasticsearch.
• 5+ years of experience building and managing APIs.
• Proficiency in SQL and experience with relational and non-relational databases.
• Strong knowledge of data modeling and query optimization.
• Experience with GitHub, Jenkins, and Terraform.
• Experience with Teradata (Vantage) or other RDBMS and ETL tools.
• Hands-on experience with large-scale distributed relational and NoSQL database systems.
• Background in healthcare data analytics.

Key Skills

Education

Any Graduate