Description

Build and maintain complex data pipelines to convert source data into data lake formats using a proprietary big data processing platform.

Responsibilities

• Build data pipelines according to transformation specifications for ingestion into a data lake.
• Develop and maintain engineering processes using T-SQL, Spark, Scala, and shell scripting.
• Review and test data to ensure accuracy and validity before loading.
• Perform data analysis, mining, and root cause investigations to troubleshoot issues.
• Mentor junior team members on data engineering and quality best practices.

Required Skills

• 8-10 years of industry experience with a Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience with big data technologies including Hadoop and Spark.
• 5+ years of experience with RDBMS such as Oracle and MS SQL Server.
• 5+ years of experience with data aggregation, standardization, linking, and quality check mechanisms.
• Proficiency in SQL, T-SQL, and ETL processes.
• Strong knowledge of Scala, Hive, and shell scripting.
• Solid understanding of Linux environments, file systems, and shell scripting.
• Experience working within a Scaled Agile Framework (SAFe) environment.

Preferred Skills

• Healthcare domain knowledge.

Key Skills

Education

Bachelor’s Degree