Description

You will design and implement scalable data platforms, pipelines, and storage systems within the Azure ecosystem.

Responsibilities

• Partner with product owners to document business requirements and produce solution designs, high-level data flow diagrams, and data standards.
• Design, develop, and implement scalable ETL processes, data pipelines, and workflows to collect and process data from various sources.
• Build and maintain data warehouses, data lakes, and storage systems to ensure data accessibility and performance.
• Develop and optimize database schemas, data models, and queries to support efficient retrieval and analysis.
• Implement data quality assurance processes, including validation, cleansing, and error handling, while monitoring systems for high availability.

Required Skills

• 8+ years of experience with data engineering tools including Spark, Hadoop, Azure Data Factory, Azure Data Lake Storage, Databricks, and Delta Lake.
• Proficiency in Python and SQLServer.
• Experience designing and implementing solutions on Hadoop, SQLServer, Oracle, or Snowflake.
• Demonstrated ability to build and tune data pipelines using Spark and cloud-native tools like EMR or Azure Data Factory.
• Experience with ETL tools such as Informatica or Ab Initio.
• Knowledge of both batch and stream ingestion from RDBMS sources like SQLServer, Oracle, and MySQL.
• Experience with other cloud providers including AWS and GCP.

Preferred Skills

• Experience within the insurance domain.

Key Skills

Education

Any Graduate