Description

Architect and implement enterprise-scale data solutions, focusing on large-scale big data application deployment and system integration.

Responsibilities

• Lead enterprise-wide data migration, transformation, and the build of data warehouses, marts, and lakes.
• Design complex technical solutions for data integration and large-scale application deployment.
• Architect streaming data pipelines and manage the transition between batch and streaming systems.
• Optimize data processing through effective indexing, partitioning, and schema evolution strategies.
• Brief stakeholders and senior management on the technical benefits and constraints of proposed solutions.

Required Skills

• 6+ years of IT experience in enterprise data architecture and management.
• Expertise in AWS, including Redshift, infrastructure, Data Lake Formation, Glue components, S3, and Lambda.
• Advanced SQL skills covering joins, aggregations, windowing functions, CTEs, and RDBMS schema design.
• Proficiency in Python, Spark Scala, and Java programming.
• Deep knowledge of Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Experience with Conceptual, Logical, and Physical Data Modelling, including Relational and Dimensional models.
• Strong understanding of ETL/ELT processes and tools like AWS Data Migration Service.
• Practical experience with data lake concepts such as time travel, schema evolution, and optimization.
• Competency in managing CI/CD pipelines using GitLab and monitoring via CloudWatch.
• Knowledge of data security, Schema Registry, and message formats like Avro and ORC.

Preferred Skills

• Experience with Great Expectations or similar data quality and validation frameworks.
• Familiarity with implementing incremental data loads using tumbling windows, sliding windows, or high watermarks.

Key Skills

Education

Any Graduate