Description

Lead the architectural design and implementation of large-scale data engineering and analytics solutions.

Responsibilities

• Define ETL architectural rules and supervise implementation across projects.
• Design technology system structures and break down large systems into manageable components.
• Produce technical architecture documentation for small to large solution implementations.
• Coordinate with IT teams to translate operational needs into structured technical requirements.
• Select suitable software and hardware and suggest integration methods to resolve technical problems.

Required Skills

• 15+ years of IT experience in Data Engineering, Data Quality, Migration, Architecture, and Data Lake formation.
• 5+ years of hands-on experience with AWS services including S3, EMR, VPC, EC2, IAM, EBS, RDS, Glue, Lambda, and Lake Formation.
• In-depth knowledge of Spark Architecture, including Spark Code, Spark SQL, Dataframes, Spark Streaming, and Spark MLlib.
• Experience handling high-volume streaming data in JSON, XML, AVRO, and Snappy formats.
• Proficiency with Kafka for capacity planning, partition planning, and read/write design.
• Experience with Big Data tools, MapReduce, and Spark.
• Working knowledge of RDBMS, NoSQL (Columnar, Document, Distributed), Cloud, and In-memory databases.
• Ability to communicate system requirements and structures to software designers and developers.

Preferred Skills

• Python exposure.

Key Skills

Education

Any Gradute