Description

You will provide end-to-end support for CDL (Common Data Layer) platforms, ensuring stability and high availability.

Responsibilities

• Troubleshoot and resolve issues across Linux systems, Hadoop clusters, Spark jobs, and cloud data services.
• Manage and support Hadoop ecosystem components like HDFS, YAARN, MapReduce, and cluster services.
• Develop and maintain ETL pipelines using Python, PySpark, and Shell scripting.
• Configure and tune large-scale Spark jobs for efficient resource allocation.
• Contribute to the design and implementation of scalable, Hadoop-based processing frameworks.

Required Skills

• 5+ years of experience in data engineering or related infrastructure roles.
• Strong hands-on expertise with Linux systems and advanced Shell scripting.
• Proficiency with Hadoop architecture, including HDFS, YAARN, HA, and MapReduce.
• Expertise in distributed data processing using Spark and PySpark.
• Solid experience with GCP services, specifically BigQuery and DataProc.
• Strong background in AWS big-data ecosystem components (EMR, Glue, S3, Lambda, Athena).
• Experience building and optimizing ETL pipelines using Python and PySpark.
• Working knowledge of Hive and general SQL querying.

Key Skills

Education

Any Gradute