Description

Build and manage large-scale data processing pipelines and cloud-based data architectures.

Responsibilities

• Develop data pipelines using Spark, Kafka, and Hadoop.
• Implement data services and workflows using GCP tools including BigQuery, Dataflow, and Cloud Composer.
• Manage data governance, security, and compliance using Dataplex on GCP.
• Execute data quality assurance and testing protocols within GCP environments.
• Automate data processes through scripting and programming.

Required Skills

• 10+ years of experience in data engineering or big data development.
• Proficiency with Hadoop and Spark.
• Strong programming skills in Python (NumPy, Pandas, PySpark), Java (Core Java, functional interfaces, lambdas, collections), or Scala.
• Experience with Kafka for data streaming.
• Hands-on experience with AWS.
• Expertise in GCP data services: BigQuery, Dataflow, Data Fusion, Dataproc, Cloud Composer, Pub/Sub, and Google Cloud Storage.
• Bachelor's degree in Computer Science.

Preferred Skills

• Experience with containerization and orchestration using Docker and Kubernetes.

Key Skills

Education

Bachelor's degree in Computer Science