Description

Build and manage heavy data pipelines and large-scale data processing workflows on Google Cloud Platform.

Responsibilities

• Build effective data pipelines using Apache Beam Java SDK and deploy them via GCP DataFlow.
• Develop processes for data transformation, data structures, metadata, dependency, and workload management.
• Perform root cause analysis on internal and external data to identify opportunities for improvement.
• Manage cluster capacity configurations and cloud optimization to meet application demand.

Required Skills

• 8+ years of experience in software development or data engineering.
• Extensive experience with BigQuery, DataProc, and DataFlow on Google Cloud Platform.
• Proficiency in Python, Shell scripting, and PySpark.
• Hands-on experience with Apache Beam Java SDK.
• Advanced SQL knowledge and experience with relational databases.
• Experience implementing CI/CD processes for pipeline deployment in GCP.
• Experience with Docker and deployment tools.
• Knowledge of message queuing, stream processing, and scalable big data stores.

Preferred Skills

• Experience with Azure Databricks.
• Experience in Machine Learning and Data Modeling.

Key Skills

Education

Any Graduate