Description

Job Description:

Data Pipeline Development

• Design and build robust ETL/ELT pipelines for batch and stream processing using GCP tools.

Data Warehousing & Modeling

• Build and optimize data models, data lakes, and data warehouses, primarily using BigQuery for analytical workloads.

Workflow Orchestration

• Automate, schedule, and monitor data workflows using orchestration frameworks.

Pipeline Optimization

• Enhance system performance and manage resource allocation for cost-efficiency.

Data Governance & Security

• Enforce data quality frameworks, lineage tracking, security protocols, and compliance standards.

Cross-Functional Collaboration

• Partner with DevOps, Software Engineering, and Business Intelligence teams to deploy data solutions into production environments.

Essential Qualifications & Skills

Cloud Services

• Proficiency with GCP ecosystem services such as:
  • BigQuery
  • Cloud Dataflow
  • Cloud Storage
  • Cloud Composer
  • Apache Airflow
  • Dataform/dbt
  • Pub/Sub

Programming

• Advanced expertise in SQL and Python/Java.

Big Data Frameworks

• Hands-on experience with distributed computing and processing frameworks, e.g., Apache Spark, PySpark, or Apache Beam.

Data Engineering Concepts

• Solid understanding of:
  • Data Warehousing
  • Star/Snowflake Schemas
  • Dimensional Modeling
  • Change Data Capture (CDC)

DevOps Practices

• Familiarity with:
  • CI/CD pipelines
  • Git version control
  • Infrastructure as Code (e.g., Terraform)

Key Skills

Education

Any Graduate