Description

• Required:
  • Extensive experience with GCP services including BigQuery, Dataflow, Cloud Storage, and Cloud Pub/Sub
  • Strong proficiency in Apache Spark for distributed data processing and analytics
  • Hands-on expertise in building and maintaining data pipelines using ETL/ELT processes
  • Proficiency in Python for data scripting, automation, and orchestration tasks
  • Experience with distributed data storage and management, including relational and NoSQL databases (PostgreSQL, MySQL, MongoDB)
  • Familiarity with version control tools such as Git
  • Knowledge of Linux/Unix environments for data processing and scripting
    
     
• Preferred:
  • Experience with data governance, metadata management, and data security best practices
  • Knowledge of data orchestration tools like Apache Airflow or Prefect
  • Understanding of containerization (Docker) and orchestration (Kubernetes) for data deployment
     

Overall Responsibilities

 

• Design, develop, and maintain scalable, resilient data pipelines on GCP to support enterprise analytics and reporting solutions
• Collaborate with business stakeholders, data scientists, and analytics teams to understand data requirements and implement optimized data workflows
• Implement and enforce data quality, security, and governance standards across platforms
• Optimize data ingestion, transformation, and processing workflows to ensure high performance and cost efficiency
• Perform data profiling, troubleshooting, and resolution of pipeline issues to ensure operational reliability
• Stay current with emerging data engineering best practices, tools, and industry trends, leading continuous improvement initiatives
  
   

Technical Skills (By Category)

 

• Programming Languages: Required: Python for scripting and orchestration Preferred: SQL, Java (for integration or data processing tasks)
• Databases & Data Management: BigQuery, PostgreSQL, MySQL, MongoDB, data modeling, data security, and query optimization
• Cloud Technologies: GCP services including BigQuery, Dataflow, Cloud Storage, Pub/Sub, IAM, and Cloud Function deployment
• Frameworks & Libraries: Apache Spark, Dataflow, Airflow (preferred), TensorFlow or PyTorch (if ML integrations are involved)
• Development & Orchestration Tools: Git, Jenkins, Docker, Kubernetes, Terraform, Apache Airflow, and other CI/CD tools
• Security & Compliance: Implementing security policies, data encryption, access controls, and compliance standards such as GDPR or HIPAA
  
   

Experience Requirements

 

• 5+ years of practical experience in data engineering, with a significant focus on GCP and Big Data ecosystems
• Hands-on experience designing and implementing end-to-end data pipelines and workflows at scale
• Proven expertise in distributed data processing, data security, and infrastructure automation
• Experience working in agile teams supporting enterprise or large-scale data platforms
• Industry experience in finance, healthcare, retail, or technology sectors is advantageous

Key Skills

Education

Bachelor's or Master's degrees