Description

Job Summary   We are seeking an experienced Data Platform Engineer to architect, build, and operate our cloud-based data lakehouse platform. The role is responsible for developing scalable   data infrastructure and processing pipelines built on open table formats, distributed query and compute engines, and containerized cloud infrastructure — enabling analysts, data   scientists, and downstream applications to access reliable, high-performance data at scale. The ideal candidate combines deep distributed-systems and data engineering expertise with   strong software engineering practices and a passion for building robust, efficient platforms.    

Key Responsibilities  

Lakehouse Architecture (Apache Iceberg)  

• Architect, implement, and maintain the data lakehouse using Apache Iceberg table formats.  
• Manage table lifecycle including partitioning, schema evolution, snapshot and metadata management, and compaction.  
• Optimize table layout and file sizing for query performance, storage efficiency, and cost.  
• Implement data governance, time-travel, and ACID transaction patterns across the lakehouse.     

Distributed Query & Compute (Trino & Spark)  

• Deploy, tune, and operate Trino clusters for interactive and federated analytics; manage connectors, catalogs, and resource groups.  
• Build and optimize large-scale batch and streaming data processing with Apache Spark.  
• Troubleshoot query plans, job performance, spills, and shuffle bottlenecks across distributed workloads.  
• Balance capacity, concurrency, and cost across query and compute engines.     

Data Engineering (Scala & Python)  

• Develop scalable data pipelines, frameworks, and services in Scala and Python.  
• Build reusable ingestion, transformation, and data-quality libraries for batch and streaming.  
• Write clean, well-tested, performant code following software engineering best practices.  
• Integrate pipelines with the Iceberg lakehouse and Trino/Spark processing layers.     

Cloud & Kubernetes Infrastructure  

• Design and operate data platform components on Kubernetes (EKS, AKS, GKE, or self-managed).  
• Build and manage containerized workloads, Helm charts, and autoscaling for data services.  
• Provision and manage cloud infrastructure (AWS, Azure, or Google Cloud Platform), including object storage and compute.  
• Implement infrastructure as code, CI/CD, monitoring, and observability for the platform.     

Platform Reliability & Collaboration  

• Ensure the reliability, scalability, and security of the data platform across environments.  
• Automate deployment, testing, and operational workflows to eliminate toil.  
• Partner with data consumers to understand requirements and improve data accessibility.  
• Mentor engineers and promote best practices in data and platform engineering

Key Skills

Education

Any Graduate