Description

Data Pipeline Development, Enhancement & Maintenance • The Data Engineer will be responsible for designing, building, governing, and optimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations. • Design, build, and maintain scalable ETL/ELT pipelines. • Automate data ingestion from multiple data sources. • Ensure reliable, high-performance, and scalable data movement. • Develop and optimize PySpark-based data processing frameworks. • Build and manage Airflow workflows and orchestration pipelines. • Work with Hive and Kafka-based batch and streaming data solutions. • Implement data quality, monitoring, governance, and operational controls. • Support production incidents, performance tuning, and continuous improvement initiatives. Required Skills Cloud Platform • Google Cloud Platform (GCP) • Strong experience with large-scale GCP Data Engineering solutions Data Processing • PySpark • Experience building and optimizing Spark/PySpark data pipelines Workflow Orchestration • Airflow • Hands-on experience with Airflow DAG development and support • Airflow workflow and orchestration pipeline development Data Technologies • Hive • Exposure to Hive-based data processing Streaming & Messaging • Kafka • Kafka-driven ingestion frameworks • Kafka-based batch and streaming data solutions Data Engineering • ETL/ELT pipeline development • Data ingestion automation • Data platform design and optimization • Data integration • Data quality • Data monitoring • Data governance • Operational controls Operations & Performance • Production incident support • Performance tuning • Continuous improvement initiatives Preferred Experience • Experience supporting analytics platforms • Experience supporting personalization platforms • Experience supporting recommendation platforms • Experience supporting search platforms

Key Skills

Education

Bachelor's degree