Description

Key Responsibilities

• Design, develop, and maintain scalable ETL and data-processing pipelines.
• Build solutions for processing real-time and high-volume data.
• Set up and enhance infrastructure supporting existing data pipelines.
• Develop and integrate APIs for data exchange and downstream consumption.
• Write efficient Python code using Asyncio for asynchronous and concurrent processing.
• Use PySpark to perform distributed data processing.
• Build and maintain data-integration workflows using AWS Glue.
• Load, transform, and manage data within Snowflake.
• Write and optimize SQL queries for data transformation and analysis.
• Monitor pipeline performance, troubleshoot failures, and improve reliability.
• Collaborate with engineering, analytics, and business teams to understand data requirements.
• Ensure data quality, accuracy, security, and scalability across pipelines.

Required Qualifications

• Strong professional experience in Python development.
• Hands-on experience with PySpark and Apache Spark.
• Experience using Python Asyncio or similar asynchronous programming frameworks.
• Strong knowledge of ETL pipeline development and data integration.
• Experience building or integrating REST APIs.
• Hands-on experience with Snowflake.
• Experience developing data workflows with AWS Glue.
• Strong SQL skills, including complex queries and performance optimization.
• Experience working with real-time or near-real-time data pipelines.
• Experience configuring infrastructure for existing data-processing environments.
• Strong troubleshooting and problem-solving abilities.

Top Skills

• Python
• PySpark
• Apache Spark
• Asyncio
• Snowflake
• AWS Glue
• SQL
• ETL Pipelines
• API Development
• Real-Time Data Processing

Key Skills

Education

Bachelor's degree