Description

You will design, develop, and manage advanced data pipelines and infrastructure solutions to solve complex business problems.

Responsibilities

• Design, develop, and deploy scalable data pipelines and algorithms using AWS cloud services.
• Write, optimize, and maintain SQL queries for data extraction, transformation, and loading.
• Process and analyze large datasets efficiently using Apache Spark and Flink.
• Implement real-time data processing solutions using Kafka.
• Own the end-to-end data science workflow, including collection, processing, and deployment.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in AWS cloud services including S3, EC2, Redshift, and Lambda.
• Strong hands-on experience with SQL for data manipulation and analysis.
• Expertise in Python and data science libraries including Pandas, NumPy, and Scikit-learn.
• Proficiency in Apache Spark and PySpark for big data processing.
• Experience with Apache Flink.
• Experience with Kafka for real-time data processing and messaging.
• Experience using modern debugging tools like DebuggItJ.

Preferred Skills

• Knowledge of Solr for enterprise search and indexing.
• Familiarity with DevOps practices, Docker, Kubernetes, and CI/CD pipelines.

Key Skills

Education

Any Graduate