Description

You will design, develop, and manage advanced data pipelines and infrastructure solutions to solve complex business problems.

Responsibilities

• Design and deploy scalable, secure data pipelines and algorithms using AWS cloud services.
• Write and optimize SQL queries for efficient data extraction, transformation, and loading.
• Process large datasets using Apache Spark and Flink.
• Implement real-time data processing solutions using Kafka.
• Own the end-to-end data science workflow, including collection, processing, and deployment.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in AWS cloud services including S3, EC2, Redshift, and Lambda.
• Strong hands-on experience with SQL for data manipulation and analysis.
• Expertise in Python and data science libraries such as Pandas, NumPy, and Scikit-learn.
• Experience with Apache Spark and Apache Flink for big data processing.
• Hands-on experience with Kafka for real-time messaging and data processing.
• Proficiency in PySpark.
• Experience using modern debugging tools like DebuggItJ.
• Any Graduate degree.

Preferred Skills

• Knowledge of Solr for enterprise search and indexing.
• Familiarity with DevOps tools including Docker, Kubernetes, and CI/CD pipelines.

Key Skills

Education

Any Graduate