Description

You will build and manage data pipelines, develop ML/AI solutions, and optimize data infrastructure to support scalable analytics.

Responsibilities

• Design, develop, and maintain complex data pipelines using Databricks and PySpark to process large datasets.
• Develop data models and warehouse architecture to support business intelligence.
• Collaborate with Data Scientists to design, deploy, and optimize ML/AI models and LLMs.
• Manage and optimize data infrastructure components on AWS, specifically S3 and Kinetics.
• Implement automated data quality checks using Databricks.
• Mentor junior team members and collaborate with cross-functional engineering teams.

Required Skills

• 2-5 years of experience in data engineering.
• Advanced proficiency in Databricks and PySpark for ETL and data transformation.
• In-depth experience with AWS services, including S3 and Kinetics.
• Strong programming skills in Python.
• Proficiency in SQL and relational database design.
• Experience with Machine Learning frameworks such as TensorFlow or PyTorch.
• Practical experience in fine-tuning, deploying, and optimizing LLMs.
• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.

Preferred Skills

• Experience with Scala.
• Exposure to real-time data processing frameworks like Apache Kafka.

Key Skills

Education

Bachelor's or Master's degree in Computer Science