Description

Design and implement scalable data architectures and automated pipelines to support data engineering, science, and business needs.

This role is on-site.

Responsibilities

• Design and document data platform solutions, infrastructure, and reusable pipeline frameworks.
• Build and optimize data pipelines using Spark and Databricks to ingest data into AWS-based Data Lakes and Delta Lakes.
• Develop and maintain Dev/Data/MLOps processes and automation to support platform scenarios.
• Partner with product owners and stakeholders to define requirements and align on data validation approaches.
• Maintain end-to-end data security for data at rest and in transit.

Required Skills

• 5+ years of technical experience with big data technologies.
• Proficiency in Python and development within Notebooks or IDEs.
• Hands-on experience with Apache Spark, Databricks, PySpark, and SparkSQL.
• Experience with Parquet, Delta Lake, and Spark Streaming.
• Practical knowledge of Kafka or Kinesis for data streaming.
• Experience with AWS services including S3, Glue, EC2, EMR, RDS, Redshift, IAM, Security Groups, and VPC.
• Experience using workflow management tools like Airflow or AWS Step Functions.
• Experience with relational, NoSQL, or MPP databases such as Snowflake or Redshift.
• Knowledge of automation and CI/CD tools including Jenkins, Terraform, CDK, and Boto3.

Preferred Skills

• Experience using Databricks with Unity Catalog and MLFlow.

Key Skills

Education

Any Graduate