Description

Design, build, and orchestrate data pipelines and distributed computing applications.

Responsibilities

• Build and orchestrate data pipelines and ETL processes.
• Develop distributed computing applications using PySpark.
• Design and implement data models including normalization, denormalization, and schema design.
• Write, maintain, and execute automated unit tests using Python following TDD practices.
• Build APIs and serverless architectures using AWS services.

Required Skills

• 4+ years of experience in big data engineering.
• Proficiency in Python and libraries such as Pandas, Polars, and NumPy.
• Strong expertise in SQL, Presto, Hive, and Spark.
• Extensive experience with AWS services including EMR, Lambda, Glue ETL, Step Functions, S3, ECS, Kinesis, IAM, RDS PostgreSQL, DynamoDB, CloudWatch Events/EventBridge, Athena, SNS, SQS, and VPC.
• Experience with relational and NoSQL databases, including Amazon Redshift and time series databases.
• Practical knowledge of trading and investment data.
• Experience with One Tick or KDB.
• Understanding of CI/CD, source control, and data warehousing concepts.

Preferred Skills

• Bachelor's degree in Computer Science.

Key Skills

Education

Bachelor's degree in Computer Science