Description

Lead the architecture and development of backend data ingestion and integration pipelines.

Responsibilities

• Manage the full development lifecycle of data pipelines, including architecture, design, testing, and deployment.
• Design and maintain near real-time ingestion pipelines from primary and third-party sources.
• Build proof of concepts for new data sources to demonstrate capabilities to analytics teams.
• Implement automated quality checks to identify outliers, missing data, and ensure accuracy.
• Architect scalable solutions supporting both experimental work and mission-critical production operations.

Required Skills

• 10+ years of experience building enterprise-level software solutions.
• 4+ years of experience architecting cloud-based software solutions.
• 4+ years of experience in API-based development using Python and Java.
• Extensive experience building secure data pipelines using Python and Spark on AWS.
• Proficiency with AWS services including Kinesis, EMR, Redshift, S3, and Lambda.
• Experience with Spark-Streaming for real-time data processing.
• Hands-on experience with SQL databases including Redshift, BigQuery, SQL Server, or Oracle.
• Experience processing structured and unstructured data, including complex JSONs and social media platform data.
• Knowledge of code versioning tools such as Git, Mercurial, or SVN.

Preferred Skills

• Experience with Python libraries including Pandas, NumPy, SciPy, Flask, and SQLAlchemy.
• Familiarity with Sailthru APIs.

Key Skills

Education

Any Graduate