Description

Job Description

Experience level: 10+ years hands-on data engineering with production pipeline ownership; at least one platform or system owned end to end.

Data transformation and modeling

• dbt (data build tool): models, tests, macros, incremental models, project structure

• Data modeling: dimensional modeling, star schema, medallion architecture (raw / curated /

presentation, or bronze / silver / gold)

• Data warehousing and lakehouse architecture concepts

Orchestration

• Apache Airflow: DAG design, task dependencies, triggers, sensors, parameterized runs, failure

handling, scheduling

• Workflow orchestration and monitoring of production pipelines

Data ingestion and loading

• Batch data ingestion and ETL / ELT pipeline development

• Load patterns: full load, incremental load, upsert / merge, change data capture (CDC) 

• Multi-source ingestion: SFTP, REST APIs, relational databases, file-based ingestion

• Data quality, data validation, schema evolution handling

Query engine, storage, and table formats

• Dremio (or equivalent lakehouse query engine: Trino, Presto, Starburst, Athena)

• Apache Iceberg (or equivalent open table format: Delta Lake, Apache Hudi) 

• S3 / S3-compatible object storage (AWS S3, MinIO, Dell ECS)

• Partitioning, table layout, and query performance tuning

Programming and databases

• Python (pipeline development, automation, scripting)

• Advanced SQL, query optimization, performance tuning 

• PostgreSQL and/or SQL Server (relational databases)

Non-relational / NoSQL databases (must-have, not just exposure)

The platform ingests from and operates alongside several non-relational stores. The candidate must be able to work with them as a data engineer, meaning read from them, model their data into the lakehouse, and reason about their consistency and performance characteristics. Screen for hands-on experience with at least two of the following families:

• Key-value / in-memory stores: Redis (caching, queues, state; TTL and eviction behaviour; using

Redis as a pipeline coordination or lookup layer)

• Document databases: MongoDB, Couchbase, or DocumentDB (nested JSON documents,

schema-on-read, flattening documents into relational / lakehouse tables)

• Wide-column stores: Cassandra, ScyllaDB, or HBase (partition-key design, read patterns,

extracting at scale)

• Search / analytics engines: Elasticsearch or OpenSearch (index design, querying, extracting data

out of an index for analytics)

Data formats and semi-structured data

• Columnar formats: Apache Parquet (required), ORC; row formats: Avro

• Text and semi-structured formats: JSON, JSON Lines, CSV / delimited files, XML; encoding,

delimiter, and quoting edge cases

• Compression codecs (Snappy, gzip, zstd) and their effect on storage and query

• Handling nested and evolving schemas end to end (schema inference, drift detection, schema

evolution: column add / delete / type change) 

Distributed processing

• Apache Spark / PySpark (DataFrame API, partitioning, joins, skew handling, writing Iceberg /

Parquet); or equivalent distributed engine

• Understanding of parallelism, shuffles, memory tuning, and cost of distributed jobs

Scripting, Linux, and connectivity

• Bash / shell scripting and Linux command line (the deployment and operational tooling is script-

based)

• SFTP / SSH: key management, automated transfers, file-based handoffs (verify / result files,

archiving, retries)

• TLS certificates and secure connections (understanding certificate verification; never disabling it in

production)

• Connectivity troubleshooting: DNS, firewalls, proxies, connection timeouts, service-to-service

networking inside Kubernetes / OpenShift

• YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests

Key Skills

Education

Any Graduate