The Lead Data Engineer is a senior software developer with strong software engineering skills who is responsible for building custom open-source-based data ingestion and MLOps platforms. He/she has deep appreciation of the complexity of the data engineering process, such as the challenges of data ingestion involving large or near-real-time datasets, the maintenance of high data quality, and the importance of automation for increasing pipeline robustness and reducing the need for human intervention.
Key Responsibilities
o Design and develop data engineering services and their ecosystem using distributed databases (relational, columnar, graph, in-memory); orchestration (Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis, Spark).
o Design and develop MLOps production pipelines; provide technical support to data scientists/ML engineers by getting their ML/DL models deployed at scale and meeting SLAs on both cloud and on-premises GPU and CPU instances.
o Design data models for mission-critical, high-volume, near-real-time/batch data; build idempotent/atomic production data pipelines to make data ingestion more fault tolerant.
o Design and develop intuitive, highly automated, self-service data platform functions for business users.
o Design, build, and operate scalable and reliable data pipelines on the Databricks platform.
Requirements
We thank all candidates for your interest in Singapore Airlines, and regret that only shortlisted candidates will be notified.
Sign in to browse authentic reviews, anonymous ratings and salary data before you apply.