Data engineer

На данный момент Senior Software Engineer.

Мои компетенции и опыт

Ethan Lim

Singapore | нужен доступ к резюме | нужен доступ к резюме | нужен доступ к резюме

Summary

Senior Software Engineer with over 10 years of experience designing and optimizing ETL/ELT pipelines and data warehouses. Skilled in SQL, Python, Greenplum, ClickHouse, DBT, and Airflow for orchestrating data pipelines, ensuring high reliability and data quality. Experienced in building scalable DWH solutions and automating workflows for analytics and BI tasks, handling millions of records daily. Proven track record of improving data throughput, reducing latency, and enabling real-time analytics.

Experience

Rescode | Senior Software Engineer May 2022 ~ Present

• Developed and optimized ETL/ELT pipelines in Python and SQL, processing millions of records daily
• Designed and maintained data warehouses on Greenplum and ClickHouse, improving query performance by 40%
• Built and configured Airflow DAGs to orchestrate data workflows, reducing pipeline errors by 35%
• Implemented DBT models for data transformation, improving data quality and maintainability
• Automated monitoring and validation of ETL processes, reducing failed job incidents by 50%
• Refined data schemas and indexing strategies, improving throughput for analytics tasks
• Led architecture improvements for scalable DWH solutions supporting multiple analytics teams
• Optimized Python scripts for large-scale data processing, reducing runtime by 30%
• Ensured secure data handling with access control and compliance with GDPR
• Mentored junior engineers in ETL/ELT design and Airflow orchestration
• Collaborated with product and analytics teams to meet BI requirements on time
• Implemented automated reporting pipelines, enabling real-time analytics for business decisions
• Conducted root-cause analysis for pipeline failures and applied preventive measures
• Developed CI/CD pipelines for automated deployment of ETL scripts and DBT models
• Maintained documentation for data workflows, pipelines, and transformations
• Performed data profiling and validation to ensure accuracy for reporting and analytics
• Monitored system performance, reduced latency, and improved reliability of data services
• Delivered measurable improvements in pipeline efficiency, throughput, and data quality

Doodle Labs | Senior Full Stack Engineer Apr 2019 ~ Apr 2022

• Designed and implemented ETL processes for financial and analytical datasets
• Built Airflow DAGs for orchestrating complex data workflows
• Developed DBT models for transforming raw data into analytics-ready tables
• Optimized SQL queries and table structures for high-performance DWH access
• Monitored ETL jobs and data quality, reducing errors and delays
• Automated pipeline deployment using CI/CD practices
• Collaborated with BI and analytics teams to deliver accurate reporting datasets
• Refined data warehouse architecture to support growing data volumes
• Implemented logging and alerting for pipeline failures and latency spikes
• Mentored junior engineers in data engineering best practices
• Improved Python scripts for ETL efficiency and maintainability
• Ensured secure and compliant data handling for sensitive financial data
• Participated in Agile workflows, delivering ETL improvements on schedule
• Delivered scalable and reliable ETL pipelines supporting cross-functional teams
• Reduced pipeline runtime and improved system throughput for analytical workloads

Wise | Full Stack Engineer May 2015 ~ Feb 2019

• Developed ETL processes and data pipelines for transactional and analytical systems
• Maintained and optimized SQL queries for large-scale data extraction and transformation
• Orchestrated workflows using Airflow, ensuring timely delivery of data to analysts
• Built DWH structures to support real-time and batch reporting
• Automated pipeline monitoring and validation, reducing errors and manual intervention
• Refined Python ETL scripts to improve performance and reliability
• Collaborated with data analysts to design data models for reporting and BI tasks
• Implemented security controls and GDPR-compliant data handling
• Mentored colleagues on ETL design and best practices
• Improved data throughput and reduced latency for critical analytical pipelines
• Participated in Agile ceremonies, delivering improvements to ETL workflows
• Optimized DWH schemas and indexing strategies for query performance
• Delivered measurable improvements in data quality, pipeline efficiency, and reporting speed

Education

National University of Singapore Aug 2009 ~ Jul 2013

Bachelor’s Degree in Computer Science

Skills

Programming Languages & Frameworks: Python, SQL, DBT, ETL/ELT frameworks
Backend / API Development: Data pipelines, workflow orchestration, DAG design, API integrations
DevOps / Infrastructure / CI-CD: Airflow, DAG orchestration, automated deployments, monitoring pipelines
Databases / Data Engineering: Greenplum, ClickHouse, data warehouse construction, data modeling, migrations
Cloud Platforms: AWS, GCP, cloud-based ETL and DWH solutions
Testing / QA / Automation: Data validation, automated testing of ETL pipelines, monitoring data quality
Security / Observability / Compliance: Access control, secure data handling, GDPR compliance
Collaboration / Agile / Product Tools: Agile/Scrum workflows, cross-functional team collaboration, JIRA, Confluence