Zhangyuan Hu

Zhangyuan, Hu works as Senior Software Engineer for Data warehouse and Monetization team at Optimizely. Before joining Optimziely, Frank was Founding partner and Lead Software Engineer at MailTime Inc, A Y-Combinator company to simplify email experience as easy as text messaging. He holds a Bachelor of Engineering in Information Engineering from The Chinese University of Hong Kong.

Speaker home page

Service Testing with Apache Airflow

DevOps & Automation, Intermediate
8/19/2018 | 3:35 PM-4:05 PM | Fisher


End-to-end (E2E) testing is a critical step in the testing pyramid to ensure features powered by multiple services are shipped with high quality. To make it easier at Optimizely, we’ve turned to Apache Airflow—a framework that makes it easy to manage complex workflows. In this talk, we’ll share a pattern we use to test our services end-to-end on top of Airflow and how you can easily adopt it!


Shipping quality software is hard for so many reasons. As engineers, we try our best to ship the best code we can and use frameworks like the testing pyramid to increase our confidence that we are. While unit tests are usually easy to write and integration tests are easy-ish to write, the final frontier of end-to-end (E2E) tests is much more difficult to develop. Despite their difficulty, E2E tests are critical to ensuring features are built in a way that meets users’ needs at launch and during its lifetime.

In a services-oriented architecture, E2E tests become even more difficult to write because services tend to be owned by different teams. We reached this problem at Optimizely when we wanted to have rock-solid quality over counting impressions, which we use for billing our customers—so it’s pretty important we get that right and have confidence that it’s always right.

The services we relied on surfaced APIs, and in the end, it’s all data, so we thought: what if we use Apache Airflow to make this easy to write, understand, monitor and schedule? Our intuition was accurate and today we use this system to ensure changes to our counting logic do not cause regressions and provide a bad experience for our customers.

Airflow defines workflows as directed acyclic graphs (DAGs). By defining a test step as a node in the DAG, we’re able to more easily debug independent failures and can share these steps across teams to reduce maintenance and communication overhead

We’re excited by this pattern we’ve been using for more than 6 months because it has proven itself invaluable for us by helping catch critical bugs before they happen. More so, we can extend the same test as a monitor by running on a timer post-production. We hope this simple pattern proves useful and makes E2E testing easier for you too!

As an outline, this talk will cover: 0 - 2 min: Why E2E testing is important but hard to implement 2 - 4 min: An introduction to Apache Airflow 4 - 5 min: How Apache Airflow workflows map to E2E tests 5 - 8 min: How to write a simple end-to-test using Airflow 8 - 10 min: The benefits of writing E2E tests this way for the debugger and the tester 10 -16 min: Scaling this pattern across teams by sharing the DAG, not the code 16 - 18 min: Using the same E2E test to also monitor in production 18 - 20 min: Conclusion and wrap-up 20 - 25 min: Open for questions