Ville is an architect on the ML Infrastructure team at Netflix. He has been building ML systems and products in large companies and startups, including one that he co-founded, for 10+ years.
We will share our experiences on building Metaflow, a Python library that is used at Netflix to build and operate hundreds of machine learning applications. This talk is for you if you want to learn how to develop systems for big data and ML in Python.
Metaflow is a Python library that empowers data scientists to prototype, build, deploy, and operate end-to-end machine learning solutions. We started building Metaflow at Netflix to provide a solid foundation for hundreds of internal ML use cases, from classical statistical analysis to large-scale applications of deep learning. Metaflow is designed with a human-centric mindset: instead of reinventing the wheel for large-scale computing or machine learning, we integrate existing solutions into a delightfully consistent and easy-to-use package.
This talk focuses on the internals of Metaflow; it will highlight lessons that we have learned in building a Python library that needs to be robust, performant, and flexible enough to solve a large set of complex real-world business problems related to machine learning. We will share our learnings around optimizing the performance of Python, in particular, related to concurrency. We will also talk about the complexities of dependency management, the overall architecture of the library and the cloud-based compute and storage systems it relies on.