Introduction

Our Goal

Walk through two examples of threading and multiprocessing to illustrate rules and best practices for taking advantage of concurrency.

Why Concurrency?

  1. Improve perceived responsiveness
  2. Improve speed
  3. Because that is how the real world works

Martelli Model of Scaleability

  • 1 core: Single thread and single process
  • 2-8 cores: Multiple threads and multiple processes
  • 9+ cores: Distributed processing

Martelli’s observation: As time goes on, the second category becomes less common and relevant. Single cores become more powerful. Big datasets grow ever larger.

Global Interpreter Lock

CPython has a lock for its internal shared global state.

The unfortunate effect of the GIL is that no more than one thread can run at a time.

For I/O bound applications, the GIL doesn’t present much of an issue. For CPU bound applications, using threading makes the application speed worse. Accordingly, that drives us to multi-processing to gain more CPU cycles.

Threads vs Processes

“Your weakness is your strength and your strength is your weakness”.

The strength of threads is shared state. The weakness of threads is shared state (managing race conditions).

The strength of processes is their independence from one another. The weakness of processes is lack of communication (hence the need for IPC and object pickling and other overhead).

Threads vs Async

Threads

Threads switch preemptively. This is convenient because you don’t need to add explicit code to cause a task switch.

The cost of this convenience is that you have to assume a switch can happen at any time. Accordingly, critical sections have to be guarded with locks.

The limit on threads is total CPU power minus the cost of task switches and synchronization overhead.

Async

Async switches cooperatively, so you do need to add explicit code “yield” or “await” to cause a task switch.

Now you control when task switches occur, so locks and other synchronization are no longer needed.

Also, the cost task switches is very low. Calling a pure Python function has more overhead than restarting a generator or awaitable.

This means that async is very cheap.

In return, you’ll need a non-blocking version of just about everything you do. Accordingly, the async world has a huge ecosystem of support tools. This increases the learning curve.

Comparison

  • Async maximizes CPU utilization because it has less overhead than threads.
  • Threading typically works with existing code and tools as long as locks are added around critical sections.
  • For complex systems, async is much easier to get right than threads with locks.
  • Threads require very little tooling (locks and queues).
  • Async needs a great deal of tooling (futures, event loops, and non-blocking versions of just about everything).