First: do you really need a background thread to sit idle in the first place?
On most platforms, starting a new stream is cheap. (With the exception of Windows and Linux, where it overloads.) So, why not just start the thread when you need it? (It's also easy to save a list of threads as a single thread, right?)
Alternatively, why not just create a ThreadPoolExecutor and just send jobs to it, and let the executor worry about when they get to start and on which thread. Every time you can just think of “tasks that you need to run without blocking the main thread” instead of “workflows that should wait at work”, you make your life easier. Under the covers, there is one or more workflows waiting in line, or something similar, but this part has been written (and debugged and optimized) for you. All you need to write are tasks that are just regular functions.
But, if you want to write explicit background streams, you can, so I will explain this.
How can I do a workflow without using CPU time? ... What is the best way to tell the workflow to do something?
A way to idle a thread until a value is ready is to wait for a synchronization object. In any modern OS, waiting on a synchronization object means that the operating system stops giving you any processor time until the object is ready for you. *
Threading module documents have many different options, but Condition is obvious for use in most cases. The workflow reporting method will then notify Condition .
However, Queue is often much simpler. To wait on Queue , just call its get method with block=True . To signal another thread to wake up, just put something on Queue . (Under the covers, a Queue wraps a list or deque or another collection, Lock and a Condition , so you just say what you want to do - check the value, until the value appears, add the value instead of having to wait and alarm and protect the collection.)
See the response to managing user interface elements in wxPython using streaming to signal in both directions, from workflow to user interface thread and vice versa.
I will have some way for the workflow to register a callback with the user interface itself, so when a button is clicked or some other user interface event occurs, the workflow will be signaled to change what it does.
You can do this if you want. Just pass self.queue.put or def callback(value): self.value = value; self.condition.notify() def callback(value): self.value = value; self.condition.notify() or something as a callback, and the GUI thread should not even know that the callback starts another thread.
Actually, this is a pretty nice design that can make you very happy later when you decide to move the code back and forth between the inline and background thread or move it to a child process instead of a background thread, or something else.
I can’t imagine it right now, but I could see how the application becomes more complex and should signal the workflow while it is really busy with something.
But what do you want if he is busy?
If you just want to say: “If you are idle, wake up and complete this task, otherwise hold on to it and do it when you are ready,” this is exactly what a Queue or Executor will do for you automatically.
If you want to say: “If you are idle, wake up, otherwise, do not worry about it,” what Condition or Event will do.
If you want to say: “If you are idle, wake up and do it, otherwise cancel what you are doing and do it instead,” it's a little more complicated. You pretty much need the background thread to periodically check the "interrupt_me" variable while it is busy (and put a Lock around it), and then you set this flag and also notify Condition ... although in some cases, you can combine cases waiting and busy in one Condition or Event (by calling endless wait() in standby mode and quickly checking wait(timeout=0) when busy).
* In some cases, for example, Linux futex or Windows CriticalSection -it, in some cases it can speed up the processor time a little, because it is a good optimization. But the fact is that you do not request processor time until you are ready to use it.