In this tutorial we will learn how to write python code that provides solution to the classic producer consumer problem with the help of an example. We will make use of inbuilt python features such as Thread and Queue.
Before we proceed with example we will quickly get familiar with python
Thread features used inside below code.
Queue class which implements queue data structure. We can put an item inside the queue and we can get an item from the queue. By default this works in FIFO manner.
In our example, function producer will put an item inside queue and function consumer will get an item from the queue. We will use following method of queue class by instantiating queue object
q = Queue().
|q.put()||To put an item inside queue.|
|q.get()||To get an item which is present inside queue.|
|q.join()||This method stops python program from exiting till it gets signal from the below method ||q.task_done()||This method should be called when item got outside from the queue using |
Python allows writing multi-threaded program using
Thread class. We will instantiate object of thread class and make use of following methods to process (consume) multiple items concurrently.
|Constructor / Method||Description|
|t = Thread(target=consumer)||Instantiate thread object which would make call to function |
|t.start()||Starts execution of thread by making call to function |
from threading import Thread from Queue import Queue q = Queue() final_results =  def producer(): for i in range(100): q.put(i) def consumer(): while True: number = q.get() result = (number, number**2) final_results.append(result) q.task_done() for i in range(5): t = Thread(target=consumer) t.daemon = True t.start() producer() q.join() print final_results
- In above example function
producerwill generate number from 0 to 99 and put each number inside queue as an item to be processed.
consumerwill try to get the number (item) from queue and square the number. Consumer will store result inside global list
final_result. Result will include tuple containing number and square of that number.
- In solution we have spawn 5 threads to call function consumer. Number of threads can be arbitrary and you can tweak it according to your speed requirement. Generally for IO bound program more threads would yield faster performance however it has upper bound too. So it is good idea to try and tweak the number it according to ones need.
t.daemon = Truewill make each thread as daemon thread. This will make sure that all threads will be killed once main program exits.