Multi-Threading in Python using the ‘Activity’ Application
Threads in the computer world are described as simple executors (which execute the given instructions). These are lightweight and used to perform tasks asynchronously.
Eg: Different tabs in browser act as different threads
In general, At the OS level, threads have their own resources but at an abstract level, they use the same code, data and files.
Source: https://www.studytonight.com
Multithreading in activity application:
Activity is an application developed by my good friend Arijit. It performs bulk server command execution and validations over ssh asynchronously.
It was developed in python 3.6. The core concept of this module is that, it uses multi-threading to perform command execution in parallel on all the given servers because of which it consumes less time and performs tasks in an organised and efficient way.
To understand multithreading, Let's dive into some code which is part of
activity app, which performs ping check on all the hosts asynchronously.
The above-defined function is a callable function, which performs the required activity.
To understand this, we should know a brief background of how parallel execution works in python.
concurrent.futures: Launching parallel tasks
The concurrent.futures module provides a high-level interface for asynchronously executing callables which is an In-built module.
In our program, _ping_single(args): is a executing callable function
The asynchronous execution can be performed using threads and processes. Sub-function for threads in concurrent.futures is ThreadPoolExecutor and processes is ProcessPoolExecutor.
To import required module
from concurrent.futures import ThreadPoolExecutor, as_completed
Once imported,
with ThreadPoolExecutor(max_workers=PARALLEL_LIMIT) as pool:
The above line is used to initialize the threads. It is initialized with with so that when execution of a threads complete, the resources of the thread are freed and then shutdown the Executor.
The execution starts once the callable function is submitted along with
arguments as shown above. In the above line hosts, as well as required arguments are submitted along with callable function.
We can use pool.map to submit the tasks to threads, The difference would be seen in while capturing the output.
for f in as_completed(futures):
output.append(f.result())
Once the execution of the thread is started, we can capture the output using as_completed or the wait function.
The as_completed() function takes an iterable of Future objects and starts yielding values as soon as the futures (threads) start resolving. The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result of the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.
The wait() function would return a named tuple which contains two set — one set contains the futures which completed (either got result or exception) and the other set containing the ones which didn’t complete.
Note: If the threads are not configured properly, there will be performance issues like deadlock, memory leaks.
Conclusion:
Multithreading if configured properly, is an efficient and organized way to reduce the time taken for execution as well as improve the performance of the program.
For more explanation on multithreading please refer the below article:
https://medium.com/@bfortuner/python-multithreading-vs-multiprocessing-73072ce5600b
Source:
Top comments (0)