{"id":134930,"date":"2023-09-23T06:50:42","date_gmt":"2023-09-23T06:50:42","guid":{"rendered":"https:\/\/blog.finxter.com\/?p=1651739"},"modified":"2023-09-23T06:50:42","modified_gmt":"2023-09-23T06:50:42","slug":"python-async-io-the-ultimate-guide-in-a-single-post","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2023\/09\/23\/python-async-io-the-ultimate-guide-in-a-single-post\/","title":{"rendered":"Python Async IO \u2013 The Ultimate Guide in a Single Post"},"content":{"rendered":"\n
\n
\n
\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n
\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n
5\/5 – (1 vote) <\/div>\n<\/p><\/div>\n

As a Python developer, you might have come across the concept of asynchronous programming. Asynchronous programming, or async I\/O, is a concurrent programming design that has received dedicated support in Python, evolving rapidly from Python 3.4 through 3.7<\/a> and beyond. With async I\/O, you can manage multiple tasks concurrently without the complexities of parallel programming, making it a perfect fit for I\/O bound and high-level structured network code.<\/p>\n

In the Python world, the asyncio<\/a><\/code> library is your go-to tool for implementing asynchronous I\/O. This library provides various high-level APIs to run Python coroutines concurrently, giving you full control over their execution<\/strong>. It also enables you to perform network I\/O, Inter-process Communication (IPC), control subprocesses, and synchronize concurrent code using tasks and queues.<\/p>\n

Understanding Asyncio<\/h2>\n

In the world of Python programming, asyncio<\/strong> plays a crucial role in designing efficient and concurrent code without using threads. It is a library that helps you manage tasks, event loops, and coroutines. To fully benefit from asyncio<\/code>, you must understand some key components.<\/p>\n

First, let’s start with coroutines<\/strong>. They are special functions that can pause their execution at specified points without completely terminating it. In Python, you declare a coroutine using the async def<\/code> syntax. <\/p>\n

For instance:<\/p>\n

async def my_coroutine(): # Your code here\n<\/pre>\n
Next, the event loop<\/strong> is a core feature of asyncio and is responsible for executing tasks concurrently and managing I\/O operations. An event loop runs tasks one after the other and can pause a task when it is waiting for external input, such as reading data from a file or from the network. It also listens for other tasks that are ready to run, switches to them, and resumes the initial task when it receives the input.<\/p>\n
Tasks<\/strong> are the coroutines wrapped in an object, managed by the event loop. They are used to run multiple concurrent coroutines simultaneously. You can create a task using the asyncio.create_task()<\/code> function, like this:<\/p>\n
async def my_coroutine(): # Your code here task = asyncio.create_task(my_coroutine())\n<\/pre>\n
Finally, the sleep<\/code> function in asyncio is used to simulate I\/O bound tasks or a delay in the code execution. It works differently than the standard time.sleep()<\/code> function as it is non-blocking and allows other coroutines to run while one is paused. You can use await asyncio.sleep(delay)<\/code> to add a brief pause in your coroutine execution.<\/p>\n
Putting it all together, you can use asyncio<\/code> to efficiently manage multiple coroutines concurrently:<\/p>\n
import asyncio async def task_one(): print('Starting task one') await asyncio.sleep(3) print('Finished task one') async def task_two(): print('Starting task two') await asyncio.sleep(1) print('Finished task two') async def main(): task1 = asyncio.create_task(task_one()) task2 = asyncio.create_task(task_two()) await task1 await task2 # Run the event loop\nasyncio.run(main())\n<\/pre>\n
In this example, the event loop will start running both tasks concurrently, allowing task two to complete while task one is paused during the sleep period. This allows you to handle multiple tasks in a single-threaded environment. <\/p>\n
You can see it play out in this Gif:<\/p>\n
\n
\"\"<\/figure>\n<\/div>\n
Async\/Await Syntax<\/h2>\n
In Python, the async\/await<\/a> syntax is a powerful tool to create and manage asynchronous tasks without getting lost in callback hell or making your code overly complex.<\/p>\n
The async\/await keywords are at the core of asynchronous code in Python. You can use the async def<\/code> keyword to define an asynchronous function. Inside this function, you can use the await<\/code> keyword to pause the execution of the function until some asynchronous operation is finished. <\/p>\n
For example:<\/p>\n
import asyncio async def main(): print(\"Start\") await asyncio.sleep(2) print(\"End\")\n<\/pre>\n
yield<\/code> and yield from<\/code><\/a> are related to asynchronous code in the context of generators, which provide a way to iterate through a collection of items without loading all of them into memory at once. In Python 3.3 and earlier, yield from<\/code> was used to delegate a part of a generator’s operation to another generator. However, in later versions of Python, the focus shifted to async\/await for managing asynchronous tasks, and yield from<\/code> became less commonly used.<\/p>\n
For example, before Python 3.4, you might have used a generator with yield<\/code> and yield from<\/code> like this:<\/p>\n
def generator_a(): for i in range(3): yield i def generator_b(): yield from generator_a() for item in generator_b(): print(item)\n<\/pre>\n
With the introduction of async\/await, asynchronous tasks can be written more consistently and readably. You can convert the previous example to use async\/await as follows:<\/p>\n
import asyncio async def async_generator_a(): for i in range(3): yield i await asyncio.sleep(1) async def async_generator_b(): async for item in async_generator_a(): print(item) await async_generator_b()\n<\/pre>\n<\/p>\n
Working with Tasks and Events<\/h2>\n
In asynchronous programming with Python, you’ll often work with tasks and events to manage the execution of simultaneous IO-bound operations. To get started with this model, you’ll need to understand the event loop<\/strong> and the concept of tasks<\/strong>.<\/p>\n
The event loop is a core component of Python’s asyncio<\/code> module. It’s responsible for managing and scheduling the execution of tasks. A task, created using asyncio.create_task()<\/code>, represents a coroutine that runs independently of other tasks in the same event loop.<\/p>\n
To create tasks, first, define an asynchronous function using the async def<\/code> syntax. Then, you can use the await<\/a><\/code> keyword to make non-blocking calls within this function. The await<\/code> keyword allows the event loop to perform other tasks while waiting for an asynchronous operation to complete. <\/p>\n
Here’s an example:<\/p>\n
import asyncio async def my_async_function(): print(\"Task started\") await asyncio.sleep(2) print(\"Task finished\") event_loop = asyncio.get_event_loop()\ntask = event_loop.create_task(my_async_function())\nevent_loop.run_until_complete(task)\n<\/pre>\n
In this example, my_async_function<\/code> is an asynchronous function, and await asyncio.sleep(2)<\/code> represents an asynchronous operation. The event_loop.create_task()<\/code> method wraps the coroutine into a task, allowing it to run concurrently within the event loop.<\/p>\n
To execute tasks and manage their output, you can use asyncio.gather()<\/code>. This function receives a list of tasks and returns their outputs as a list in the same order they were provided. Here’s an example of how you can use asyncio.gather()<\/code>:<\/p>\n
import asyncio async def async_task_1(): await asyncio.sleep(1) return \"Task 1 completed\" async def async_task_2(): await asyncio.sleep(2) return \"Task 2 completed\" async def main(): tasks = [async_task_1(), async_task_2()] results = await asyncio.gather(*tasks) print(results) asyncio.run(main())\n<\/pre>\n
In this example, asyncio.gather()<\/code> awaits the completion of both tasks and then collects their output in a list, which is printed at the end.<\/p>\n
Working with tasks and events in Python’s asynchronous IO model helps improve the efficiency of your code when dealing with multiple IO operations, ensuring smoother and faster execution. Remember to use asyncio.create_task()<\/code>, await<\/code>, and asyncio.gather()<\/code> when handling tasks within your event loop.<\/p>\n
Coroutines and Futures<\/h2>\n
In Python, async IO is powered by coroutines and futures. Coroutines<\/strong> are functions that can be paused and resumed at specific points, allowing other tasks to run concurrently. They are declared with the async<\/code> keyword and used with await<\/code>. Asyncio coroutines<\/a> are the preferred way to write asynchronous code in Python.<\/p>\n
On the other hand, futures<\/strong> represent the result of an asynchronous operation that hasn’t completed yet. They are primarily used for interoperability between callback-based code and the async\/await syntax. With asyncio, Future objects<\/a> should be created using loop.create_future()<\/code>.<\/p>\n
To execute multiple coroutines concurrently, you can use the gather<\/code><\/a> function. asyncio.gather()<\/code> is a high-level function that takes one or more awaitable objects (coroutines or futures) and schedules them to run concurrently. Here’s an example:<\/p>\n
import asyncio async def foo(): await asyncio.sleep(1) return \"Foo\" async def bar(): await asyncio.sleep(2) return \"Bar\" async def main(): results = await asyncio.gather(foo(), bar()) print(results) asyncio.run(main())\n<\/pre>\n
In this example, both foo()<\/code> and bar()<\/code> coroutines run concurrently, and the gather()<\/code> function returns a list of their results.<\/p>\n
Error handling in asyncio is done through the set_exception()<\/code> method. If a coroutine raises an exception, you can catch the exception and attach it to the associated future using future.set_exception()<\/code><\/a>. This allows other coroutines waiting for the same future to handle the exception gracefully.<\/p>\n
In summary, working with coroutines and futures helps you write efficient, asynchronous code in Python. Use coroutines along with the async\/await syntax for defining asynchronous tasks, and futures for interacting with low-level callback-based code. Utilize functions like gather()<\/code> for running multiple coroutines concurrently, and handle errors effectively with future.set_exception()<\/code>.<\/p>\n
Threading and Multiprocessing<\/h2>\n
In the world of Python, you have multiple options for concurrent execution and managing concurrency. Two popular approaches to achieve this are threading<\/strong> and multiprocessing<\/strong>.<\/p>\n
Threading<\/strong> can be useful when you want to improve the performance of your program by efficiently utilizing your CPU’s time. It allows you to execute multiple threads in parallel within a single process. Threads share memory and resources, which makes them lightweight and more suitable for I\/O-bound tasks. However, because of the Global Interpreter Lock (GIL) in Python, only one thread can execute at a time, limiting the benefits of threading for CPU-bound tasks. You can explore the threading<\/code> module for building multithreaded applications.<\/p>\n
Multiprocessing<\/strong> overcomes the limitations of threading by using multiple processes working independently. Each process has its own Python interpreter, memory space, and resources, effectively bypassing the GIL. This approach is better for CPU-bound tasks, as it allows you to utilize multiple cores to achieve true parallelism. To work with multiprocessing, you can use Python’s multiprocessing<\/code> module.<\/p>\n
While both threading and multiprocessing help manage concurrency, it is essential to choose the right approach based on your application’s requirements. Threading is more suitable when your tasks are I\/O-bound, and multiprocessing is advisable for CPU-bound tasks. When dealing with a mix of I\/O-bound and CPU-bound tasks, using a combination of the two might be beneficial.<\/p>\n
Async I\/O offers another approach for handling concurrency and might be a better fit in some situations. However, understanding threading and multiprocessing remains crucial to make informed decisions and efficiently handle concurrent execution in Python.<\/p>\n
Understanding Loops and Signals<\/h2>\n
In the world of Python async IO, working with loops and signals is an essential skill to grasp. As a developer, you must be familiar with these concepts to harness the power of asynchronous programming.<\/p>\n
Event loops are at the core of asynchronous programming in Python. They provide a foundation for scheduling and executing tasks concurrently. The asyncio<\/code> library helps you create and manage these event loops. You can experiment with event loops using Python’s asyncio<\/code> REPL, which can be started by running python -m asyncio<\/code> in your command line.<\/p>\n
Signals, on the other hand, are a way for your program to receive notifications about certain events, like a user interrupting the execution of the program. A common use case for handling signals in asynchronous programming involves stopping the event loop gracefully when it receives a termination signal like SIGINT<\/code> or SIGTERM<\/code>.<\/p>\n
A useful method for running synchronous or blocking functions in an asynchronous context is the loop.run_in_executor()<\/code> method. This allows you to offload the execution of such functions to a separate thread or process, preventing them from blocking the event loop. For example, if you have a CPU-bound operation that cannot be implemented using asyncio<\/code>‘s native coroutines, you can utilize loop.run_in_executor()<\/code> to keep the event loop responsive.<\/p>\n
Here’s a simple outline of using loops and signals together in your asynchronous Python code:<\/p>\n
    \n
  1. Create an event loop using asyncio.get_event_loop()<\/code>.<\/li>\n
  2. Register your signal handlers with the event loop, typically by using the loop.add_signal_handler()<\/code> method.<\/li>\n
  3. Schedule your asynchronous tasks and coroutines in the event loop.<\/li>\n
  4. Run the event loop using loop.run_forever()<\/code>, which will keep running until you interrupt it with a signal or a coroutine stops it explicitly.<\/li>\n<\/ol>\n
    Managing I\/O Operations<\/h2>\n
    When working with I\/O-bound tasks in Python, it’s essential to manage I\/O operations efficiently. Using asyncio<\/strong> can help you handle these tasks concurrently, resulting in more performant and scalable code.<\/p>\n
    I\/O-bound tasks are operations where the primary bottleneck is fetching data from input\/output sources like files, network requests, or databases. To improve the performance of your I\/O-bound tasks, you can use asynchronous programming techniques. In Python, this often involves using the asyncio<\/strong> library and writing non-blocking code.<\/p>\n
    Typically, you’d use blocking code for I\/O operations, which means waiting for the completion of an I\/O task before continuing with the rest of the code execution. This blocking behavior can lead to inefficient use of resources and poor performance, especially in larger programs with multiple I\/O-bound tasks.<\/p>\n
    Non-blocking code, on the other hand, allows your program to continue executing other tasks while waiting for the I\/O operation to complete. This can significantly improve the efficiency and performance of your program. When using Python’s asyncio<\/strong> library, you write non-blocking code with coroutines<\/strong>.<\/p>\n
    For I\/O-bound tasks involving file operations, you can use libraries like aiofiles<\/strong> to perform asynchronous file I\/O. Just like with asyncio, aiofiles provides an API to work with files using non-blocking code, improving the performance of your file-based tasks.<\/p>\n
    When dealing with network I\/O, the asyncio<\/code><\/strong> library provides APIs to perform tasks such as asynchronous reading and writing operations for sockets and other resources. This enables you to manage multiple network connections concurrently, efficiently utilizing your system resources.<\/p>\n
    In summary, when managing I\/O operations in Python:<\/p>\n