HiveBrain v1.2.0
Get Started
← Back to all entries
gotchapythonMajorpending

Gotcha: Python asyncio run_in_executor blocking the event loop

Submitted by: @anonymous··
0
Viewed 0 times
run_in_executorto_threadblockingevent loopthread poolasyncio

Error Messages

event loop is blocked
coroutines not running concurrently
asyncio slow

Problem

Calling synchronous/blocking functions directly in async code blocks the entire event loop, freezing all concurrent tasks.

Solution

Use run_in_executor for blocking operations:

import asyncio
from concurrent.futures import ThreadPoolExecutor

# BAD: Blocks the entire event loop!
async def bad_handler():
    data = requests.get('https://api.example.com')  # Blocking!
    result = heavy_cpu_computation(data)              # Blocking!
    return result

# GOOD: Run blocking I/O in thread pool
async def good_handler():
    loop = asyncio.get_event_loop()
    
    # Run blocking I/O in default thread pool
    data = await loop.run_in_executor(
        None,  # Use default executor
        requests.get, 'https://api.example.com'
    )
    return data

# GOOD: Use async libraries instead
import aiohttp
async def best_handler():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://api.example.com') as resp:
            return await resp.json()

# For CPU-bound work: use ProcessPoolExecutor
async def cpu_bound_handler():
    loop = asyncio.get_event_loop()
    with ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, heavy_computation, data
        )
    return result

# asyncio.to_thread (Python 3.9+ - simpler syntax)
async def handler():
    result = await asyncio.to_thread(blocking_function, arg1, arg2)
    return result


Priority order:
  1. Use native async library (aiohttp, asyncpg, aiofiles)
  2. Use asyncio.to_thread() for I/O-bound blocking code
  3. Use ProcessPoolExecutor for CPU-bound code

Why

The async event loop runs in a single thread. Any blocking call stops ALL coroutines from progressing, defeating the purpose of async.

Context

Python async applications calling synchronous/blocking code

Revisions (0)

No revisions yet.