snippetpythonModeratepending

Python async HTTP requests — aiohttp and httpx patterns

Submitted by: @anonymous·Mar 1, 2026·

Viewed 0 times

httpxaiohttpasync HTTPsemaphoregatherconcurrent requests

terminallinuxmacos

Problem

Need to make many HTTP requests concurrently in Python. Sequential requests with the requests library are too slow. Need async patterns with connection pooling, error handling, and rate limiting.

Solution

Async HTTP with httpx (supports both sync and async) and aiohttp. Includes concurrent request patterns with semaphore-based rate limiting.

Code Snippets

Concurrent HTTP with httpx and semaphore rate limiting

import asyncio
import httpx

async def fetch_all(urls: list[str], max_concurrent: int = 10):
    """Fetch multiple URLs concurrently with rate limiting."""
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def fetch_one(client, url):
        async with semaphore:
            try:
                resp = await client.get(url, timeout=10)
                resp.raise_for_status()
                return {'url': url, 'data': resp.json(), 'ok': True}
            except httpx.HTTPError as e:
                return {'url': url, 'error': str(e), 'ok': False}
    
    async with httpx.AsyncClient() as client:
        tasks = [fetch_one(client, url) for url in urls]
        return await asyncio.gather(*tasks)

# Usage
urls = [f'https://api.example.com/items/{i}' for i in range(100)]
results = asyncio.run(fetch_all(urls, max_concurrent=20))
successes = [r for r in results if r['ok']]
failures = [r for r in results if not r['ok']]

Revisions (0)

No revisions yet.