How many threads per proxy? Find your safe concurrency

How many threads per proxy you can run depends on the website, not the proxy. Start with two to four requests in flight per IP, double the number while the error rate stays near zero and response times stay flat, and stop at the first level where either one moves. Then run at about half of that level.

That answer needs a test to be useful, so this post gives you one: a short Python script that ramps concurrency up against your own target and reports where it stops holding. It also covers how rotating residential and static IPs change the arithmetic, and the connection-pooling detail that quietly undoes rotation. Snippets use HOST, PORT, USERNAME and PASSWORD for the values on your service's page in the dashboard.

Why there is no single answer to how many threads per proxy

Four different things cap your concurrency, and the lowest one wins:

• The target's limit per IP. Most sites count requests per address. Ten requests in flight from one IP is a lot more visible than ten spread over ten IPs.
• The target's overall tolerance. Bot rules, a small server, or a slow search page can all push back no matter how many IPs you use.
• Your own machine. Connection pools, open file limits and your uplink run out before most proxies do. Proxy timeout errors shows what that looks like from the inside.
• Your provider's plan. Some providers cap threads or ports per plan. On ours, the answer to "Are there port, thread or monthly fees?" in the FAQ reads: no port fees, no thread fees, no monthly minimum.

The first two are the ones that decide the number, and only the target can tell you what they are. That is why the answer comes from a measurement.

Concurrency, rate and threads are different numbers

Concurrency is how many requests are in flight at the same moment. Rate is how many you send per second. They are tied together by how long each request takes:

rate ≈ concurrency ÷ seconds per request

Four requests in flight, each taking half a second, is about eight requests a second. If the site slows to two seconds per page, the same four in flight drop to two a second. Your code sets concurrency, but sites usually limit rate, and a site under load often slows down before it refuses anything. The ramp test below watches response times for that reason.

Threads are one way to get concurrency. A thread pool of eight is eight requests in flight:

from concurrent.futures import ThreadPoolExecutor

import requests

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"
PROXIES = {"http": PROXY, "https": PROXY}
THREADS = 4


def status(url):
    try:
        return requests.get(url, proxies=PROXIES, timeout=(5, 30)).status_code
    except requests.RequestException as error:
        return type(error).__name__


urls = [f"https://TARGET_SITE/page/{n}" for n in range(1, 41)]
with ThreadPoolExecutor(max_workers=THREADS) as pool:
    for url, result in zip(urls, pool.map(status, urls)):
        print(result, url)

With asyncio, a semaphore does the same job without threads, and the rest of this post uses that form.

Rotating residential vs static IPs

Residential with Randomize IP. Each new connection leaves from a different home address, so your concurrency spreads across many exits and no single IP carries much of it. The per-IP limit mostly stops being your problem; the site's overall tolerance and your per-GB traffic become the limits. Exits are ordinary home connections, some slower than others, so expect a wider spread of response times than on a static IP.

Static ISP or datacenter IPs. Every request from one IP counts against that IP. Your total concurrency is shared by the IPs you rent, so the useful question becomes "how many per IP", and the answer is multiplied by the number of IPs. You pick the country, region and city when you order, which also keeps results comparable. ISP or datacenter proxies? covers the choice between the two.

One honest note from our own honesty page: our residential pool is smaller than the big providers'. At very high concurrency on a heavily defended target, that gap is real. Measure your job before buying for it.

Connection pooling can undo rotation

An HTTP client reuses connections. For an https site, a reused connection is the same tunnel through the proxy, and so the same exit IP. With httpx.AsyncClient and four requests in flight, twenty requests travel over about four tunnels, which means four exits doing five requests each. Rotating proxies in Python covers the same trap with requests.Session.

If you want a fresh exit per request, turn keep-alive off in the client:

import httpx

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"
limits = httpx.Limits(max_keepalive_connections=0)
client = httpx.AsyncClient(proxy=PROXY, limits=limits, timeout=httpx.Timeout(30.0, connect=5.0))

We checked this against a local proxy that logs each tunnel: twenty requests at four in flight opened 4 tunnels with default settings and 20 with keep-alive off. The trade-off is a new connection and TLS handshake per request, which adds time, so keep the default when one exit per worker is fine for your target.

A ramp-up test that finds your number

The script sends a batch at each concurrency level, doubling from 2, and measures three things per level: pages per second, the median response time, and the share that failed. It stops at the first sign of trouble.

pip install httpx

import asyncio
import statistics
import time
from itertools import cycle, islice

import httpx

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"
SAMPLE_URLS = [
    "https://TARGET_SITE/page/1",
    "https://TARGET_SITE/page/2",
    "https://TARGET_SITE/page/3",
]
START, CEILING = 2, 32
REQUESTS_PER_SLOT = 10
MAX_ERROR_RATE = 0.02
MAX_SLOWDOWN = 2.0
MIN_GAIN = 1.2
COOL_DOWN = 30


async def timed_get(client, gate, url):
    async with gate:
        start = time.monotonic()
        try:
            response = await client.get(url)
            status = response.status_code
        except httpx.HTTPError as error:
            status = type(error).__name__
        return status, time.monotonic() - start


async def run_level(concurrency):
    urls = list(islice(cycle(SAMPLE_URLS), concurrency * REQUESTS_PER_SLOT))
    gate = asyncio.Semaphore(concurrency)
    timeout = httpx.Timeout(30.0, connect=5.0)
    started = time.monotonic()
    async with httpx.AsyncClient(proxy=PROXY, timeout=timeout) as client:
        results = await asyncio.gather(*(timed_get(client, gate, url) for url in urls))
    elapsed = time.monotonic() - started
    failed = sum(1 for status, _ in results if status != 200)
    median = statistics.median(seconds for _, seconds in results)
    return len(results) / elapsed, median, failed / len(results)


async def ramp():
    safe, baseline, best_rate = None, None, 0.0
    level = START
    while level <= CEILING:
        rate, median, error_rate = await run_level(level)
        baseline = baseline or median
        print(f"{level:>3} at once: {rate:5.1f} req/s, median {median:.2f}s, {error_rate:.0%} failed")
        if error_rate > MAX_ERROR_RATE:
            print("  errors: the site is pushing back")
            break
        if median > baseline * MAX_SLOWDOWN:
            print("  responses slowed down: the site is queuing you")
            break
        if best_rate and rate < best_rate * MIN_GAIN:
            print("  no real gain in throughput: more workers only wait longer")
            break
        safe, best_rate = level, rate
        level *= 2
        await asyncio.sleep(COOL_DOWN)
    print(f"Highest level that held up: {safe}")


asyncio.run(ramp())

Put three to ten representative pages from your target in SAMPLE_URLS. Each level sends ten requests per slot, so level 8 sends 80, and the whole run to level 32 sends about 600. Set CEILING to the most you would ever run, and remember that a test on a per-GB line is billed like any other traffic.

Reading the result

The script stops for one of three reasons:

• Errors above 2%. 429s, 403s, timeouts and connection errors all count. The site is pushing back.
• The median response time doubled compared with the first level. The site, or a slow exit, is queuing you, and errors usually follow.
• Pages per second rose by less than 20%. More workers are only waiting longer. This is the quiet one: nothing fails, and nothing gets faster either.

Here is a run against a local test server that handles six requests at a time and refuses anything beyond ten in flight:

  2 at once:   8.0 req/s, median 0.24s, 0% failed
  4 at once:  16.4 req/s, median 0.23s, 0% failed
  8 at once:  25.8 req/s, median 0.28s, 0% failed
 16 at once:  68.5 req/s, median 0.05s, 60% failed
  errors: the site is pushing back
Highest level that held up: 8

Notice the jump in pages per second at 16. Refusals are fast, so a failing level can look like the best one. The script checks errors first for that reason.

From the result to your settings

• Run at half to three quarters of the level that held up. The test is short; your job runs for hours, and sites tighten up as the load goes on.
• On static IPs, divide by the number of IPs if you ran the test through one of them, and give each IP its own share.
• Add a rate limit on top. Concurrency caps how much is in flight; it does nothing about speed when pages come back quickly. Fixing 429 Too Many Requests has a per-host limiter.
• Test again when things change: a new target, a new line, or a week of rising 429s.

Spreading work across static IPs

With a list of static IPs, give each one a fixed number of workers so no IP ever carries more than its share. Every worker pulls the next URL from one shared queue:

import asyncio
from collections import Counter

import httpx

PROXIES = [
    "http://USERNAME:PASSWORD@HOST_1:PORT_1",
    "http://USERNAME:PASSWORD@HOST_2:PORT_2",
    "http://USERNAME:PASSWORD@HOST_3:PORT_3",
]
PER_IP = 2


async def worker(proxy, queue, results):
    exit_label = proxy.rsplit("@", 1)[1]
    timeout = httpx.Timeout(30.0, connect=5.0)
    async with httpx.AsyncClient(proxy=proxy, timeout=timeout) as client:
        while not queue.empty():
            url = queue.get_nowait()
            try:
                response = await client.get(url)
                results.append((exit_label, url, response.status_code))
            except httpx.HTTPError as error:
                results.append((exit_label, url, type(error).__name__))


async def main(urls):
    queue = asyncio.Queue()
    for url in urls:
        queue.put_nowait(url)
    results = []
    await asyncio.gather(*(
        worker(proxy, queue, results) for proxy in PROXIES for _ in range(PER_IP)
    ))
    return results


urls = [f"https://TARGET_SITE/page/{n}" for n in range(1, 61)]
results = asyncio.run(main(urls))
print(Counter((label, status) for label, _, status in results))

PER_IP = 2 with three IPs is six requests in flight in total, never more than two per address. Against a local server that counted requests in flight, the peak was exactly six. Add IPs to go faster; raising PER_IP only makes each address more visible.

Quick answers

How many threads per proxy for residential? There is no per-IP number to find, since each connection gets a new exit. Ramp the total with the script and watch for 429s and slower responses.

How many threads per datacenter or ISP IP? Start at two per IP and ramp from there. Many sites are fine with a few; busy or defensive ones want one.

Does ProxyPanda limit threads? Our pricing has no port fees, no thread fees and no monthly minimum. The practical limit is the target site.

Will more threads make my scraper faster? Only until the site or your machine becomes the bottleneck. After that, more threads add waiting, then errors.

Start with a small test

Run the ramp script against your target through one service before you size a bigger order. Rates for each line are on the pricing page, and if the numbers you get look odd, share them in Discord along with the target and the line you used.