How to check your proxy provider's usage meter

If you have ever watched a 5GB balance vanish in an afternoon of light scraping, you have had the thought: is this thing even counting properly?

Usually you cannot answer it. Most providers show one number, updated every few minutes, with nothing behind it. You are asked to trust a total you have no way to reconstruct. That is not an accusation against any particular company. It is a bad arrangement, and it is the arrangement almost the entire budget end of this market runs on.

The good news is that you can measure your own side. Here is how.

Measure your side first

You want the number of bytes your client received through the proxy: the bytes over the wire, which are neither the size of the pages nor the content length header.

In Python, wrap the response:

import requests

TOTAL = 0

def get(url, **kw):
    global TOTAL
    r = requests.get(url, **kw)
    # Response body plus the header block, which is what a proxy meters.
    header_bytes = sum(len(k) + len(v) + 4 for k, v in r.headers.items())
    TOTAL += len(r.content) + header_bytes
    return r

# ... run your job ...
print(f"{TOTAL / 1_000_000_000:.3f} GB")

In Node, if you are using undici or fetch, count the chunks as they arrive, before any buffering, so you catch responses you abandon halfway.

Two things people get wrong here:

1. Compression. If the server sent gzip and your client transparently decompressed it, len(r.content) is the decompressed size, which can be three or four times the billable figure. Set Accept-Encoding: identity while you are measuring, or count at the socket.
2. Retries. Your HTTP library may be retrying failed requests without telling you. Each attempt is traffic. Turn retries off for the test.

Then compare

Run a job big enough to matter, a gigabyte or so, and compare your total against your provider's.

A gap of a percent or two is normal and boring. It is TLS overhead, connection setup, and the difference between where you measure and where they do.

A gap of twenty percent is a conversation.

The questions worth asking

Whoever you buy from, these four questions separate a meter from a number:

• Can I see per-request usage, or only a total? A total cannot be checked. Rows can.
• Which failed requests get billed? A timeout or a reset brought nothing back from the site. A 429 or a block page did, and most meters count it. Ask where the line sits.
• Are request bytes billed, or only response bytes? Both are defensible. Not saying which is not.
• Can I export the log? If the answer is no, you have no way to audit anything, ever.

If a provider cannot answer those in one line each, that tells you something on its own.

What we do

We log every request separately with its ID, host, country, status code and byte count, and you can export the lot as CSV. Connection failures (timeouts, resets and errors from our own gateway) bill zero and stay on the log marked free, so you can see they happened and see they cost nothing. A 429, a 5xx or a block page the site sends back is traffic, like any other response.

And the standing offer: if your count and ours differ by more than the tolerance written on the honesty page, send us both figures and we re-bill the session at yours while we work out the difference. You do not have to prove we are wrong first.

That is written up properly on the honesty page.

Run the test above against us. That is what it is for.