Crawl4AI proxy setup with ProxyConfig

Steps

1. Install Crawl4AI in a virtual environment

   crawl4ai-setup downloads the Chromium build that Playwright drives and prepares Crawl4AI’s local database. Keeping it all in .venv stops one project’s browser and packages from colliding with another’s.

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -U crawl4ai
   crawl4ai-setup

2. Give BrowserConfig a ProxyConfig

   server carries the scheme, host and port and nothing more. username and password sit in separate fields, and the browser answers the proxy’s login challenge with them. Every page this crawler opens goes through the proxy.

   import asyncio
   
   from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig
   
   proxy = ProxyConfig(server="http://HOST:PORT", username="USERNAME", password="PASSWORD")
   browser_config = BrowserConfig(headless=True, proxy_config=proxy)
   
   
   async def main():
       async with AsyncWebCrawler(config=browser_config) as crawler:
           result = await crawler.arun("https://api.ipify.org?format=json", config=CrawlerRunConfig())
           print(result.success, result.status_code)
           print(result.markdown)
   
   
   asyncio.run(main())

3. Read what came back

   The script prints True 200, then the JSON from api.ipify.org wrapped in a markdown code block. The ip value is the proxy’s exit address.

4. Rotate across a list with RoundRobinProxyStrategy

   ProxyConfig.from_string reads a line in HOST:PORT:USERNAME:PASSWORD order, the order the dashboard exports. The strategy gives each arun call the next entry, and the log prints a Switch proxy line every time. page_timeout is in milliseconds.

   import asyncio
   
   from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, RoundRobinProxyStrategy
   
   with open("proxies.txt") as f:
       proxies = [ProxyConfig.from_string(line) for line in f if line.strip()]
   
   run_config = CrawlerRunConfig(
       proxy_rotation_strategy=RoundRobinProxyStrategy(proxies),
       check_robots_txt=True,
       page_timeout=60000,
   )
   
   
   async def main():
       urls = ["https://api.ipify.org?format=json"] * 6
       async with AsyncWebCrawler() as crawler:
           for url in urls:
               result = await crawler.arun(url, config=run_config)
               print(result.success, result.markdown if result.success else result.error_message)
   
   
   asyncio.run(main())

5. Keep the crawl polite

   An AI crawler is still a crawler: robots.txt and a site’s terms apply to it as to any scraper, and a steady pace is kinder to the site than a burst. check_robots_txt=True skips disallowed URLs. In 0.9.4 that robots.txt request goes out directly from your machine, not through the proxy, with a two-second limit, and a file it cannot fetch counts as allowing everything.

Common errors and fixes

Which line to pick

Crawls that feed a model often read many pages once each. Datacenter IPs keep that cheap on sites that accept server traffic. Use residential for sites that block data-centre ranges, and keep pages light while you do: residential is billed by the gigabyte, and text_mode=True on BrowserConfig tells the browser to skip images.

Other setup guides

Not sure what a word means? The glossary explains it in plain English.