Golang HTTP proxy with net/http and auth

Steps

1. Create a module

   go version confirms which toolchain runs the code. Each example below is a complete main.go for this folder, so replace the file each time and run it with go run .

   go version
   mkdir proxycheck && cd proxycheck
   go mod init proxycheck

2. Point the Transport at the proxy

   http.ProxyURL wraps a fixed URL in the function the Transport calls for each request. url.UserPassword holds the login and percent-encodes it when the URL is written out, so a password with @, : or / needs no escaping by hand. Client.Timeout caps the whole exchange, the CONNECT to the proxy included.

   package main
   
   import (
   	"fmt"
   	"io"
   	"net/http"
   	"net/url"
   	"time"
   )
   
   func main() {
   	proxyURL := &url.URL{
   		Scheme: "http",
   		User:   url.UserPassword("USERNAME", "PASSWORD"),
   		Host:   "HOST:PORT",
   	}
   
   	client := &http.Client{
   		Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
   		Timeout:   30 * time.Second,
   	}
   
   	resp, err := client.Get("https://api.ipify.org?format=json")
   	if err != nil {
   		fmt.Println("request failed:", err)
   		return
   	}
   	defer resp.Body.Close()
   
   	body, _ := io.ReadAll(resp.Body)
   	fmt.Println(resp.Status, string(body))
   }

3. Run it and check the exit

   go run . prints 200 OK followed by the JSON from api.ipify.org. The ip field is the address the site saw, which should belong to the proxy.

4. Set Transport timeouts and a new connection per request

   The Dialer timeout limits how long reaching the proxy may take, TLSHandshakeTimeout covers the handshake with the site inside the tunnel, and ResponseHeaderTimeout the wait for the first response byte. DisableKeepAlives closes each connection after one request: in testing, three requests opened three CONNECT tunnels with it and a single tunnel without it.

   package main
   
   import (
   	"fmt"
   	"io"
   	"net"
   	"net/http"
   	"net/url"
   	"time"
   )
   
   func main() {
   	proxyURL := &url.URL{
   		Scheme: "http",
   		User:   url.UserPassword("USERNAME", "PASSWORD"),
   		Host:   "HOST:PORT",
   	}
   
   	transport := &http.Transport{
   		Proxy:                 http.ProxyURL(proxyURL),
   		DialContext:           (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
   		TLSHandshakeTimeout:   10 * time.Second,
   		ResponseHeaderTimeout: 30 * time.Second,
   		DisableKeepAlives:     true,
   	}
   	client := &http.Client{Transport: transport, Timeout: 60 * time.Second}
   
   	for i := 0; i < 3; i++ {
   		resp, err := client.Get("https://api.ipify.org")
   		if err != nil {
   			fmt.Println(i, "error:", err)
   			continue
   		}
   		body, _ := io.ReadAll(resp.Body)
   		resp.Body.Close()
   		fmt.Println(i, string(body))
   	}
   }

5. Read the proxy from HTTPS_PROXY instead

   http.ProxyFromEnvironment picks HTTPS_PROXY for https URLs and HTTP_PROXY for http ones, and skips hosts listed in NO_PROXY. http.DefaultTransport already calls it. A Transport you build yourself does not: with no Proxy field, it went straight out even with the variable exported.

   package main
   
   import (
   	"fmt"
   	"io"
   	"net/http"
   	"time"
   )
   
   func main() {
   	client := &http.Client{
   		Transport: &http.Transport{Proxy: http.ProxyFromEnvironment},
   		Timeout:   30 * time.Second,
   	}
   
   	resp, err := client.Get("https://api.ipify.org")
   	if err != nil {
   		fmt.Println("request failed:", err)
   		return
   	}
   	defer resp.Body.Close()
   
   	body, _ := io.ReadAll(resp.Body)
   	fmt.Println(string(body))
   }

6. Export the variable and run

   The login stays in the variable, so the Go code carries no credentials. The URL here is plain text, so percent-encode any @, : or / in the password yourself. Unset the variable afterwards if other tools share the shell.

   export HTTPS_PROXY="http://USERNAME:PASSWORD@HOST:PORT"
   go run .

Common errors and fixes

Which line to pick

Go workers that poll APIs or pull pages in bulk run cheapest on datacenter IPs wherever the target accepts them. Move to residential once requests from server ranges get refused, and give a long-lived worker an ISP IP when it must keep one address across sessions.

Other setup guides

Not sure what a word means? The glossary explains it in plain English.