Java HTTP proxy authentication with HttpClient

Steps

1. Check the JDK and make a folder

   java -version shows which release will compile and run the files. Each example is a complete source file: save it in this folder and start it with java followed by its file name.

   java -version
   mkdir proxycheck && cd proxycheck

2. Build a client with a proxy and an Authenticator

   ProxySelector.of sends every request to one proxy address. The Authenticator hands over the login only when the requestor type is PROXY, so a site asking for a password never receives the proxy’s. connectTimeout limits how long reaching the proxy may take, and the timeout on the request bounds the whole wait for an answer.

   import java.net.Authenticator;
   import java.net.InetSocketAddress;
   import java.net.PasswordAuthentication;
   import java.net.ProxySelector;
   import java.net.URI;
   import java.net.http.HttpClient;
   import java.net.http.HttpRequest;
   import java.net.http.HttpResponse;
   import java.time.Duration;
   
   public class CheckIp {
       public static void main(String[] args) throws Exception {
           HttpClient client = HttpClient.newBuilder()
                   .proxy(ProxySelector.of(new InetSocketAddress("HOST", PORT)))
                   .authenticator(new Authenticator() {
                       @Override
                       protected PasswordAuthentication getPasswordAuthentication() {
                           if (getRequestorType() != RequestorType.PROXY) return null;
                           return new PasswordAuthentication("USERNAME", "PASSWORD".toCharArray());
                       }
                   })
                   .connectTimeout(Duration.ofSeconds(10))
                   .build();
   
           HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.ipify.org?format=json"))
                   .timeout(Duration.ofSeconds(30))
                   .build();
   
           HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
           System.out.println(response.statusCode() + " " + response.body());
       }
   }

3. Run it with Basic allowed for tunnels

   The JDK’s conf/net.properties sets jdk.http.auth.tunneling.disabledSchemes to Basic, so the client refuses to put a username and password on the CONNECT request that opens an HTTPS tunnel. Clearing that list on the command line lets the login through, and the program prints 200 with the JSON from api.ipify.org. Without the flag, the same file printed 407 null.

   java -Djdk.http.auth.tunneling.disabledSchemes="" CheckIp.java

4. Or clear the property from code

   When you cannot change how the program is launched, make System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "") the first line of main. The rotation example does exactly that. Editing the line in conf/net.properties also worked in testing, but it changes the rule for every program that JDK runs.

5. Spread requests over several static IPs

   ISP and datacenter services give you a set of fixed addresses. Put them in proxies.txt, one HOST:PORT:USERNAME:PASSWORD line each, and start the file with java Rotate.java. It builds one HttpClient per line, each with its own Authenticator, and gives request i to client i modulo the list length. A proxy that fails prints its exception while the loop carries on.

   import java.net.Authenticator;
   import java.net.InetSocketAddress;
   import java.net.PasswordAuthentication;
   import java.net.ProxySelector;
   import java.net.URI;
   import java.net.http.HttpClient;
   import java.net.http.HttpRequest;
   import java.net.http.HttpResponse;
   import java.nio.file.Files;
   import java.nio.file.Path;
   import java.time.Duration;
   import java.util.ArrayList;
   import java.util.List;
   
   public class Rotate {
       static HttpClient clientFor(String line) {
           String[] p = line.trim().split(":", 4);
           return HttpClient.newBuilder()
                   .proxy(ProxySelector.of(new InetSocketAddress(p[0], Integer.parseInt(p[1]))))
                   .authenticator(new Authenticator() {
                       @Override
                       protected PasswordAuthentication getPasswordAuthentication() {
                           if (getRequestorType() != RequestorType.PROXY) return null;
                           return new PasswordAuthentication(p[2], p[3].toCharArray());
                       }
                   })
                   .connectTimeout(Duration.ofSeconds(10))
                   .build();
       }
   
       public static void main(String[] args) throws Exception {
           System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
   
           List<HttpClient> clients = new ArrayList<>();
           for (String line : Files.readAllLines(Path.of("proxies.txt"))) {
               if (!line.isBlank()) clients.add(clientFor(line));
           }
   
           for (int i = 0; i < 6; i++) {
               HttpClient client = clients.get(i % clients.size());
               HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.ipify.org"))
                       .timeout(Duration.ofSeconds(30))
                       .build();
               try {
                   System.out.println(i + " " + client.send(request, HttpResponse.BodyHandlers.ofString()).body());
               } catch (Exception e) {
                   System.out.println(i + " " + e);
               }
           }
       }
   }

Common errors and fixes

Which line to pick

Java crawlers and API pollers aimed at sites that accept server traffic run cheapest on datacenter IPs, with Rotate.java sharing the load between them. Move to residential when requests from server ranges start failing, and keep an ISP IP for a service that must hold one address for weeks.

Other setup guides

Not sure what a word means? The glossary explains it in plain English.