DEV Community

Cover image for What to do when Go ignores HTTP_PROXY for 127.0.0.1
Maël Valais
Maël Valais

Posted on • Updated on • Originally published at maelvls.dev

What to do when Go ignores HTTP_PROXY for 127.0.0.1

I use Proxyman for inspecting the HTTP and HTTPS traffic coming from applications. For example, you may want to know which API calls are made by docker when running docker search. What I would do is

HTTPS_PROXY=http://localhost:9090 docker search ubuntu
Enter fullscreen mode Exit fullscreen mode

Since Proxyman acts like a proxy listening on 0.0.0.0:9090, I can see and replay all the HTTP calls, similarly to what you would do with the Chrome DevTools.

Screenshot with macOS' Proxyman for a local Openstack instance

Note that HTTPS is a bit tricky and requires an additional step on macOS (settings > Proxyman CA > Generate new) but works flawlessly with any Go binary since Go supports HTTPS_PROXY by default. For Python, it becomes a bit trickier often requires the app to have a flag like --cacert.

But by default, Go [ignores HTTPS_PROXY] for localhost and 127.0.0.1 hosts (see ProxyFunc and useProxy):

As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned.

Note that the documentation forgets to mention that 127.0.0.1 (and related) are also ignored. That's a bummer since using a proxy may be the only way to debug a CLI or an application that makes network calls without logging properly.

Two solutions: either you have access (and are willing to) hack the source code in order to force Go to use the proxy with localhost anyway, or you can hack your /etc/hosts and use a different host.

Solution 1: hack the proxy setup by tweaking http.Transport

You need to have access to the place where the http.Client is set. Imagine that you are working on something that uses the Openstack API. You would have something like:

package main

import (
    "fmt"
    "os"

    "github.com/gophercloud/gophercloud"
    "github.com/gophercloud/gophercloud/openstack"
    "github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
)

func main() {
    pr, err := openstack.NewClient("http://localhost/identity/v3")
    if err != nil {
        fmt.Fprintf(os.Stderr, "could not create an openstack client")
        os.Exit(1)
    }

    err = openstack.Authenticate(pr, gophercloud.AuthOptions{
        IdentityEndpoint: "http://localhost/identity/v3",
        DomainName:       "Default",
        Username:         "admin",
        Password:         "secret",
        TenantName:       "admin",
    })
    if err != nil {
        fmt.Fprintf(os.Stderr, "auth failed")
        os.Exit(1)
    }

    compute, err := openstack.NewComputeV2(pr, gophercloud.EndpointOpts{Region: ""})
    srvs, err := servers.List(compute, nil).AllPages()
    if err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }

    fmt.Printf("servers:\n%v\n", srvs)
}
Enter fullscreen mode Exit fullscreen mode

We must be able to swap the http.Transport used. The http.DefaultTransport uses ProxyFromEnvironment:

// DefaultTransport is the default implementation of Transport and is
// used by DefaultClient. It establishes network connections as needed
// and caches them for reuse by subsequent calls. It uses HTTP proxies
// as directed by the $HTTP_PROXY and $NO_PROXY (or $http_proxy and
// $no_proxy) environment variables.
var DefaultTransport RoundTripper = &Transport{
    Proxy: ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
        DualStack: true,
    }).DialContext,
    ForceAttemptHTTP2:     true,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}
Enter fullscreen mode Exit fullscreen mode

Let's change ProxyFromEnvironment with something of our own:

Proxy: func(req *http.Request) (*url.URL, error) {
    return url.Parse("http://127.0.0.1:9090")
},
Enter fullscreen mode Exit fullscreen mode

Now, the code looks like this:

package main

import (
    "fmt"
    "net"
    "net/http"
    "net/url"
    "os"
    "time"

    "github.com/gophercloud/gophercloud"
    "github.com/gophercloud/gophercloud/openstack"
    "github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
)

func main() {
    pr, err := openstack.NewClient("http://localhost/identity/v3")
    if err != nil {
        fmt.Fprintf(os.Stderr, "could not create an openstack client")
        os.Exit(1)
    }
    pr.HTTPClient.Transport = &http.Transport{
        Proxy: func(req *http.Request) (*url.URL, error) {
            return url.Parse("http://127.0.0.1:9090")
        },
        DialContext: (&net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 30 * time.Second,
            DualStack: true,
        }).DialContext,
        ForceAttemptHTTP2:     true,
        MaxIdleConns:          100,
        IdleConnTimeout:       90 * time.Second,
        TLSHandshakeTimeout:   10 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }

    err = openstack.Authenticate(pr, gophercloud.AuthOptions{
        IdentityEndpoint: "http://localhost/identity/v3",
        DomainName:       "Default",
        Username:         "admin",
        Password:         "secret",
        TenantName:       "admin",
    })
    if err != nil {
        fmt.Fprintf(os.Stderr, "auth failed")
        os.Exit(1)
    }

    compute, err := openstack.NewComputeV2(pr, gophercloud.EndpointOpts{Region: ""})
    srvs, err := servers.List(compute, nil).AllPages()
    if err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }

    fmt.Printf("servers:\n%v\n", srvs)
}
Enter fullscreen mode Exit fullscreen mode

Now, all traffic including calls to Host: localhost or Host: 127.0.0.1 will be properly proxied to localhost:9090.

Solution 2: use a different host name

In your /etc/hosts, add the line such as local 127.0.0.1. Now, you can use the host local instead of localhost, which means Go will proxy your requests properly.

Top comments (0)