Problem

As evidenced by recent engagements, customers in the 💰 financial services space have shown a growing interest in applying additional layers of protection to their public-facing APIs. This is on top of the safety blanket afforded by current mechanisms in place. By current mechanisms, what is referred to is a set of decade old security practices that, although effective, require additional layers of Swiss cheese. In this day and age, malicious entities (inside or outside of organizations) have significantly broadened the attack surface and organizations can no longer rely on standard credential challenges. This is the reason why multi-factor authentication has become a hard requirement in recent years. However, what are the options available for machine-to-machine integration?

One already available option is to use a combination of 🔑 public-key cryptography and 🔑 symmetric-key cryptography with the goal of adding another layer of Swiss cheese (so to speak) as a moat around the current security infrastructure.

Public Key Cryptography

This is also known as asymmetric cryptography as the keys to encrypt and decrypt the message is not identical but is linked mathematically. The public key for encryption is actually generated from the private key. RSA is a common cryptosystem that falls under this category. Asymmetric encryption is generally slower than symmetric key encryption.

Symmetric Key Cryptography

As the name suggests, the encryption and decryption keys are identical in this case. A good example of symmetric key cryptography is AES. This method of encryption is generally faster but its use case is severely limited by the use of a single key as malicious attackers need to only secure one key to pose a threat.

Hybrid Cryptography

A 🔑 hybrid cryptosystem combines the security afforded by a public-key cryptosystem with the efficiency of a symmetric-key cryptosystem.

To secure the message being sent across the wire:

1. The ciphertext will be created by encrypting the message using a symmetric key.
2. The symmetric key will, in turn, be encrypted using a public key.
3. The ciphertext will be sent with the encrypted symmetric key to the intended recipient that bears the corresponding private key.
4. The recipient will decrypt the symmetric key first by using his private key and then use the symmetric key to decrypt the encrypted payload.

This approach is able to leverage the raw performance of symmetric keys while mitigating the risks associated with using a single key through asymmetrical keys that lock down the symmetric key.

This approach does come with a slight disadvantage. There is no standardized way of transmitting the encrypted key and payload. Because of this, developers for years have become very creative when it comes to transmitting both artifacts. Some would plug them in to custom response headers. Others would dump everything including the initialization vector in the HTTP response body while using obscure names to mislead prying eyes from the encrypted key. This manner of security by obscurity is rarely helpful long term.

Solution

Conceptually, the combination of symmetric and asymmetric encryption addresses the problem above. However, it is becoming clear that any advantage the hybrid approach has is greatly diminished by the lack of protocols to follow. Luckily, this is no longer the case. RFC 7516 proposes a clever alternative - JSON Web Encryption (JWE). JWE takes off from where JSON Web Tokens (JWT) left off to provide a standard way of carrying over the encrypted key and payload to the intended recipient.

JWE represents encrypted content using JSON data structures and base64url encoding. These JSON data structures MAY contain whitespace and/or line breaks before or after any JSON values or structural characters, in accordance with Section 2 of RFC 7159.

It is not the goal of this article to discuss JWE at length. The RFC is available for that purpose as well as the massive body of literature already available online about the subject. This article will, however, run through the salient points to provide proper context to the source code to be discussed in later sections.

The JWE specification standardizes the way to represent an encrypted content in a JSON-based data structure. The JWE specification defines two serialized forms to represent the encrypted payload:

1. Compact serialization
2. JSON serialization

This article will be focusing on the former. It should be noted that the message to be encrypted using the JWE standard need not be a JSON payload. It can be any content. The term JWE token is only used to refer to the serialized form of an encrypted message following any of the serialization techniques defined in the JWE specification.

JWE Token Structure

JWE using compact serialization uses the data structure below. All components are base64url-encoded prior to transmission.

1. JOSE Header - This contains two algorithms. The first one is used to encrypt the content encryption key and should be from the JSON Web Signature and Encryption Algorithms registry defined by the JSON Web Algorithms (JWA) specification. The content encryption key is going to be used to encrypt the actual payload. The second algorithm is the one used by the content encryption key for actual encryption. This algorithm should be a symmetric Authenticated Encryption with Associated Data (AEAD) algorithm.
2. JWE Encrypted Key - This contains the content encryption key encrypted using the first algorithm identified in the JOSE header.
3. Initialization Vector - Add randomness to the encrypted data.
4. Ciphertext - The ciphertext is computed by encrypting the payload using the JWE Encrypted Key and Initialization Vector with the second encryption algorithm defined in the JOSE header.
5. Authentication Tag - The authentication tag ensures the integrity of the ciphertext and is generated with the ciphertext using the second algorithm mentioned in the JOSE header.

Implementation

Creating an HTTP Server Using NET/HTTP

Create a ping handler function to be registered to a URL route later.

func ping(w http.ResponseWriter, req *http.Request) {
    out := time.Now().Format(time.RFC1123)
    w.Header().Set("Content-Type", "text/plain")
    w.Write([]byte("Pinged on: " + out))
}

Initialize mux as an HTTP request multiplexer that will monitor a predefined set of URLs and invoke the corresponding handler function. Create an HTTP server and assign mux as the handler.

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /ping", ping)

    server := http.Server{
        Addr:         ":8080",
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 90 * time.Second,
        IdleTimeout:  120 * time.Second,
        Handler:      mux,
    }

    err := server.ListenAndServe()
    if err != nil {
        if err != http.ErrServerClosed {
            panic(err)
        }
    }
}

Run a quick test against the 🚀 ping endpoint.

GET /ping HTTP/1.1
Host: localhost:8080

The 🚀 ping endpoint should yield the raw response below.

Pinged on: Fri, 29 Mar 2024 13:58:40 CST

Using Middleware Components to Add Non-Standard Functionality

Create the following handler functions that will compose a semi-realistic request pipeline. One of the handler functions here (🛡️ protect) will take care of creating the JWE token that contains the encrypted payload. Although encryption can be done on each API endpoint, it makes more sense to consolidate the entire process into a middleware component to keep the blast radius manageable in the event that changes need to be made. Furthermore, in a multi-tenant setup, the same encryption key will be used for all API endpoints for every tenant anyways.

📔 Audit

audit is the handler function that will serve as the entry point of the request pipeline. Its purpose is to create an audit trail for API invocations. It creates a correlation id and passes it into the HTTP context for use by subsequent middleware components.

func audit(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        id := uuid.NewString()
        fmt.Printf("Correlation Id %s\n", id)
        ctx := context.WithValue(req.Context(), "correlation_id", id)

        // Write request details to log file
        // Write response details to log file
        h.ServeHTTP(w, req.WithContext(ctx))
    })
}

🔒 Authenticate

authenticate intends to centralize all authentication and authorization operations into this middleware component. In real-world implementations, cross-cutting functions like these are relegated to an API gateway. authenticate was only added to the mix for illustration purposes.

func authenticate(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        id := req.Context().Value("correlation_id")
        fmt.Printf("Authenticating %s\n", id)

        // Verify the API key used
        h.ServeHTTP(w, req)
    })
}

♻️ Transform Request

transformRequest is another cross-cutting function that can be moved to an API gateway but is added here because there are still use cases where the HTTP request body still needs to be manipulated but the operation is too expensive to be done on the API gateway.

func transformRequest(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        id := req.Context().Value("correlation_id")
        fmt.Printf("Transforming %s\n", id)

        // Clean up bad data
        h.ServeHTTP(w, req)
    })
}

💥 Do

do contains the actual process to be executed by the API call. For simplicity, the code below just creates a simple JSON structure and passes it on to the next middleware component.

func do(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        id := req.Context().Value("correlation_id")
        body := make(map[string]string)
        body["correlation_id"] = id.(string)
        body["raw"] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
        body["current_time"] = time.Now().Format(time.RFC1123)

        res, err := json.Marshal(body)
        if err != nil {
            panic(err)
        }

        ctx := context.WithValue(req.Context(), "data", res)
        fmt.Printf("Printing raw reponse for %s  >>>  %s\n", id, res)

        h.ServeHTTP(w, req.WithContext(ctx))
    })
}

🛡️ Protect

protect is the handler function that wraps the actual payload from do inside a JWE token.

func protect(w http.ResponseWriter, req *http.Request) {
    id := req.Context().Value("correlation_id")
    fmt.Printf("Using JWE on %s\n", id)

    data := req.Context().Value("data").([]byte)

    path := `public.pem`
    publicKeyPEM, err := ioutil.ReadFile(path)
    if err != nil {
        panic(err)
    }
    publicKeyBlock, _ := pem.Decode(publicKeyPEM)
    publicKey, err := x509.ParsePKIXPublicKey(publicKeyBlock.Bytes)
    if err != nil {
        panic(err)
    }

    recipient := jose.Recipient{Algorithm: jose.RSA_OAEP_256, Key: publicKey}
    encrypter, err := jose.NewEncrypter(jose.A256GCM, recipient, nil)
    if err != nil {
        panic(err)
    }

    cipher, err := encrypter.Encrypt(data)
    if err != nil {
        panic(err)
    }

    serialized, err := cipher.CompactSerialize()
    if err != nil {
        panic(err)
    }

    fmt.Println(serialized)
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(serialized))
}

protect uses RSA-OAEP-256 to encrypt the content encryption key. The content encryption key, on the other hand, uses A256GCM to create the ciphertext. It should be noted that protect assumes that there is an existing public key in the local file system in PEM format. It takes care of extracting the public key from the PEM file and cloaks the content encryption key. The above code is not transparent when it comes to the generation of the symmetric key as it uses an external package - go-jose - to create it. Once the encryption of the payload and content encryption keys are done, the referenced package performs the compact serialization process that results in the format described in a previous section - JWE Token Structure.

Employing JWE in the HTTP Server to Secure Restricted Content

Now that all the handler functions have been created, the only task left is to wire everything together into a contiguous request pipeline. Pay close attention to the new HTTP endpoint created below - 🚀 secure-message.

func main() {
    mux := http.NewServeMux()

    mux.HandleFunc("GET /ping", ping)
    mux.Handle("POST /secure-message", audit(authenticate(transformRequest(do(http.HandlerFunc(protect))))))

    server := http.Server{
        Addr:         ":8080",
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 90 * time.Second,
        IdleTimeout:  120 * time.Second,
        Handler:      mux,
    }

    err := server.ListenAndServe()
    if err != nil {
        if err != http.ErrServerClosed {
            panic(err)
        }
    }
}

🚀 secure-message strings together closures to form the request pipeline. In a previous section - Using Middleware Components to Add Non-Standard Functionality, handler functions were created as middleware components. Each subsequent function will be passed in as a parameter of the previous function to build a pipeline of connected activities that possess a shared context.

Run a quick test against the 🚀 secure-message endpoint.

POST /secure-message HTTP/1.1
Host: localhost:8080

The above request would yield the response below.

eyJhbGciOiJSU0EtT0FFUC0yNTYiLCJlbmMiOiJBMjU2R0NNIn0.1SkP54dymMXE0UvLpN4BwFeV5YikSqKLGEmiSZxG6pcZ1lghiGYApHVBjBmvFxLQmVsCBMhpkQWT7V5ICrmmR2LQ2K8MsQ-OUrPtiDXzJNO2EjHMihSbysqC91iZ-ODuAFqbmOdzRPNI4GTKfho-i_PzyRygqeGHXCHaEyDSBhhWmFskbdwqDH3zKw_2HE-bUmr-yJg8wcCyaG9By-5NiF5KypjxTk_b3GTwlVzKLolH4QErZd5qds9-sH96iq5-5OcjJt9yxNkxCJcgZ-PbLDfDxNF1WKb0sfG0u9TAWhUn_80bSb7VODcuzpBZT9gi6PYL1CUGG_yO88QGMLBHCA.OYTcnZfeoV3ol3bl.54uE4oo82kHgsNaNridBtyv9itiqIBDVv7sVYYLLDEbFy6cLTNDlXd3gu6Am1XfsKTVz8gMgI7i__tYbvcsoeNI0D3l1LVTvDqwDJCtH7BPSEJyy0eivFHq2wZuYGLnpOcB9qaRO3-4Y327akSOrh9EdY1t6z81KtPlkYw1uTINFE5yZoX-7mG0pANfXgslGe_mZQrOe7TarsznwZcIJQqDFptZn3s9SpTkwMiwxpNaEW5AZdJvj6YG3LeLwZw3SoGGSGgKyHBqdi3BPldp783MKg7W8uA_UU3DoBSi7miDtY8Pxr9MIN7zeOK52qzg2qM7xO_DWKo5vfPijiEJF7glgALVurlIPkGyXwkycEihuMs2AIP1d1uLnd8cIlPfBaKRBKB6CtYeqd25zPn_jaNyuqlI4DrLxBwgIgll3WUrnYSinIdlRBAuKq6nxSEZhkgzX1v4Qrvl5anetw1BxXS6GvhayJUxz8iP_9NB7LI4-YERdySfRwf-HlSy9wVhcS6xsjmx5lizRFEhlEU3kEYWIYKLZDYSLhGSYod_lW0rBRxvPmoKnNBNVcQbL7OGvu9f6eFdHf9biFWkRpuHsYWLeRP9zH8-lfwdjRuNWFXjmK-bpjlLPkR0KYrRHgNnSbU3ZZMzewL67uvOWSce70OxYyeSvLM-scF87FLnKwpdzzI38ek8sjOgk_fqfiTUK4W4YI1W3TSpEoF_Lmx9L7XvGBOiWQSxGo8JewnKX.1O4smA_u2KTQ-KZCn0GGuA

Creating an HTTP Client to Consume the Encrypted Content

To close the loop, below is a sample HTTP client that invokes 🚀 secure-message and decrypts the HTTP response using the private key used to generate the public key in the previous section - 🛡️ Protect.

func printDecryptedContent(client *http.Client, url string) {
    req, err := http.NewRequestWithContext(context.Background(), "POST", url, nil)
    if err != nil {
        panic(err)
    }

    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }

    defer res.Body.Close()
    if res.StatusCode != http.StatusOK {
        panic(fmt.Sprintf("Unexpected Status: %v", res.Status))
    }

    resBytes, err := ioutil.ReadAll(res.Body)
    if err != nil {
        panic(err)
    }
    resString := string(resBytes)

    alg := []jose.KeyAlgorithm{jose.RSA_OAEP_256}
    enc := []jose.ContentEncryption{jose.A256GCM}
    jwe, err := jose.ParseEncrypted(resString, alg, enc)
    if err != nil {
        panic(err)
    }

    path := `private.pem`
    privateKeyPEM, err := ioutil.ReadFile(path)
    if err != nil {
        panic(err)
    }

    privateKeyBlock, _ := pem.Decode(privateKeyPEM)
    privateKey, err := x509.ParsePKCS1PrivateKey(privateKeyBlock.Bytes)
    if err != nil {
        panic(err)
    }

    decrypted, err := jwe.Decrypt(privateKey)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(decrypted))
}

func main() {
    client := &http.Client{
        Timeout: 30 * time.Second,
    }

    printDecryptedContent(client, "http://localhost:8080/secure-message")
}

Generating the RSA Private and Public Keys

The sample application above assumes that the private and public keys are already present in the file system. Standard commandline utilities can be used to generate them. As an alternative, the keys can also be generated using the Go standard library. Below is an example of how to do this.

func main() {
    privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
    if err != nil {
        panic(err)
    }

    publicKey := &privateKey.PublicKey

    privateKeyBytes := x509.MarshalPKCS1PrivateKey(privateKey)
    privateKeyPEM := pem.EncodeToMemory(&pem.Block{
        Type:  "RSA PRIVATE KEY",
        Bytes: privateKeyBytes,
    })
    err = os.WriteFile("private.pem", privateKeyPEM, 0644)
    if err != nil {
        panic(err)
    }

    publicKeyBytes, err := x509.MarshalPKIXPublicKey(publicKey)
    if err != nil {
        panic(err)
    }
    publicKeyPEM := pem.EncodeToMemory(&pem.Block{
        Type:  "RSA PUBLIC KEY",
        Bytes: publicKeyBytes,
    })
    err = os.WriteFile("public.pem", publicKeyPEM, 0644)
    if err != nil {
        panic(err)
    }
}

Summary

JSON Web Encryption (JWE) can be used to protect restricted content exposed through public APIs. JWE is a powerful alternative to arbitrary structures that make up most secure envelope implementations. JWE can be easily implemented in Golang today using contemporary packages like go-jose. 💪🏻🍾🥇