DEV Community

Cover image for Day 23: The API Payload Security Pipeline: Order, Protection Levels and Keys
Jeff Tsai
Jeff Tsai

Posted on Originally published at hackmd.io

Day 23: The API Payload Security Pipeline: Order, Protection Levels and Keys

The previous post, The Division of Labor Between XML, JSON and MessagePack, closed by saying the other half of value comes here. Encoded and Encrypted differ by one number on the API payload, and underneath that number are three stages that run in a fixed order, while on the way back the order is more than the same run in reverse.

An ERP system's API has to answer two questions that are independent of each other: what a call's content should be processed into, and who is entitled to decide it may be processed less. The first is a fixed pipeline, the second is one declaration written onto the method. The two are kept apart, and are compared against each other at one stop only, which is access control.

This post covers:

  1. The order of serialization, compression and encryption, and what breaks when it is swapped
  2. That ApiProtectionLevel and PayloadFormat are two different things, and how access control compares them
  3. What actually sits under the name aes-cbc-hmac
  4. Where keys come from, and why cryptographic primitives and security policy live in separate layers

1. The order of the three stages

An Encoded value goes out as a Base64 string. Getting from an object to that string takes three stages, each carried by its own interface and each replaceable as a whole:

Stage What it does Current implementation Can be replaced with
Serialization Turns the object into bytes messagepack json
Compression Makes those bytes shorter gzip none
Encryption Turns bytes into bytes carrying an authentication code aes-cbc-hmac none

The latter two names are in the deployment's SystemSettings, while the first is declared by the caller on the API payload of every individual call, with no declaration meaning messagepack. That split is what the previous post's three components and two ways of deciding was about. What can be replaced is each stage's implementation, not the order they run in. The encoding side, ApiPayloadTransformer, is two lines, and the source puts the order at the end of each:

byte[] bytes = serializer.Serialize(payload, type);              // Serialize
return ApiServiceOptions.PayloadCompressor.Compress(bytes);      // Compress
Enter fullscreen mode Exit fullscreen mode

Encryption is not in those two lines. It is one further step the layer outside decides on according to the format: Encoded stops here, and Encrypted goes one stage further.

Compression comes before encryption because compression does nothing at all to ciphertext. The output of good encryption is statistically indistinguishable from random data, and repetition and skew are exactly what a compression algorithm looks for. Running the case study's order FormSchema through both orders shows it:

Order Result
Serialize, compress, then encrypt Leaves a little over a quarter after compression, with encryption adding a few dozen bytes on top
Serialize, encrypt, then compress Comes out larger after compression than it went in, and the final size is more than three times the first

Swapping the order makes no call fail, and no check anywhere says a word. It just means the compression stage stops doing anything from then on.

The way back is those three steps reversed: decrypt, decompress, deserialize. But what matters about the reverse direction is not that it runs backwards, it is that each stage only ever sees bytes that got past the one before it. The authentication code is verified before decryption, and decryption comes before decompression, so the gzip decompressor never at any point holds a stretch of data that has not been verified.

A decompressor is a parser that accepts arbitrary bytes. Feed it ciphertext with one bit changed and it will start parsing all the same, then throw because it cannot read a structure out of it. This order means it never gets that far.


2. The level declared, and the format actually used

The three values of PayloadFormat were covered in that same earlier post. What access control looks at is not that one on its own, it is the result of comparing it with another enum, and that one has four values.

The question it answers Values Who decides
PayloadFormat What this call actually did Three The caller, writing it onto the API payload
ApiProtectionLevel The least this method will accept Four Whoever writes the method, attaching it to the method or to the type

Two of the values share a name across the two enums, being Encoded and Encrypted, which makes it easy to read them as two ways of saying the same thing. They are not: one records a fact, the other declares a requirement.

What the four levels ask for, and the kind of method each suits:

Level Meaning Where it suits
Public HTTPS is enough Calls where the server seeing the content does no harm, such as the connectivity probe Ping
Encoded At least encoded Content that should not be readable straight out of a packet capture, but that is not worth paying for encryption
Encrypted Encrypted Content that should not be plain text inside the server either, such as managing API keys
LocalOnly Remote calls are not accepted Things only the host process itself should start, such as writing a definition

How the framework's own methods are distributed across those four is itself a decision: the trust boundary is drawn at HTTPS, and only the handful whose authors judged explicitly that the content must not be visible once inside the server are marked Encrypted. Nothing declares the middle level at present, which is where the format side lands rather than something the level side asks for.

Drawing the boundary at HTTPS does not mean application layer encryption is pointless. HTTPS protects only the stretch where data travels over the network, and it is unwrapped the moment it reaches the server, so every stop that handles it after that sees plain text. A node gets compromised, logging and monitoring keep the content, something is intercepted and replayed across network segments or over a VPN: all of these are real in actual ERP deployments.

Drawing the line at Public buys one thing: a front end with no .NET runtime does not have to implement the whole encryption pipeline before it can reach its first method. The cost is that for calls at that level, value is visible once it is inside the server, and a method needing that layer has to be marked for it.

The test is that the actual format has to be greater than or equal to the declared level: doing more is fine, doing less is refused. What follows is all about remote calls, and a remote call gets past LocalOnly at no format level at all:

Actual format \ Declared level Public Encoded Encrypted LocalOnly
Plain Passes Refused Refused Refused
Encoded Passes Passes Refused Refused
Encrypted Passes Passes Passes Refused

Firing at the case study's endpoint, the three refusals each carry their own message:

Plain    calling System.ListApiKeys → This API requires encoded or encrypted transmission.
Encoded  calling System.ListApiKeys → This API requires encrypted transmission.
Plain    calling System.SaveDefine  → This API is restricted to local calls only.
Enter fullscreen mode Exit fullscreen mode

The other half, that doing more is fine, has been measured too: send an Encoded API payload to the Public list query and it passes, and what comes back is Encoded as well. The server does not touch the format, it wraps the response back up using the value the caller sent, so both directions of one round trip are always the same format.

LocalOnly is not stricter encryption. It asks something else: this method does not arrive over the network at all. Not being on the same axis, it is not compared with the format for size, it is judged on its own. The Connector and the Convergence of Client Responsibilities mentioned that a few server side methods accept in-process calls only, and this is the slot it meant. They can still be called, just not over the network. A local call is let through whole the moment it reaches this check, without the format being looked at, so the table above does not govern them.

This check can sit ahead of decryption because both values it compares are available from the start: the format is a number on the API payload, and the level is attached to the type and fixed once compiled. Neither side requires opening value and looking inside. JSON-RPC 2.0: Protocol Choice and the Request Pipeline said access control sits ahead of restoration and does no decryption work for a request that has not passed, and this is the condition that makes it hold.

A method with no declaration is not unrestricted, it throws. The search order is the method itself, then the one it overrides, then the type declaring it. The cost is that a missing declaration only surfaces when somebody calls it for the first time, so there is a separate build time rule that points such methods out.


3. What sits under the name aes-cbc-hmac

The name puts three parts together: a symmetric encryption algorithm, a mode of operation, and a message authentication code. In the implementation it comes to four decisions, and the body of the encrypting and decrypting is AesCbcHmacCryptor:

  • What goes in is one 64 byte key, cut in two and used as two: the first 32 as the AES key, the last 32 as the HMAC key. One key does not hold both jobs
  • Every encryption generates a new IV on the spot and writes it into the output in the clear, because decryption needs it. Encrypting the same data with the same key twice in a row was measured to produce different bytes
  • Encrypt first, then stamp an authentication code over the encrypted result, not the other way round. What that buys is a decrypting end that turns data away before decrypting it, whereas the version done the other way round has to decrypt before it can verify, which amounts to letting the decryptor eat a stretch of data of unknown origin first. That line from the first section, each stage only sees bytes that got past the one before it, starts here
  • Verifying the authentication code uses a fixed time comparison, not ==. An ordinary loop returns at the first byte that differs, which turns how many bytes were guessed right into something measurable from outside, and lets an attacker assemble the authentication code one byte at a time

That fixed time comparison is three lines in the decryption stage:

byte[] computedHmac = hmac.ComputeHash(dataToVerify);
if (!CryptographicOperations.FixedTimeEquals(hmacBytes, computedHmac))
    throw new CryptographicException("HMAC validation failed.");
Enter fullscreen mode Exit fullscreen mode

The message itself hides a choice too. Three different ways of breaking it were measured: one bit changed in the ciphertext, one bit changed in the authentication code, and an entirely unrelated key. All three come back with the same HMAC validation failed.. Being unable to tell them apart is correct. Being able to distinguish a wrong authentication code from a wrong key is itself a signal that can be probed.


4. Where the keys come from

Getting all four of the previous section's decisions right still leaves one thing unanswered: who supplies that 64 byte key. The framework keeps this and the previous section in two separate layers:

Layer Location What it holds Examples
Cryptographic primitives Bee.Base Computing an output from an input, with no business meaning AES-CBC-HMAC, RSA, password hashing, the key generator, file hashing
Security policy Bee.Definition Where a key comes from, who may reach it and how it is verified The master key source, protecting keys, access token validation, the protection level from the second section

The dividing line takes one question: does this code know what it is computing? The encryption code in the previous section recognizes only bytes and a key, and does not know it is encrypting an API call, so it lives in the primitive layer. That is also why this post's payload and the key protection coming up use the same code.

The master key is the first item in the chain

The Purpose and Classification of Definition Files reached a conclusion about the startup dependency chain that carries straight over here: the master key is the first item in the chain, and it cannot be encrypted itself, so SystemSettings records only where it is. The framework's default copy, SystemSettings.xml, records the name of an environment variable:

<MasterKeySource>
  <Type>Environment</Type>
  <Value>BEE_MASTER_KEY</Value>
</MasterKeySource>
Enter fullscreen mode Exit fullscreen mode

An environment variable is the present default and a file is the other option. Each route has its own exposure surface, and the environment variable is not chosen for being safer. It is chosen because containers and secret management tools inject through environment variables as a matter of course, while going through a file means either mounting a volume or baking the file into the image. The master key does not encrypt any call directly. Its use is unlocking the other keys in the settings, which are stored as ciphertext, and what wraps them is AesCbcHmacCryptor.

The transport key is one per session

Session Management and Lifecycle listed what a session carries, one of them being this connection's symmetric key, and it named this post at the time. That key comes from an interface, and the framework ships three implementations:

Provider Where its key comes from Survives a session rebuild?
Shared The whole deployment shares a single key Survives
Derived (the default) Derived from a root key and the access token Survives
Random Generated at login, living only inside the session Does not survive

What the third column decides is whether rebuilding a session from a database seed, which that same earlier post described, can hold at all. The interface carries a property that answers exactly this: if the key cannot be recovered, a rebuilt session leaves the user looking as though they are still signed in while every encrypted call fails.

The derived one's root key comes from the master key by default, and a value given in the settings takes precedence. The cost of making it the default is that the same access token always derives the same key, rather than a random one at each login. What that buys is a key that never has to be stored and that every node can compute for itself, while the token itself is a GUID nobody can guess and the root key never leaves the server. The order here is counterintuitive as a result: the token has to be generated first, and only then can the key be derived.

How that key reaches the caller

At login the caller generates an RSA key pair and writes the public key into the login request. Once the server has built the session it wraps the symmetric key with that public key and returns it alongside the access token. Every call asking for encryption after that uses it.

That same post on the Connector wrote up the exception on the browser side: an RSA key pair cannot be generated there, so the public key sent is an empty string and the server returns an empty string too. The degrading happens on the calling end rather than on the server, and a call asking for Encrypted falls back to Encoded by itself when it has no key in hand. Set that against the second section and it is clear how far this fallback route goes: the Encoded format gets past Public and Encoded, and does not get past the handful marked Encrypted.


Back to Northwind

The application code in the case study relating to this post is a few lines at startup: put a demo master key in if the environment variable is not set, build the payload options from the definition file into components, then allow automatic creation when the framework starts. The rest is all in the definition files: gzip for compression, aes-cbc-hmac for encryption, and the master key from BEE_MASTER_KEY. For the value slot the case study declares nothing, so it takes messagepack, which is what applies when nothing is declared.

[ApiAccessControl] appears zero times in the whole case study. The order business object overrides GetNewData, and the overriding method carries no declaration, relying on exactly the second section's rule that the search moves on to the method it overrides.

The implementation that does not encrypt is only allowed to be constructed in debug mode, and the case study has debug mode on. The Error Contract and User Visible Messages used the same flag, where what it decided was whether error messages get masked. So that check may as well not exist here, and the case study picked real encryption anyway. The check has a gap of its own as well: it only stops the route that builds an implementation by name, and the route that hands a component in directly goes around it.

What actually makes the case study's encryption not count is not that check, it is the demo master key written into the source. The comment beside it says outright that production has to inject a real one through a real deployment mechanism before startup. The framework cannot tell which kind of environment it is running in, and blocking startup outright would take down test environments wanting to run the sample data, so this is recorded in a comment and nowhere else.


In closing

A call's value carries three stages of processing and one declaration:

  • The three run in the order serialize, compress, encrypt, and the way back runs them in reverse, where the point of reversing is that each stage only sees bytes that got past the check before it
  • A protection level records the least this method accepts and a format records what this call actually did, and what is compared is the former being less than or equal to the latter. Three of the four levels sit on one axis, and the fourth only borrows that axis to stand in line
  • The four decisions in the encryption stage can each be verified one at a time: two keys, a new IV every time, encrypt then stamp, and a fixed time comparison

All three are things anybody can go and check. A pipeline with the order swapped shows it in the compression ratio on the spot. Whether a level holds is answered by firing once at the endpoint for each. Whether the authentication code is stamped in the right place is answered by changing one bit and trying.

One thing is not. Whether that master key really is known only to whoever deployed it cannot be seen from inside the framework, and cannot be measured either. Once the three above are all done right, the strength of the whole thing equals whatever is in that environment variable, and that happens outside the code: the real line of defense is the deployment checklist.

In a security pipeline, the part about whether it is built right can be verified, and the part about whether it deserves trust cannot. The first is a design question, closed once it is written into the mechanism. The second has to be answered again at every deployment.

The next post, The Three Axes of Permissions: Action, Row and Column, changes axis. A protection level governs the form of one call, and at no point does it ask which data this person may see.


📚 Part of the thirty-post series ERP Architect's Notes: Designing a Definition-Driven Framework

📘 Original article on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/day-23-payload-pipeline

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook | HackMD | GitHub | NuGet

Top comments (0)