Our trade counter app gets each account's negotiated prices from the pricing service over gRPC. One call, GetPriceList, takes an account number and returns every price line that account has: product, unit, price and the agreement it came from. For most accounts that is a few hundred lines. Two years ago we wrote it as a single unary response, because that was simple and nobody expected a price list to grow much.
On a Monday in October our largest builders' merchant account could not see prices at the counter. Every other account was fine. The client logged RESOURCE_EXHAUSTED, and the message said the received message was larger than the maximum of 4194304 bytes. Over the weekend the account's buyers had agreed a new contract that added eleven thousand lines, and the response had grown to a little over five megabytes.
gRPC limits the size of a message that a client or server will accept, and in the common implementations the default receive limit is four mebibytes. The default send limit is far higher, unlimited in practice. So the pricing service built a five megabyte response, serialised it and sent it without complaint, and the client read the length prefix, compared it with its own limit and refused. The error looked like the server's fault and came from a client setting that neither team had ever chosen. Our tests used an account with three hundred lines.
That morning we raised the client's receive limit to sixteen megabytes and the counter worked again. We knew that was only a higher ceiling. The real change took two weeks: GetPriceList is now a server streaming call that sends pages of a thousand lines, so the size of any one message no longer depends on how big an account's contract is. The old unary method stayed for one release and was then removed. We also record the size of every pricing response by account and alert at half of the configured limit. That would have given us months of warning, because this account had been growing towards the edge since the spring.
A message size limit is not about the network or the server. It is the receiving side deciding in advance how big an answer it is willing to accept, and we had let it decide without us.
– Sergey Shinder
Top comments (0)