Parsing Data Packets In .NET

One of the most common tasks that I come up against when using .NET at work is to parse data received via some transport mechanism into a form that can be processed and further handled. Over the years, my approach has changed along with some of the newer APIs that have become available within .NET. In this post, I want to briefly look at the approach I currently take, making use of the <Span<T> type.

For those unaware, Span<T> is a value type that represents a contiguous array of memory. For those familiar with other modern languages such as Go or Rust, Span<T> is essentially the same as a slice. It can be thought of as a pointer and length pair, which can be easily iterated over and sliced into sub-sections.

There's a companion repository for this post on GitHub, which has all of the code along with some basic unit tests for the parsing logic.

Example: A simple binary based protocol

Let's start with a simple made up binary protocol. I wanted something simple to illustrate my approach, so let's define a packet type that consists of two parts:

• Command — some kind of textual command that tells the application what to do with the packet.
• Data — some kind of textual data for the packet.

On the wire, this will be encoded as follows:

• A start byte, SOH (byte value 1), to indicate the start of the packet.
• 4 bytes for the length of the Command, in little endian byte order.
• 4 bytes for the length of he Data, in little endian byte order.
• The Command bytes,
• The Data bytes.

This can be represented as a structure like the following:

public readonly struct Packet
{
    public const byte Soh = 1;

    public readonly int CommandLength;

    public readonly int DataLength;

    public readonly string Command;

    public readonly string Data;

    public Packet(string command, string data)
    {
        CommandLength = Encoding.UTF8.GetByteCount(command);
        DataLength = Encoding.UTF8.GetByteCount(data);
        Command = command;
        Data = data;
    }

    public Packet(int commandLength, int dataLength, string command, string data)
    {
        CommandLength = commandLength;
        DataLength = dataLength;
        Command = command;
        Data = data;
    }
}

Writing a simple array based parser

We'll write a parser function that will take an array, offset and length.

I generally write state-less parsers, which tend to have a method signature like the following:

public static ParseResult ParsePacket(
    byte[] buff,
    int offset,
    int count,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
);

Where ParseResult is an enum of three possible values:

public enum ParseResult
{
    Partial,
    Complete,
    Error
}

This keeps the buffering logic within the consumer of the parser and makes testing much easier.

The array based parse function

The parse function will start off by looking for the starting SOH byte within the buffer. If we don't find an SOH byte, then we know there isn't a packet within the buffer:

public static ParseResult ParsePacket(
    byte[] buff,
    int offset,
    int count,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
)
{
    bytesConsumed = 0;

    // packets start with a SOH byte - if there is no such byte in the buffer, then we don't have a packet.
    int indexOfSoh = Array.IndexOf(buff, Packet.Soh, offset, length);

    if (indexOfSoh == -1)
    {
        bytesConsumed = length;

        packet = default;
        error = default;
        return ParseResult.Partial;
    }

We know that all packets have a fixed length header, which consists of the SOH byte and two 32-bit integer values (each of which take up 4 bytes), so the next step is to ensure that there is enough data for that header:

if (length < indexOfSoh + 9)
{
    // the packet header is two 32-bit integers encoded as little endian following a SOH byte - this means a packet must be at least 9 bytes long
    packet = default;
    return ParseResult.Partial;
}

Now we need to read those two 32-bit integer values. Remember, they're encoded in little endian byte order and we need to make sure that we handle that if the system we're parsing on isn't little endian:

if (!BitConverter.IsLittleEndian)
{
    // flip the bits as the system is not little endian
    Array.Reverse(buff, indexOfSoh + 1, sizeof(int));
}

int commandLength = BitConverter.ToInt32(buff, indexOfSoh + 1);

if (!BitConverter.IsLittleEndian)
{
    // flip the bits as the system is not little endian
    Array.Reverse(buff, indexOfSoh + 1 + sizeof(int), sizeof(int));
}

int dataLength = BitConverter.ToInt32(buff, indexOfSoh + 1 + sizeof(int));

Now that we have the length of the command and the length of the data in bytes, reading them is fairly trivial — we first ensure the buffer is long enough again, then we simply extract the strings:

if (length < indexOfSoh + 9 + commandLength + dataLength)
{
    // not enough data for the command and data
    packet = default;
    return ParseResult.Partial;
}

string command = Encoding.UTF8.GetString(buff, indexOfSoh + ((2 * sizeof(int)) + 1), commandLength);

string data = Encoding.UTF8.GetString(buff, indexOfSoh + ((2 * sizeof(int)) + 1) + commandLength, dataLength);

Then it's simply a matter of returning the right parse status and creating a Packet instance from the values:

packet = new Packet(commandLength, dataLength, command, data);
bytesConsumed = indexOfSoh + ((2 * sizeof(int)) + 1) + commandLength + dataLength;
return ParseResult.Complete;

Note that we set bytesConsumed to the total number of bytes that make up the packet, including any data that occurred before the SOH byte — this is so the consumer knows it can remove that data from its buffer.

As you can see, there's quite a lot of book-keeping of lengths and offsets. This can be made more simple by creating an internal offset variable to work from, or we can use Span<T>.

The Span<T> based parse function

The span based solution looks very similar to the array based version, except the book-keeping is simplified by a long way, and we make use of some new APIs such as the System.Buffers.Binary.BinaryPrimitives helper functions to read out integer values.

Again, we'll start of by finding the SOH byte:

public static ParseResult ParseBinaryPacket(
    in ReadOnlySpan<byte> buff,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
)
{
    bytesConsumed = 0;

    // packets start with a SOH byte - if there is no such byte in the buffer, then we don't have a packet.
    int indexOfSoh = buff.IndexOf(Packet.Soh);

    if (indexOfSoh == -1)
    {
        bytesConsumed = buff.Length;

        packet = default;
        return ParseResult.Partial;
    }

We now diverge slightly, however. Remember I said earlier that spans can be sliced into sub-sections? This means we can create a new span without copying the underlying data at all, which simply points to a new region of the buffer. In this case, we'll create a new span that points to the content after the SOH byte:

// slice past the soh, as we do not wish to include it
ReadOnlySpan<byte> b = buff[(indexOfSoh + 1)..];

We again check the length is long enough to hold the two 32-bit integer values, then we use the BinaryPrimitive APIs to read them before once again re-slicing the span:

if (b.Length < sizeof(int) * 2)
{
    // the packet header is two integers encoded as little endian - this means a packet must be at least 8 bytes long
    packet = default;
    return ParseResult.Partial;
}

int commandLength = BinaryPrimitives.ReadInt32LittleEndian(b);
int dataLength = BinaryPrimitives.ReadInt32LittleEndian(b[sizeof(int)..]);

b = b[(sizeof(int) * 2)..];

Now that we have the lengths of the Command and Data, it's a case of reading them:

if (b.Length < commandLength)
{
    // not enough data for the command
    packet = default;
    return ParseResult.Partial;
}

string command = Encoding.UTF8.GetString(b[..commandLength]);

b = b[commandLength..];

if (b.Length < dataLength)
{
    // not enough data for the data
    packet = default;
    return ParseResult.Partial;
}

string data = Encoding.UTF8.GetString(b[..dataLength]);

b = b[dataLength..];

This looks very similar to before, as Encoding.UTF8.GetString has an overload that will take a Span<byte>.

We finish up by creating a Packet instance and returning the parse status:

packet = new Packet(commandLength, dataLength, command, data);
error = default;
bytesConsumed = buff.Length - b.Length;
return ParseResult.Complete;

Comparing the two

The two approaches bear a lot of similarities, but I personally find the span based approach much easier to follow. It also ties in with some other APIs such as the MemoryPool quite nicely, as you can easily get a Span<T> from a Memory<T>.

Example: A simple text based protocol

This same approach applies to text based protocols as well. Taking the previous Packet type, let's change it so that instead of encoding the Command length and the Data length as raw bytes we encode them both as hexadecimal ASCII characters. As they're both 32-bit integers, they'll take up 8 bytes (the maximum value for a 32-bit integer in hex is 8 ASCII characters long).

Our parsing logic doesn't change much, except when reading the lengths. When using a byte array, the common approach I usually see is to use int.TryParse:

string commandLengthString = Encoding.UTF8.GetString(buff, indexOfSoh + 1, 8);
if (
    !int.TryParse(
        commandLengthString,
        NumberStyles.HexNumber,
        NumberFormatInfo.InvariantInfo,
        out int commandLength
    )
)
{
    bytesConsumed = indexOfSoh + 17;

    packet = default;
    error = "Failed to parse command length";
    return ParseResult.Error;
}

string dataLengthString = Encoding.UTF8.GetString(buff, indexOfSoh + 9, 8);
if (
    !int.TryParse(
        dataLengthString,
        NumberStyles.HexNumber,
        NumberFormatInfo.InvariantInfo,
        out int dataLength
    )
)
{
    bytesConsumed = indexOfSoh + 17;

    packet = default;
    error = "Failed to parse data length";
    return ParseResult.Error;
}

You might be tempted to do something similar when working with a Span<T>, but there is a much nicer API: System.Buffers.Text.Utf8Parser. This has lots of handy overloads for reading all sorts of types from a ReadOnlySpan<byte>, including reading hexadecimal:

if (!Utf8Parser.TryParse(b[..8], out int commandLength, out int consumed, 'X') || consumed != 8)
{
    b = b[16..];
    bytesConsumed = buff.Length - b.Length;

    packet = default;
    error = "Failed to parse command length";
    return ParseResult.Error;
}

b = b[8..];

if (!Utf8Parser.TryParse(b[..8], out int dataLength, out consumed, 'X') || consumed != 8)
{
    b = b[8..];
    bytesConsumed = buff.Length - b.Length;

    packet = default;
    error = "Failed to parse data length";
    return ParseResult.Error;
}

b = b[8..];

Specifying the standardFormat parameter as X means the format is uppercase hexadecimal.

There is also a handy System.Buffers.Text.Utf8Formatter type for formatting values into Span<T> types, which I'll likely look at in a later post.

Wrap-up

I hope this post serves as a good starting point for playing with spans, they certainly help me reason about what's actually going on much more clearly arrays do in the vast majority of cases. And being able to use a ReadOnlySpan to ensure I don't accidentally write into the data is always a nice safety net!