- But First Some Binary
- Starting A Packet Sniff
- IP Header
- Understanding Unpack
- Parsing An IP Header Field
- The Whole Package
- Conclusion
In the previous installment we talked about a basic network connection. Now when sending something to a server it's not just the data that's being sent, but also metadata about the data. This is used so routers know where to direct network traffic (or in some cases block it). In this article we'll look at the part of a network transmission that deals with IP information.
But First Some Binary
Binary is a sequence of 1s and 0s which represent different levels of power of two. For example if we take the binary value 1111 it becomes:
2^0 + 2^1 + 2^2 + 2^3
or 15. 2^0 being 1 allows for odd numbered values. It's important to note that a power will only be calculated if there's a 1 in the sequence. So:
0011 = 3 (2^0 + 2^1)
0101 = 5 (2^0 + 2^2)
In terms of how binary data can work on a lower level, it's often put into a fixed value. Take for example an 8 bit (1 byte) value. This would hold a maximum value of 255. This does bring up the issue of how to handle the value 1. In binary 1 is represented simply by 1 which is 2^0. So what to do with the other 7 bits? Binary values in a fixed length handles this by 0 padding on the left side. This means 1 now becomes:
00000001
In python data types are meant to take up a certain number of bytes. The lowest amount of space taken up by such data types is 1 byte or 8 bits. As network information can be packed in less than a byte for space conservation, bitwise operators are used to work with the underlying bit level data of byte values. Take for example an 8 bit field broken up into two 4 bit parts. One holding the value 6, and the other holding the value 3:
| 6 | 3 |
|---|---|
| 0110 | 0011 |
The question now becomes how do we get the respective values? To deal with this I'll open up python and setup a temporary value with the combined 4 bit values:
my_value = 0b01100011
We'll first look at obtaining the value 6, which is the first 4 bits of the 8 bit value. The right shift operator can be used for this. A few examples of how it works:
>>> format(my_value, '08b')
'01100011'
>>> format(my_value >> 1, '08b')
'00110001'
>>> format(my_value >> 2, '08b')
'00011000'
>>> format(my_value >> 3, '08b')
'00001100'
>>> format(my_value >> 4, '08b')
'00000110'
format with 08b lets us print the binary values of a variable to a specific length (8bits) and pad the left side with 0s if required. In each iteration a value from the right side is removed and a 0 is added to the left side. By moving the value for 6 over 4 spaces and padding the left side with 0 it essentially removes the second 4 bit section and because the left side is padded with 0s we obtain the value 6:
>>> 0b00000110
6
So one logical idea would be to try the reverse of right shift, which is left shift. The problem with that is because the direction is opposite replacement 0s are no longer padding, but part of the actual value. For example:
>>> format(my_value, '08b')
'01100011'
>>> format(my_value << 1, '08b')
'11000110'
>>> format(my_value << 2, '08b')
'110001100'
>>> format(my_value << 3, '08b')
'1100011000'
>>> format(my_value << 4, '08b')
'11000110000'
>>> my_value << 4
1584
This is because the left shift is causing the 1s to hit and apply to higher values of power of two than they originally did. To deal with this we can use a bitwise operator for the last 4 bits. The bitwise operator "&" will take two binary values, left padding with zeros if necessary, and based on this logic compare each value:
- 0 and 0 is 0
- 0 and 1 is 0
- 1 and 1 is 1
Due to this logic, we can use the value 00001111 as a shortcut to mask out the first 4 bits as zero padding and the last 4 bits will become as is. The shorthand 0xF also works as hexadecimal F is 1111 which will get left padded with 4 0s to do the same thing:
>>> my_value & 0b00001111
3
>>> my_value & 0xF
3
Note that trying to do this for first x bits doesn't work for the same reasons for left shift. So the rules are:
- If you want the first x bits of a value: value >> (length_of_value_in_bits - x)
- If you want the last x bits of a value: value & 0b[1 repeated x times]
Both of these assume that the length of bits for the value is greater than or equal to how many bits you want to obtain.
Starting A Packet Sniff
With python you can utilize the socket library for a simple built in packet sniffer:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind(("192.168.1.81", 0))
s.ioctl(socket.SIO_RCVALL,socket.RCVALL_ON)
Note that this action will require administrative permissions such as root/sudo in *NIX or an elevated command prompt in Windows. So first off a raw socket is created. This type of socket let's us view packets in their raw form versus after being parsed and abstracted to us. The bind is needed so our packet sniffer can receive data. Providing port 0 essentially tells the OS to pick a random free port as we don't really care what port we're going to use. Finally, SIO_RCVALL is set to RCVALL_ON which tells our network interface controller (NIC) to enter a special state called promiscuous mode. This effectively enables our packet sniffer to receive packets.
Now to receive a packet we need to know how much data to work with. It turns out that the Internet Protocol Specification dictates that the max a packet can be is 65,535 octets. Octets are simply a way to specify 8 bits at a time when 1 byte did not necessarily equal 8 bits. So receiving a packet:
packet = s.recvfrom(65535)
packet = packet[0]
The packet = packet[0] assignment is because s.recvfrom returns a tuple of (data, address) which the address part is not of concern to us and only the first data value is. Now the packet looks something like this (truncated as it was pretty large):
>>> packet
(b'E\x00\x05\x1dw\'@\x00\x80\x06\x00\x00\xc0\xa8\x01Qh\xf4*D\xccS\x01\xbb\xe5\x80\xad\x0b\xdeU\xce\xb3P\x18\x04\x01ZA\x00\x00\x17\x03\x03\x04\xf0\xa5\xd2ns\xa4\x14\xe1\xda\x03[\x06\x11[hC\\x8\x8f\xd1\x02\x06\xb2\xa1%\x91|D\xbclt\x0f GB4\xf1h\xf2Y\xfa\xee\x05i~\xfb\x88\xe9\xbeN\x17t\x1f\xb0K#\x8b\xa9\xa1P\x107l{\x9e]\xb2\x9c\x13\x10%\x02{% raw %}`?\xdd\x0b\xc20\x95\xbf\x07>\xa1\xd1\xb8\xc5\xe8d\x8e\xbf{\xb5\x84ip\x9aJ\x1c\x8e{\x1f\xae1y\xc0\x9f\x89\xda\xaaZ\xac[\x80\xb6\xa3\xd1YX\xed\xde\x8c\xcea\x84\x13w5\xd2\x8aD9Ur\xe1\xdf\x1c-\x1aWB\tl_\x921Ek\xf7\xd7\xca\xb7\x148\x18\x91\x15\\!M\xaf\xab\xc2\xbf\\F\x06\xba\x8d\xfe\t%\xc4b\xa1\xf6\xb2\x0eS\x9b@F\xb1&\x8e\x19Q\xa7\x80\x10\xe3|L~\xbbr\x8f\xb2\x06\x844\xab\xfe\x0e\xd1\x08[<\xc1;~i\xfc\x92\x89\xf9\x8f\xe6\xf7[\xee\xf9\xa4;o\xaf\xde\xcd<snip>
```
## IP Header
The first part of the packet is an IP header. This is a special series of bytes which gives information about the source, destination, and some other useful metadata of the packet. This information is utilized by networking hardware such as routers to know where to direct the traffic to (or in some cases drop the traffic). Request for comments lists an IP header's format in [section 3.1](https://datatracker.ietf.org/doc/html/rfc791#section-3.1). A cleaner visual format can be found on the IPv4 Wikipedia page:
The header itself is made up of a minimum of 5 32 bit fields with each field spanning 4 bytes. This makes 20 bytes the minimum size of an IPv4 header. The rest can have up to 10 option fields making the total maximum size of an IP header out to be 60 bytes. The layout already shows some values that are sub byte level such as 3 bits for Flags and 4 bits for Version. This means we'll need to use bitwise operators to obtain some of the values.
## Understanding Unpack
As-is the stream of bytes is pretty difficult to work with. You could use something like splices to target individual byte groups. Instead though we can use the handy [struct.unpack](https://docs.python.org/3/library/struct.html#struct.unpack) method. This reads in binary data that's packed in sequential order which is essentially what our IP packet is. According to the RFC 791 standard the mandatory fields for a minimum size 20 byte IP header is:
### 1st 32 bit field
- Version: 4 bits
- IHL: 4 bits
- Type of Service: 8 bits
- Total Length: 16 bit
### 2nd 32 bit field
- Identification: 16 bits
- Flags: 3 bits
- Fragment Offset: 13 bits
### 3rd 32 bit field
- Time to Live: 8 bits
- Protocol 8 bits
- Header Checksum: 16 bits
### 4th 32 bit field
- Source Address: 32bits
### 5th 32 bit field
- Destination Address: 32bits
The way `struct.unpack` works is it takes binary streams and maps them to specific data types of various byte sizes. The first part of the arguments will be related to [byte ordering](https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment). In the old days there was a divide between big endian and little endian. [Kohei Otsuka](https://levelup.gitconnected.com/little-endian-vs-big-endian-eb2a2c3a9135) has a nice article on the differences. Most modern PCs are little endian encoding which you can find through `sys.byteorder`:
```
>>> import sys
>>> sys.byteorder
'little'
```
Network protocols on the other hand were standardized around when big endian had a bigger share so that's what's used in network communications. Thankfully python has a "!" format flag to indicate network byte ordering or big endian. Now it's time to map out the values. The [format chart](https://docs.python.org/3/library/struct.html#format-characters) for unpack shows the various format identifiers, what it maps to in C, and their size in bytes. Given that headers work with positive values the unsigned version tends to be used in most unpack statements. There's also a bytes format if you still want to keep the underlying binary data intact. Taking from the format table into account:
- unsigned char (1 byte/8 bit values) B
- unsigned short (2 byte/16 bit values) H
- unsigned long (4 byte/32 bit values) L
- Xs = X number of bytes as is
Ignoring sub byte values we'll pull everything in as:
- B: Version + IHL (8 bits)
- B: Type of Service (8 bits)
- H: Total Length (16 bits)
- H: Identification (16 bits)
- H: Flags + Fragment Offset (16 bits)
- B: Time to Live (8 bits)
- B: Protocol (8 bits)
- H: Header Checksum (16 bits)
- 4s: Source (IP) Address (32 bits as-is)
- 4s: Dest (IP) Address (32 bits as-is)
So we'll take the first 20 bytes of the packet (which we can do with list splices since each index of the packet is a byte) and then unpacking it using our final expression:
```
>>> ip_header_bytes = packet[0:20]
>>> import struct
>>> ip_header = struct.unpack('!BBHHHBBH4s4s', ip_header_bytes)
>>> ip_header
(69, 0, 1309, 30503, 16384, 128, 6, 0, b'\xc0\xa8\x01Q', b'h\xf4*D')
```
Now we can work on parsing the actual fields.
## Parsing An IP Header Field
We'll take the first 8 bytes which contains the Version and Internet Header Length (IHL). In binary form it currently looks like:
```
>>> format(ip_header[0], '08b')
'01000101'
```
So it's `0100` in binary and `0101` in binary packed into a total of 8 bits. Using right shift we can get the first 4 bits. As mentioned in the binary section to get the first X bits you take the total number of bits, 8, and subtract how many you want, 4, to get 4:
```
>>> ip_version = ip_header[0] >> 4
>>> format(ip_version, '08b')
'00000100'
>>> ip_version
4
```
Now technically as we're only pulling in IPv4 packets there's no real need to care about this since we always know the version is 4. In a similar fashion the Internet Header Field is the last 4 bits, and can be obtained via a bitwise operator:
```
>>> ihl = ip_header[0] & 0xf
>>> ihl
5
>>> format(ihl, '08b')
'00000101'
```
Looking at the full value both 4 bit values are shown so we know the code works.
## The Whole Package
Now that the basics are down we can work on pulling the rest of the IP header. Type of Service is a value that can potentially be used for prioritization of traffic should network equipment support it. Looking at the break out of the field it's 3 bits for precedence flags, another 3 bits that deal with weighing a packet, and two more bits that are reserved. If we look at the entire one byte value:
```
>>> format(ip_header[1], '08b')
'00000000'
```
According to the specification there is normal priority and the rest of the traffic properties are set to normal as well. Another example with prioritization in effect:
10111000
This has a higher level CRITIC/ECP precedence and sets proprieties for low delay and high throughput. Next is the total length of the packet:
```
>>> ip_header[2]
1309
```
This means the total data size is 1309 bytes, which matches the length of the list of bytes:
```
>>> len(packet)
1309
```
Identification is the next value:
```
>>> format(ip_header[3], '08b')
'111011100100111'
>>> ip_header[3]
30503
```
This is simply used as an identifier in packet assembly. The value has flags which take up the first 3 bits of a 16 bit value:
```
>>> format(ip_header[4] >> 13, '03b')
'010'
```
Looking at the specification the first value is always 0. The next value indicates that this packet should not be broken up into fragments. Because of this the fragment offset is not set as well:
```
>>> format(ip_header[4] & 0b0001111111111111, '08b')
'00000000'
```
There's an interesting article by Steven Iveson that [talks about fragmentation in more detail](https://packetpushers.net/ip-fragmentation-in-detail/). Next is the Time to Live (TTL) of the packet as it goes across the internet in seconds:
```
>>> ip_header[5]
128
```
This value can also get modified while in route to add additional time. Next is the protocol:
```
>>> ip_header[6]
6
```
[RFC 790](https://www.rfc-editor.org/rfc/rfc790.html#page-6) lists the respective decimal and octal values to their respective protocols. In this case we're dealing with protocol 6 or TCP. Next is the header checksum:
```
>>> format(ip_header[7], '016b')
'0000000000000000'
```
This is something you'll generally not see at the client side. Instead modifications to it will be made as it passes on in the event TTL values get modified, which in turn changes the checksum. Source and Destination Address are the same thing in terms of parsing:
```
>>> socket.inet_ntoa(ip_header[8])
'192.168.1.81'
>>> socket.inet_ntoa(ip_header[9])
'104.244.42.68'
```
[socket.inet_ntoa](https://docs.python.org/3/library/socket.html#socket.inet_ntoa) will take in the binary form of an IPv4 address packed in bytes and return the standard dot notation you may be used to. Doing an ARIN whois on the destination side we find that the IP belongs to Twitter, so this packet is communication with Twitter servers.
## Conclusion
This concludes our look into the IP header structure and working with it at the binary level using python. It's interesting to take a peak behind the curtains of what's going on with standard internet traffic. In the next installment of the series we'll be looking at two popular protocols: TCP and UDP.
Top comments (1)
I love networking stuff in python. Well explained.