Don’t trust AI, trust tests

In my very first story, I talked about my experience with AI in the form of GitHub Copilot. It betrayed me again. But I was gently caught by my true lover: UnitTest

I am currently working on code that receives a file using a grpc stream. The file is sent in byte chunks. Go has a nice functionality where you can determine the media type of a file from the first 512 bytes. I do not want to keep all the bytes sent in memory so my goal is to have a byte array of exactly 512 bytes at the end to sniff the media type. All other bytes should be written to a physical file storage and then discarded.

I am not that experienced in working with arrays and slices in Golang, nor in other languages. For my test cases, I choose to test chunks smaller than 512 bytes, exactly 512 bytes, and larger than 512 bytes. If you wonder why, check out what boundary tests are. I have a lot of experience in writing tests.

Not surprisingly, the test with only 4 bytes failed. It took me some time to get deeper into the go standard libraries. I (mis)use tests for this because it is so easy to write, execute, and debug small snippets of code. Here is my learning example:

func TestArray(t *testing.T) {
 //target2 := [6]int{}
 target := make([]int, 6)
 first := []int{1, 2, 3}
 second := []int{4, 5}
 size := 0
 copy(target[size:], first)
 size += len(first)
 copy(target[size:], second)
 size += len(second)
 target = target[:size]
}

AI helps me with explanations and gives me a better understanding of how to use slices in Go. It is always a pleasure for an old man to learn something from the youth full of new ideas.

With the help of GitHub Copilot, my first and second tests pass. Here is the code I used:

var totalFileSize uint64 = 0
sniff := make([]byte, 512)

copy(sniff[totalFileSize:], chunkMessage.Chunk)

The test for more than 512 bytes failed because my slice was out of range. Maybe it is time for me to admit to myself that I still have a lot to learn. GitHub Copilot came up with the following solution:

if totalFileSize < 512 {
    remaining := 512 - totalFileSize
    if len(chunkMessage.Chunk) > remaining {
        chunkMessage.Chunk = chunkMessage.Chunk[:remaining]
    }
    copy(sniff[totalFileSize:], chunkMessage.Chunk)
}

In my arrogance as an old wise man, I thought I could do better. In my defense, chunkMessage.Chunk cannot be modified because all bytes must be copied in the final file. I implemented a shorter version that worked well, at least in my eyes.

if totalFileSize < 512 {
  missingBytes := 512 - totalFileSize
  copy(sniff[totalFileSize:], chunkMessage.Chunk[:missingBytes])
}

I suggested this shorter version to the AI and asked for its opinion on my code. The AI was very pleased with my solution.

…but when I reran the tests, the scales fell from my eyes. GitHub Copilot is right, I do not copy more than 512 bytes. But in the test case where I have less than 512 bytes, this code does not work. The AI chose an answer to please me and avoided pointing out what I’d done wrong. I ended up with the code below. This is the best of both worlds.

if totalFileSize < 512 {
  missingBytes := 512 - totalFileSize
  remaingBytesInChunk := uint64(len(chunkMessage.Chunk))
  if remaingBytesInChunk < missingBytes {
    missingBytes = remaingBytesInChunk
  }
  copy(sniff[totalFileSize:], chunkMessage.Chunk[:missingBytes])
}

I strongly believe that a software engineer has to write tests. Tests are sometimes hard to write, it is stupid boring work and you have to spend time to maintain them. But like a mother, they secure your life and take care of you. With them, I can sleep like a baby without worries. Now the AI does the same.

Foto von Isaac Quesada auf Unsplash

Sleep well, AI. UnitTest loves and protects you.