A Review of Coding with ChatGPT: A Simple Express Server

Code samples included in this post were generated by ChatGPT, OpenAI's large-scale language model optimized for dialogue.

The cover image in this post was generated by DALL-E 2, OpenAI's image generation model.

Scenario

Having played around a bit randomly asking ChatGPT to write code, I wanted to put it to the test with a simple, but vaguely 'real' scenario.

A simple Express server with a series of endpoints related to the processing of orders.

Taking an order object with various information such as items purchased, total price, customer details etc., the endpoints will return a string for a shipping label, an amount of commission for the salesperson, and an amount of loyalty points for the customer. To add a bit more complexity, there will also be a couple of endpoints returning CSV's.

Work

First, I create a new Express server. I can just use a command so I don't need ChatGPT for this.

npx express-generator --no-view

Now to bring in ChatGPT! First, I ask it to generate a route that returns a hard coded JSON object for an order with billing address, shipping address, line items, total, payment method, shipping method, customer, salesperson and order number. This will not be directly used by the application, but be useful later.

It returns this, which gets the gist of it but is slightly limited. I definitely need the addresses to include the country so I add that manually.

app.get('/order/:id', (req, res) => {
  const orderId = req.params.id;
  const order = {
    billingAddress: {
      street: '1 Main St',
      city: 'Boston',
      state: 'MA',
      zip: '01234'
    },
    shippingAddress: {
      street: '1 Main St',
      city: 'Boston',
      state: 'MA',
      zip: '01234'
    },
    lineItems: [
      {
        productId: 1,
        name: 'Product 1',
        quantity: 2,
        price: 9.99
      },
      {
        productId: 2,
        name: 'Product 2',
        quantity: 1,
        price: 14.99
      }
    ],
    total: 34.97,
    paymentMethod: 'credit card',
    shippingMethod: '2-day',
    customer: {
      id: 1,
      name: 'John Doe'
    },
    salesperson: {
      id: 2,
      name: 'Jane Doe'
    },
    orderNumber: orderId
  };
  res.json(order);
});

I try to start the server and it crashes because app is not defined. Since I am using Express Router from the generator, I need to change app.get to router.get. I do that and it works!

I want to try some test driven development with ChatGPT so I install Jest.

I ask ChatGPT to generate a Jest test for the main functions the program is going to need.

A function that takes an order object and returns the shipping address as a string.

test('getShippingAddressFromOrder', () => {
  const order = {
    shippingAddress: {
      street: '1 Main St',
      city: 'Boston',
      state: 'MA',
      zip: '01234'
    }
  };

  const expectedShippingAddress = '1 Main St, Boston, MA 01234';
  const actualShippingAddress = getShippingAddressFromOrder(order);

  expect(actualShippingAddress).toBe(expectedShippingAddress);
});

A function that takes an order object and returns the salesperson's commission of 0.1%.

test('getSalespersonCommission', () => {
  const order = {
    total: 100,
    salesperson: {
      id: 1,
      name: 'Jane Doe'
    }
  };

  const expectedCommission = 0.1;
  const actualCommission = getSalespersonCommission(order);

  expect(actualCommission).toBe(expectedCommission);
});

A function that takes an order object and returns the amount of customer loyalty points at 2 per dollar.

test('getCustomerLoyaltyPoints', () => {
  const order = {
    total: 100,
    customer: {
      id: 1,
      name: 'John Doe'
    }
  };

  const expectedLoyaltyPoints = 200;
  const actualLoyaltyPoints = getCustomerLoyaltyPoints(order);

  expect(actualLoyaltyPoints).toBe(expectedLoyaltyPoints);
});

A function that takes an order object and returns a csv for the packer with line items and the order number.

test('getPackerCsv', () => {
  const order = {
    orderNumber: 123456,
    lineItems: [
      {
        productId: 1,
        name: 'Product 1',
        quantity: 2
      },
      {
        productId: 2,
        name: 'Product 2',
        quantity: 1
      }
    ]
  };

  const expectedCsv = '123456,Product 1,2\n123456,Product 2,1';
  const actualCsv = getPackerCsv(order);

  expect(actualCsv).toBe(expectedCsv);
});

A function that takes an order object and returns a csv for finance with country and revenue.

test('getFinanceCsv', () => {
  const order = {
    billingAddress: {
      country: 'US',
    },
    total: 100,
  };

  const expectedCsv = 'US,100';
  const actualCsv = getFinanceCsv(order);

  expect(actualCsv).toBe(expectedCsv);
});

Now I have five test that run and they all fail, which is great as the functions have not yet been written!

Next I ask ChatGPT to write my functions.

A function that takes an order object and returns the shipping address as a string.

function getShippingAddressFromOrder(order) {
  const { shippingAddress } = order;
  return `${shippingAddress.street}, ${shippingAddress.city}, ${shippingAddress.state} ${shippingAddress.zip}`;
}

A function that takes an order object and returns the salesperson's commission of 0.1%.

function calculateCommission(order) {
  return order.total * 0.001;
}

A function that takes an order object and returns the amount of customer loyalty points at 2 per dollar.

function calculateLoyaltyPoints(order) {
  return order.total * 2;
}

A function that takes an order object and returns a csv for the packer with line items and the order number.

function generatePackerCsv(order) {
  let csv = `Order Number: ${order.number}\n`;

  for (const item of order.items) {
    csv += `${item.name}, ${item.quantity}\n`;
  }

  return csv;
}

A function that takes an order object and returns a csv for finance with country and revenue.

function generateFinanceCsv(order) {
  return `${order.country}, ${order.total}\n`;
}

It tells me how to use the functions and gives example objects to call them with.

Next I export them and import them into my test files. This step I do manually.

I now run my tests again and one passes (🎉) and the other four fail. I see this is because the function names don't match the names in the tests, which I correct manually.

Now three of them pass. One fails because the function returns the wrong result, the other fails because the function expects an iterable and is not getting one.

I look at this and find that one is using the wrong property name and the other is expecting the wrong nesting. These issues are easy to manually fix.

They both still fail, this time because the CSV strings returned by the functions do not match the ones expected by the tests. This is a bigger bug.

After some manual debugging, all tests are now passing.

I think I may want another function that takes a CSV string and generates a CSV file, so I will ask ChatGPT for a test and a function for this.

const fs = require("fs");
const { generateCsvFile } = require("./csv-utils");

describe("generateCsvFile", () => {
  it("should generate a CSV file from a CSV string", () => {
    const csvString = "A,B,C\n1,2,3\n4,5,6";
    const filePath = "./test.csv";

    generateCsvFile(csvString, filePath);

    const csvFile = fs.readFileSync(filePath, "utf8");
    expect(csvFile).toEqual(csvString);

    fs.unlinkSync(filePath);
  });
});

const fs = require("fs");

function generateCsvFile(csvString, filePath) {
  fs.writeFileSync(filePath, csvString, "utf8");
}

module.exports = { generateCsvFile };

This time it has generated the imports and exports for me as well, so I just need to change the path to suit my directory structure and it should pass.

It does!

The final step is to make the routes that will use these functions, and I ask ChatGPT to do this as well.

A route that accepts an order object and uses a getShippingAddressFromOrder function to return the shipping address.

app.get('/shipping-address', (req, res) => {
  const { order } = req.body;
  const shippingAddress = getShippingAddressFromOrder(order);

  res.send({ shippingAddress });
});

A route that accepts an order object and uses a getSalespersonCommission function to return the salesperson's commission.

app.get('/salesperson-commission', (req, res) => {
  const { order } = req.body;
  const commission = getSalespersonCommission(order);

  res.send({ commission });
});

A route that accepts an order object and uses a getCustomerLoyaltyPoints function to return the customer's loyalty points.

app.get('/customer-loyalty-points', (req, res) => {
  const { order } = req.body;
  const loyaltyPoints = getCustomerLoyaltyPoints(order);

  res.send({ loyaltyPoints });
});

A route that accepts an order object and uses a getPackerCsv function to return a CSV file for the packer.

app.get('/packer-csv', (req, res) => {
  const { order } = req.body;
  const csv = getPackerCsv(order);

  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Disposition', 'attachment; filename="packer.csv"');
  res.send(csv);
});

A route that accepts an order object and uses a getFinanceCsv function to return a CSV file for finance.

app.get('/finance-csv', (req, res) => {
  const { order } = req.body;
  const csv = getFinanceCsv(order);

  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Disposition', 'attachment; filename="finance.csv"');
  res.send(csv);
});

I have to change app.get to router.get again. I also have to add the function imports manually.

Then I start the server and try using the order object returned by the /order/:id endpoint to test the new routes.

The shipping address endpoint works and I get this response:

{
    "shippingAddress": "1 Main St, Boston, MA 01234"
}

The salesperson's commission endpoint works and I get this response:

{
    "commission": 0.03497
}

The customer loyalty points endpoint works and I get this response:

{
    "loyaltyPoints": 69.94
}

I now realize that it should be rounding the loyalty points to an integer.

The packer CSV endpoint works and I get this response:

1,Product 1,2
1,Product 2,1

The finance CSV endpoint works and I get this response:

US,34.97

I now have a working server and I've hardly written any code by hand!

Review

Now to do a code review of what GPTChat has produced.

While it has certainly done an impressive job, it is by no means flawless.

I have a few comments on the code style, I prefer arrow functions and no semicolons, but if this tool was integrated into an IDE it would probably be trivial to make it understand that from existing code style or linter rules, so no worries there.

The bugs that broke the tests mostly seemed to be about a lack of consistency, rather than any fundamental lack of understanding, and it is worth remembering that I didn't do all this in one go, so it didn't have the context of what it had previously written. Again, this would probably be resolved if the tool was integrated into an IDE.

I don't like the address objects it generated - the properties included would not easily support addresses in some parts of the world, and it did not include the country, which would be essential in any international setting (and let's face it, most tech is international). The shipping address string function assumes it will always and only have the same four properties it generated for the sample address. This may indicate that ChatGPT does not have sufficient understanding of the range of standards used around the world, or just assumes a US context (and I'm not in the US). It also did not round the customer loyalty points, whereas when I get my credit card statement, these are always rounded down.

Overall, I think it is necessary to remember that this is a tool and not a source of truth. It can be a useful assistant for humans but its output should always be checked by humans before being used in production.