Fun stuff: representing arrays and objects in query strings

Question: what's the correct way to represent arrays and objects in a URL query string?

What do you think is the *correct* way to pass array query parameters in a URL?

— jukai (樹海) (@theshalvah) September 17, 2020

Here's an expanded version of the question from the tweet: Supposing you have this object...

{
  dog: { // an object
    name: 'John',
    age: 12
  },
  user_ids: [1, 3] // an array 
}

...what's the correct format it should be in within a query string?

Answer: There's no "correct" way. It's pretty much dependent on your runtime environment (language, framework, platform). Let's see how some popular environments handle it.

PHP

In PHP, you can encode with http_build_query:

$params = [
  'dog' => ['name' => 'John', 'age' => 12], 
  'user_ids' => [1, 3]
];
urldecode(http_build_query($params));
// Gives you: "dog[name]=John&dog[age]=12&user_ids[0]=1&user_ids[1]=3"

(Note: I'm urldecode-ing so the output is easy to read.)

So PHP pretty much flattens the array/object out with keys nested in square brackets. Supports multidimensional arrays/objects too:

$params = [
  'dogs' => [
    ['name' => 'John', 'age' => 12], 
    ['name' => 'Kim', 'age' => 13], 
  ]
];
urldecode(http_build_query($params));
// Gives you: "dogs[0][name]=John&dogs[0][age]=12&dogs[1][name]=Kim&dogs[1][age]=13"

How about decoding? Decoding a query string into an array is done with parse_str. It supports the same format returned http_build_query.

$queryString = "dog[name]=John&dog[age]=12&user_ids[0]=1&user_ids[1]=3";
parse_str($queryString, $result);
// $result will be:
// [
//  'dog' => ['name' => 'John', 'age' => 12], 
//  'user_ids' => ['1', '3']
// ];

parse_str also doesn't mind if you omit the integer keys for lists (ie arrays, not objects):

$queryString = "dog[name]=John&dog[age]=12&user_ids[]=1&user_ids[]=3";
parse_str($queryString, $result);
// Same thing! $result will be:
// [
//  'dog' => ['name' => 'John', 'age' => 12], 
//  'user_ids' => ['1', '3']
// ];

Pretty straightforward, yeah? Don't get excited.

JavaScript

JavaScript in the browser gives you this nice API called URLSearchParams, while Node.js gives you the querystring module. Let's try encoding.

First in the browser:

let params = {
  dog: {
    name: 'John',
    age: 12
  },
  user_ids: [1, 3]
};
let query = new URLSearchParams(params);
decodeURIComponent(query.toString());
// Gives you: "dog=[object+Object]&user_ids=1,3"

"[object+Object]"? Yep, URLSearchParams does not support objects as values. It casts your supplied value to a string. .toString() of a generic object returns "[object Object]".

Also: it looks like it handled the array parameter, but it didn't. .toString() of an array will return the values joined by commas. To test this, if you try calling query.getAll('user_ids'), you'll get an array containing the string "1,3" as a single item, instead of an array with two separate items.

URLSearchParams does have support for arrays, though. But you need to "append" them one at a time. In our case, this will be:

let query = new URLSearchParams();
query.append('user_ids', 1);
query.append('user_ids', 3);
decodeURIComponent(query.toString());
// Gives you: "user_ids=1&user_ids=3"
query.getAll('user_ids');
// Gives you: [1, 3] (an actual array)

I definitely don't fancy that!😕 Anyway, let's go to Node.js:

let qs = require('querystring');
let params = {
  dog: {
    name: 'John',
    age: 12
  },
  user_ids: [1, 3]
};
qs.stringify(params);
// Gives you: "dog=&user_ids=1&user_ids=3"

Ha! Looks like it just skips the dog object. Well, the docs explain:

It serializes the following types of values passed in obj: <string> | <number> | <boolean> | <string[]> | <number[]> | <boolean[]>. Any other input values will be coerced to empty strings.

Welp. Better than [object Object], I guess. ¯\_(ツ)_/¯

For arrays, querystring follows URLSearchParams, only that it doesn't require you to append the items severally.

Okay, how about decoding?

Browser:

let query = new URLSearchParams("user_ids=1&user_ids=3");
query.getAll('user_ids');

Node:

qs.parse("dog=&user_ids=1&user_ids=3");
// Gives you: { dog: '', user_ids: [ '1', '3' ] }

Pretty similar behaviour.

You can try decoding the PHP-style query string, but it won't work in the way you expect. All keys will be returned as-is.

let queryString = "dog[name]=John&dog[age]=12&user_ids[]=1&user_ids[]=3";
let query = new URLSearchParams(queryString);
query.getAll('user_ids'); // Gives you: []
query.get('dog'); // Gives you: null

// *This* is what it parses
query.get('dog[name]'); // Gives you: "John"
query.get('dog[age]'); // Gives you: "12"
query.get('user_ids[]'); // Gives you: ["1", "3"]

qs.parse("dog[name]=John&dog[age]=12&user_ids[]=1&user_ids[]=3");
// Gives you:  {
//   'dog[name]': 'John',
//   'dog[age]': '12',
//   'user_ids[]': [ '1', '3' ]
// }

If you try parsing JS-style array query parameters with PHP, it fails too. You only get the last value.

parse_str("user_ids=1&user_ids=3", $result);
// $result is ["user_ids" => "3"]

But there's a twist: Node.js also supports URLSearchParams. So that's two different ways (with subtle differences) of working with query parameters in Node.js!

And remember what I said about it being framework-specific? Node.js doesn't support PHP-style query parameters, but Express (a Node.js framework) does! Express parses "dog[name]=John&dog[age]=12&user_ids[]=1&user_ids[]=3" correctly into an object and an array! So, yeah, it's not just a language thing.

Oh, and these are just some of the possible approaches. There are others I didn't mention, like JSON-encoding the object and putting that in the URL.

What to do when building a backend?

First off, it's probably smart to avoid having to use array or object query parameters where you can. That said, sometimes you can't. In such a case, your best bet is to pick a scheme, communicate it, and stick to it.

To pick a scheme, find out what system works in your framework or language by running simple tests like the ones above👆. (Don't forget to test from a frontend <form> too if that's how your service will be used.)

Alternatively, you can make your own scheme up. That isn't usually a good idea, but it can be better if your needs are simple. For instance, if you only need a list of strings, you can specify the parameter as a regular string separated by commas, and on your server you intentionally split the string by commas to make your array. That way, you don't have to worry about the framework.

And then communicate. Let your consumers know what format you're using. If you're building an API, give examples in your API documentation. That way, they can know to not rely on whatever framework their client is built on, but to handle the encoding of this themselves.

Finally, stick to it. Whatever scheme you pick, be consistent with it across your entire backend.