Array of objects not parsed correctly

[lmanzurv]

When parsing the URL ?param[][id]=5a53336d48ad1c6cba0e6ae8&param[][id]=5a53336d48ad1c6cba0e6ae7 I get

param: [{ id: ['5a53336d48ad1c6cba0e6ae8', '5a53336d48ad1c6cba0e6ae7'] }]

instead of

param: [ { id: '5a53336d48ad1c6cba0e6ae8' }, { id: '5a53336d48ad1c6cba0e6ae7' } ]

I am parsing as qs.parse(queryString, { arrayLimit: 50, ignoreQueryPrefix: true }).

Am I missing something?

[ljharb]

This is expected; because of the [], qs can't know if you want one array or two. Try using indexes: ?param[0][id]=5a53336d48ad1c6cba0e6ae8&param[1][id]=5a53336d48ad1c6cba0e6ae7.

[lmanzurv]

But the structure references an array of objects. To me it is very evident that having the same key twice inside an array references 2 different objects, not an array inside the property. In that case, why bother with building an array with just 1 object at all?

I suspect wanting an array of objects where a property of the object has an array would be something like ?param[][id][]=5a53336d48ad1c6cba0e6ae8&param[][id][]=5a53336d48ad1c6cba0e6ae7, which is clearly not the case.

I also see this is not the first time this issue pops up, which it seems to point that there's something wrong with that logic. See issue https://github.com/ljharb/qs/issues/215#issuecomment-335313445.

[ljharb]

[] at the end of a key means "items with this key belong to the same array". I don't think there's as wide of a convention between a[][b] and a[][c] being one array or two.

Can you think of a configuration option we could use that would clearly indicate that choice, and would mesh well with the default?

[lmanzurv]

Sure, and that means that saying param[][id][]=X&param[][id][]=Y would be hard to know if you refer to [{id: [X,Y]}] or [{id: [X]}, {id: [Y]}]. But since I have param[][id]=X&param[][id]=Y where I'm using the same key without the suffixed [], it is safe to assume that they DON'T belong to [{id: [X,Y]}], but in fact are [{id: X}, {id: Y}].

In the case of saying param[][id]=X&param[][key]=Y, you're right, hard to know if you want [{id: X}, {key:Y}] or [{id: X, key: Y}], but I think that that is another issue altogether. For this one, the only solution I would think of is to have an option that would allow you to define a schema that would indicate the desired output structure, though I don't have any suggestions as to how to do it.

[Phanx]

I'm running into this problem. My use-case is having a number of checkboxes on the page with their name attribute set to categories[][id] to reflect that they are multi-select values for a single "categories" option. If the user checks 3 checkboxes, this happens:

• Query string: categories[][id]=1 & categories[][id]=3 & categories[][id]=4
• Expected result: [{ id: 1 }, { id: 3 }, { id: 4 }]
• Actual result: [{ id: [1, 3, 4] }]

(The numbers are actually strings, but I've omitted the quotes for readability. Spaces are inserted between query parameters for the same reason.)

Additionally, I have a text field with name="categories[][name]" where the user can enter a name to create a new category. If the user checks 2 checkboxes, and types "foo" in the text field, this happens:

• Query string: categories[][id]=1 & categories[][id]=3 & categories[][name]=foo
• Expected result: [{ id: 1 }, { id: 3 }, { name: "foo" }]
• Actual result: [{ id: [1, 3], name: "foo" }].

For now I'm working around both problems by using unique name attributes with manually specified indices, which produces the desired behavior:

• String: categories[0][id]=1 & categories[2][id]=3 & categories[3][name]=4
• Result: [{ id: 1 }, { id: 3 }, { id: 4 }]

and:

• String: categories[0][id]=1 & categories[2][id]=3 & categories[20][name]=foo
• Result: [{ id: 1 }, { id: 3 }, { name: "foo" }]

However, the utility of this workaround is limited by this stated qs behavior:

qs will also limit specifying indices in an array to a maximum index of 20. Any array members with an index of greater than 20 will instead be converted to an object with the index as the key:

The workaround won't work if I have more than 20 categories (indices 0-19 for the 20 categories, index 20 for the "new category" field). (Whether or not a multi-select option with more than 20 values should be presented as a set of checkboxes is a valid UX question, but doesn't, I think, have any bearing on questions about how a querystring parser should work.)

I could work around that by adding post-processing on the return value from qs, e.g.

if (result.categories && !Array.isArray(result.categories)) {
   result.categories = Object.entries(result.categories)
}

... but this would require hard-coding a list of keys that potentially need such processing, and it would be nice if the library could just do these things itself.

It seems like both cases described above could be covered by the same option, e.g. separateArrayItems, which would govern how [] should be treated.

• never (default) - The current behavior. Everything is merged.

• always - Every appearance of [] creates a new entry in the array, regardless of anything else.

  • param[][id]=X & param[][id]=Y (same keys) becomes [{ id: X }, { id: Y }]
  • param[][id]=X & param[][key]=Y (different keys) becomes [{ id: X }, { key: Y }] following the same pattern
  • param[][][id]=X & param[][][id]=Y becomes [ [{ id: X }], [{ id: Y }] ]
  • param[][id][]=X & param[][id][]=Y becomes [{ id: [X] }, { id: [Y] }]

• keys - Like "always" except that different key names are merged into the previous entry; once a duplicate key name is encountered, a new entry begins.

  • param[][id]=X & param[][key]=Y & param[][id]=Z becomes [{ id: X, key: Y }, { id: Z }]

I haven't looked deeply into the qs code to see how this could be implemented, but if the above sounds reasonable and would be a desired addition, I might be able to attempt a PR at some point.

[ljharb]

An option like that seems reasonable, as long as we could come up with corresponding logic on both parse and stringify so that round trips are possible.

[jchook]

Just want to weigh in here that, for example, in PHP:

$x[]['id'] = 5;
$x[]['id'] = 6;

// produces:
// [['id' => 5], ['id' => 6]]

I agree with OP and think x[] = 5 is widely understood to mean x[x.length] = 5.

[ljharb]

@jchook i'm more concerned with how PHP parses a query string that has this syntax, not how its unrelated variable syntax works. What does PHP do with ?x[][id]=5&x[][id]=6?

[jchook]

It works the same way.

I dug for some kind of spec or RFC that clarifies what to do here, but I don't think it exists.

Edit: want to add that allowing x[] to mean (roughly) x[x.length] seems more useful to me than meaning x[0] or x[x.length - 1].

PHP also defines how to handle mixed key types, e.g.

parse_str('x[cars]=Audi&x[5]=five&x[]=6', $r);

[
  "x" => [
    "cars" => "Audi",
    5 => "five",
    6 => "6",
  ],
]

Also kind of relevant...

parse_str('x=5&x=6', $r);

["x" => "6"]

[ljharb]

Fair enough. https://github.com/ljharb/qs/issues/252#issuecomment-412300241 still stands; if someone wants to submit a PR that adds the option.

[caasi]

Is there any update of this issue?

[ljharb]

@caasi https://github.com/ljharb/qs/issues/252#issuecomment-449697810

[caasi]

I try to draft a parse function:

const INIT = 0;
const BEGIN = 1;

function parseQuery([key, value]) {
  let mode = INIT;
  let idx = key.length - 1;
  let c;
  let v = '';
  while (c = key[idx]) {
    switch (mode) {
      case INIT:
        if (c === ']') {
          mode = BEGIN;
        } else {
          v = c + v;
        }
        break;
      case BEGIN:
        if (c === '[') {
          if (v) {
            value = { [v]: value };
            v = '';
          } else {
            value = [value];
          }
          mode = INIT;
        } else {
          v = c + v;
        }
        break;
    }
    idx -= 1;
  }
  switch (mode) {
    case INIT:
      return { [v]: value };
    case BEGIN:
      return { ']': value };
  }
}

function concatQuery(a, b) {
  if (!a) return b;
  if (!b) return a;
  // concat arrays
  if (Array.isArray(a) && Array.isArray(b)) {
    return a.concat(b);
  }
  // concat objects
  let ko = {};
  let ks = Object.keys(a);
  for (let k of ks) {
    ko[k] = true;
  }
  for (let k in b) {
    if (ko[k]) continue;
    ks.push(k);
  }
  let res = {};
  for (let k of ks) {
    res[k] = concatQuery(a[k], b[k]);
  }
  return res;
}

function parse(input) {
  return input.split('&').map(s => s.split('='))
    .map(parseQuery)
    .reduce(concatQuery, {});
}

const inputs = [
  'param[][id]=5a53336d48ad1c6cba0e6ae8&param[][id]=5a53336d48ad1c6cba0e6ae7',
  'param[][id][]=X&param[][id][]=Y',
  'categories[][id]=1&categories[][id]=3&categories[][id]=4',
  'categories[][id]=1&categories[][id]=3&categories[][name]=foo',
  'categories[0][id]=1&categories[2][id]=3&categories[3][name]=4',
  'categories[0][id]=1&categories[2][id]=3&categories[20][name]=foo',
];

for (let input of inputs) {
  console.log(input, '=>', JSON.stringify(parse(input)));
}

output:

param[][id]=5a53336d48ad1c6cba0e6ae8&param[][id]=5a53336d48ad1c6cba0e6ae7 => {"param":[{"id":"5a53336d48ad1c6cba0e6ae8"},{"id":"5a53336d48ad1c6cba0e6ae7"}]}
param[][id][]=X&param[][id][]=Y => {"param":[{"id":["X"]},{"id":["Y"]}]}
categories[][id]=1&categories[][id]=3&categories[][id]=4 => {"categories":[{"id":"1"},{"id":"3"},{"id":"4"}]}
categories[][id]=1&categories[][id]=3&categories[][name]=foo => {"categories":[{"id":"1"},{"id":"3"},{"name":"foo"}]}
categories[0][id]=1&categories[2][id]=3&categories[3][name]=4 => {"categories":{"0":{"id":"1"},"2":{"id":"3"},"3":{"name":"4"}}}
categories[0][id]=1&categories[2][id]=3&categories[20][name]=foo => {"categories":{"0":{"id":"1"},"2":{"id":"3"},"20":{"name":"foo"}}}

But I don't understand how qs works. I think I need some help and feedbacks.

[ljharb]

@caasi i think the first thing I'd suggest is, open up a draft PR that contains just the tests - ie, what output you expect for both parse and stringify. then, we can look at which ones are failing, and see what would be needed to make them pass.