Passing array in query via axios

[LaundroMat]

Hi,

I'm using axios to send a GET request to my flask-smorest endpoint. I know there's no real agreement on this spec-wise, but the request's querystring contains an array, and axios sends it in this format:

http://127.0.0.1:5000/api/v1/items/?types[]=cd&types[]=dvd

I've defined the schema used for reading the arguments as

class FilterSchema(ma.Schema):
    ...
    types = ma.fields.List(ma.fields.String(), missing=[])

Yet when I try to read the data received in my endpoint, types is empty:

@items_blp.route('/')
class ItemCollection(MethodView):
    @items_blp.arguments(FilterSchema(unknown=ma.EXCLUDE), location="query")
    @listings_blp.response(ItemSchema(many=True))
    def get(self, payload):
        # payload['types'] is empty...
        if payload['types']:
            qs = Item.objects.filter(item_type__in=payload['types'])
        return qs

Is this a missing feature in flask-smorest or should I ask on SO whether I should use another way to pass data?

[lafrech]

There's two issues

• 1/ Parsing this query string format correctly.
• 2/ Documenting it correctly

1/ is a webargs issue, so I'll transfer to webargs. You'll have to write a custom parser (see https://webargs.readthedocs.io/en/latest/advanced.html#custom-parsers).

2/ is an apispec issue. I'm not sure there's a way to write a spec that will display correctly in a frontend such as ReDoc or Swagger-UI, so you might end up just adding a generic comment at the beginning of the spec.

[LaundroMat]

Great, thanks for pointing me in the right direction!

Update: I just realized I probably should leave this ticket open so that the webargs team can have a look at it and decide to take it up or not.

[sirosen]

As @lafrech said, you could write a custom parser which handles this for you. That's the best approach available today.

Since axios appears to be a pretty popular, I'd be happy to have a new section in the docs for examples, with an AxiosQueryParser example, or something similar.

It looks to me like axios is actually using another library, qs, for the encoding? Do we know how widespread usage of that tool is? I'm not sure if we should be thinking about direct support within webargs for this querystring format yet, but I'll sleep on the idea, at least.

Looking at how qs encodes arrays and objects (link: https://www.npmjs.com/package/qs ), I think you can get something pretty good quite similar to how the current example parser is done.

[LaundroMat]

Thank you for considering this.

FWIW, I cobbled something together that serves my purpose, but it's far from a full parser of qs generated querystrings. It doesn't handle complex nested objects for example (see the last test in the code below).

I might try and create a PR later, but I wanted to leave this already here if that's ok with you.

By the way, I couldn't find a way for marshmallow or flask_smorest to use this parser. If someone can point me in the right direction, I'd like to add an explanation to the docs about that.

import re
from webargs.flaskparser import FlaskParser
from werkzeug.datastructures import ImmutableMultiDict

class AxiosQueryFlaskParser(FlaskParser):
    """
    WIP: Parse axios generated query args
    
    This parser handles query args generated by the Javascript qs library's stringify methods as explained at https://www.npmjs.com/package/qs#stringifying

    For example, the URL query params `?names[]=John&names[]=Eric`
    will yield the following dict:

        {
            'names': ['John', 'Eric']
        }
    """

    def load_querystring(self, req, schema):
        return _structure_dict(req.args)

def _structure_dict(dict_):
  # dict is req.args, a ImmutableMultiDict that _can_ contain double entries
  # e.g. ([('filters[]', 'size'), ('filters[]', 'color'), ('sortBy', 'price')
  return_value = {}
  for k,v in dict_.items():
    m = re.match(r"(\w+)\[(.*)\]", k)

    if m:
      print("groups: ", m.groups())
      if m.group(2) == '':
        # empty [], i.e. a list of values k
        # eg a[]=b, a[]=c
        return_value[k[:-2]] = dict_.getlist(k)
      else:
        try:
          int(m.group(2))
          if return_value.get(m.group(1)):
            # there is already an item with this key in the return_value, so add it
            return_value[m.group(1)].append(v)
          else:
            return_value = {m.group(1): [v]}
        except ValueError:  # not an int
          # eg a[b]=c
          return_value[m.group(1)] = {m.group(2): v}
    else:
      # no [] in args
      if "," in v:
          # eg a=b,c
          return_value[k] = v.split(',')
      else:
        if len(dict_.getlist(k)) > 1:
          # eg a=b, a=c
          return_value[k] = dict_.getlist(k)
        else:
          # eg a=b
          return_value[k] = v
  return return_value

def test_extract_args_from_qs(args):
  return _structure_dict(args)

# tests based on https://www.npmjs.com/package/qs#stringifying

# simple
args = ImmutableMultiDict([('a', 'b')])
assert test_extract_args_from_qs(args) == {'a': 'b'}

# array
args = ImmutableMultiDict([('a[]', 'b'), ('a[]', 'c')])
assert test_extract_args_from_qs(args) == {'a': ['b', 'c']}

# object
args = ImmutableMultiDict([('a[b]', 'c')])
assert test_extract_args_from_qs(args) == {'a': {'b': 'c'}}

# different array formats
args = ImmutableMultiDict([('a[0]', 'b'), ('a[1]', 'c')])
assert test_extract_args_from_qs(args) == {'a': ['b', 'c']}

args = ImmutableMultiDict([('a[]', 'b'), ('a[]', 'c')])
assert test_extract_args_from_qs(args) == {'a': ['b', 'c']}

args = ImmutableMultiDict([('a', 'b'), ('a', 'c')])
assert test_extract_args_from_qs(args) == {'a': ['b', 'c']}

args = ImmutableMultiDict([('a', 'b,c')])
assert test_extract_args_from_qs(args) == {'a': ['b', 'c']}

args = ImmutableMultiDict([('a', 'b'), ('c[0]', 'd'), ('c[1]', 'e=f'),('f[0][0]', 'g'), ('f[1][0]', 'h')])
assert test_extract_args_from_qs(args) == {'a': 'b', 'c': ['d', 'e=f'], 'f':[['g'],['h']]}

[ThiefMaster]

Just FYI, you can configure axios to not do this [] nonsense that started in PHP when they thought it'd be even remotely a good idea to let the client decide whether unstructured form data should become an array or not...

Just pass paramsSerializer: params => qs.stringify(params, {arrayFormat: 'repeat'}) to your axios calls (or better: use axios.create() to create an instance where you pass this as a setting and use that instance everywhere)

[BorjaEst]

To support both communities (agree axios looks wired... ejm ejm) the better would might be to "decode" the axios transformation back.

To do so, you mentioned that you are usingmarshmallow&flask_smorest, so my best approach would be to use schemas and apre_load:

from marshmallow import Schema, pre_load
from werkzeug.datastructures import ImmutableMultiDict
...

class BaseSchema(Schema):
    """Base schema to control your common schema features."""
    class Meta:
        """Normally your Meta go here"""        
        ....

    @pre_load   # Support PHP¿? and axios query framework
    def process_input(self, data, **kwargs):
        fixed_args = [(x.replace('[]', ''), y) for x,y in data.data._iter_hashitems()]
        data.data = ImmutableMultiDict(fixed_args)
        return data

Note it is apre_loadin all your schemas, it adds a bit of overhear to all your request lineal to the amount of parameters in the query that is why I put it into a list comprehension. Flexibility comes at a cost, then it is up to you to decide.