Bug when flattening with a non-existent sequence and colander.drop

[iwillau]

Hi All!

I have been using colander for year now, but I've never come across this one. I am using Colander to validate some incoming JSON and then flatten that structure. Several portions of the JSON are optional and if they are missing from the input struct I want to ensure they are also missing from the deserialized and subsequently flattened data, hence I don't want to use a default of any kind.

When applying the missing=colander.drop to a SequenceSchema and then flattening a struct that does not contain the sequence it triggers a bug, trying to iterate colander.null. I am presuming that this should do what I am trying, which is to omit that key from flattened output entirely.

Example code to trigger the bug:

import colander
from pprint import pprint

class MySeq(colander.SequenceSchema):
    key = colander.SchemaNode(colander.String())

class MySchema(colander.MappingSchema):
    title = colander.SchemaNode(colander.String())
    items = MySeq(missing=colander.drop)

schema = MySchema()


def example(cstruct):
    print('===================================')
    pprint(cstruct, width=40)
    try:
        deserialized = schema.deserialize(cstruct)
        pprint(deserialized, width=40)
        pprint(schema.flatten(deserialized), width=40)
    except Exception as err:
        print('Error')
        print(err)

cstruct_original = {
    'title': 'My Things',
    'items': ['one', 'two'],
}

cstruct_empty_items = {
    'title': 'My Things',
    'items': [],
}

cstruct_empty_none_items = {
    'title': 'My Things',
    'items': None,
}

cstruct_no_items = {
    'title': 'My Things',
}

example(cstruct_original)
example(cstruct_empty_items)
example(cstruct_empty_none_items)
example(cstruct_no_items)

I have added some tests and fixed the problem in my fork: 65afce14771618553ae651b9071aac0695eaf839

I'd be happy to create a PR if you want to: https://github.com/Pylons/colander/compare/master...iwillau:master

Below is the output of the script above before and after my changes:

(env) wwheatley solo:colander $ python example.py 
===================================
{'items': ['one', 'two'],
 'title': 'My Things'}
{'items': ['one', 'two'],
 'title': 'My Things'}
{'items.0': 'one',
 'items.1': 'two',
 'title': 'My Things'}
===================================
{'items': [], 'title': 'My Things'}
{'items': [], 'title': 'My Things'}
{'title': 'My Things'}
===================================
{'items': None, 'title': 'My Things'}
Error
{'items': '"None" is not iterable'}
===================================
{'title': 'My Things'}
{'title': 'My Things'}
Error
'_null' object is not iterable
(env) wwheatley solo:colander $ git checkout master
Switched to branch 'master'
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
(env) wwheatley solo:colander $ python example.py 
===================================
{'items': ['one', 'two'],
 'title': 'My Things'}
{'items': ['one', 'two'],
 'title': 'My Things'}
{'items.0': 'one',
 'items.1': 'two',
 'title': 'My Things'}
===================================
{'items': [], 'title': 'My Things'}
{'items': [], 'title': 'My Things'}
{'items': [], 'title': 'My Things'}
===================================
{'items': None, 'title': 'My Things'}
Error
{'items': '"None" is not iterable'}
===================================
{'title': 'My Things'}
{'title': 'My Things'}
{'title': 'My Things'}

[fmigneault]

For anyone interested about an "easy" solution to handle this, my solution was to insert the following definitions to drop the mapping and sequences with minimal code refactoring when missing=drop.

from colander import MappingSchema as MapSchema, SequenceSchema as SeqSchema

class DropableSchema(colander.SchemaNode):
    def deserialize(self, cstruct):
        if self.default is colander.null and self.missing is colander.drop and cstruct is None:
            return colander.drop
        return super(DropableSchema, self).deserialize(cstruct)

class MappingSchema(DropableSchema, MapSchema):
    """Override the default :class:`colander.MappingSchema` to auto-handle dropping missing definition as required."""


class SequenceSchema(DropableSchema, SeqSchema):
    """Override the default :class:`colander.SequenceSchema` to auto-handle dropping missing definition as required."""

Following will then resolve as expected by dropping the missing s1 definition:

class SchemaA(MappingSchema):
    field = SchemaNode(String())

class SchemaB(MappingSchema):
    s1 = SchemaA(missing=drop)   # optional
    s2 = SchemaA()               # required

SchemaB().deserialize({"s1": {"field": "ok"}, "s2": {"field": "ok"}})
# {'s1': {'field': 'ok'}, 's2': {'field': 'ok'}}
SchemaB().deserialize({"s1": None, "s2": {"field": "ok"}})   # this would raise normally
# {'s2': {'field': 'ok'}}
SchemaB().deserialize({"s2": {"field": "ok"}})
# {'s2': {'field': 'ok'}}