colander
colander copied to clipboard
Pylons
Add support for json schema
It would be great if colander models would be able to generate json schema. This way it would be possible to use any kind of form library that supports json schema to render forms, making it more standard and compatible.
I added basic JSON schema support (alpha) - https://github.com/plamut/colander/commit/5385f02603c6086fb32662a57ecaa60d5be8080d However, this is work is far from finished, there are several things that deserve a closer look:
- JSON schema specification can be found in this draft. There wasn't enough time at the sprint to study it in detail, so I used this WIki example as a reference as well. JSON schema as currently generated by colander might not be valid (e.g. there might be some properties/attributes which are not allowed by the draft).
- There are no tests yet. It would also be beneficial to test JSON schema compatibility with existing form libraries which consume JSON schema as an input.
- For some of the colander SchemaType objects it wasn't clear how to serialize them to JSON to retain all information. For example, a Tuple node is converted to JSON 'array' with a custom descriptive attribute length, specifying the number of items in the tuple. As mentioned before this might not be in accordance with the aforementioned draft.
I'm attaching a script I was using during the development, it might come handy. It basically defines a colander schema, dumps it to JSON and prints out the result.
from datetime import time
import colander
import json
class Friend(colander.TupleSchema):
rank = colander.SchemaNode(colander.Int(),
validator=colander.Range(0, 9999))
name = colander.SchemaNode(colander.String())
class Phone(colander.MappingSchema):
location = colander.SchemaNode(colander.String(),
validator=colander.OneOf(['home', 'work']))
number = colander.SchemaNode(colander.String())
class Friends(colander.SequenceSchema):
friend = Friend()
class Phones(colander.SequenceSchema):
phone = Phone()
class NumberList(colander.SequenceSchema):
number = colander.SchemaNode(colander.Int())
class OrderedList(colander.List):
item = colander.SchemaNode(colander.String())
class Person(colander.MappingSchema):
name = colander.SchemaNode(colander.String())
nickname = colander.SchemaNode(colander.String(), missing=colander.null)
age = colander.SchemaNode(colander.Int(),
validator=colander.Range(0, 200))
friends = Friends()
phones = Phones()
fav_numbers = NumberList()
is_happy = colander.SchemaNode(colander.Bool())
ordered_item_list = colander.SchemaNode(OrderedList())
my_unique_items = colander.SchemaNode(colander.Set())
date_of_birth = colander.SchemaNode(colander.Date())
time_of_arrival = colander.SchemaNode(colander.DateTime())
wakes_up_at = colander.SchemaNode(colander.Time())
my_global_object = colander.SchemaNode(colander.GlobalObject(time))
schema = Person()
json_schema = schema._to_json_dict()
json_string = schema.to_json()
assert(json_string == json.dumps(json_schema))
print json_string
import pprint
pp = pprint.PrettyPrinter(indent=4)
print "************ final JSON schema **************"
pp.pprint(json_schema)
Nice, thanks, i will look it up and test it as soon as possible for sure, and make contributions when time will allow. ColanderAlchemy+colander+json_schema makes a new world of possibilities.
I'm already sorry for not attending the sprint, but i had more important things to do. On Aug 17, 2013 6:47 PM, "Peter Lamut" [email protected] wrote:
I added basic JSON schema support (alpha) - plamut@5385f02https://github.com/plamut/colander/commit/5385f02603c6086fb32662a57ecaa60d5be8080dHowever, this is work is far from finished, there are several things that deserve a closer look:
JSON schema specification can be found in this drafthttp://tools.ietf.org/html/draft-zyp-json-schema-04. There wasn't enough time at the sprint to study it in detail, so I used this WIki example http://en.wikipedia.org/wiki/JSON#JSON_Schema as a reference as well. JSON schema as currently generated by colander might not be valid (e.g. there might be some properties/attributes which are not allowed by the draft).
There are no tests yet. It would also be beneficial to test JSON schema compatibility with existing form libraries which consume JSON schema as an input.
For some of the colander SchemaType objects it wasn't clear how to serialize them to JSON to retain all information. For example, a Tuple node is converted to JSON 'array' with a custom descriptive attribute * length*, specifying the number of items in the tuple. As mentioned before this might not be in accordance with the aforementioned draft.
I'm attaching a script I was using during the development, it might come handy. It basically defines a colander schema, dumps it to JSON and prints out the result.
from datetime import timeimport colanderimport json class Friend(colander.TupleSchema): rank = colander.SchemaNode(colander.Int(), validator=colander.Range(0, 9999)) name = colander.SchemaNode(colander.String()) class Phone(colander.MappingSchema): location = colander.SchemaNode(colander.String(), validator=colander.OneOf(['home', 'work'])) number = colander.SchemaNode(colander.String()) class Friends(colander.SequenceSchema): friend = Friend() class Phones(colander.SequenceSchema): phone = Phone() class NumberList(colander.SequenceSchema): number = colander.SchemaNode(colander.Int()) class OrderedList(colander.List): item = colander.SchemaNode(colander.String())
class Person(colander.MappingSchema): name = colander.SchemaNode(colander.String()) nickname = colander.SchemaNode(colander.String(), missing=colander.null) age = colander.SchemaNode(colander.Int(), validator=colander.Range(0, 200)) friends = Friends() phones = Phones() fav_numbers = NumberList() is_happy = colander.SchemaNode(colander.Bool()) ordered_item_list = colander.SchemaNode(OrderedList()) my_unique_items = colander.SchemaNode(colander.Set()) date_of_birth = colander.SchemaNode(colander.Date()) time_of_arrival = colander.SchemaNode(colander.DateTime()) wakes_up_at = colander.SchemaNode(colander.Time()) my_global_object = colander.SchemaNode(colander.GlobalObject(time))
schema = Person()json_schema = schema._to_json_dict()json_string = schema.to_json() assert(json_string == json.dumps(json_schema))print json_string import pprintpp = pprint.PrettyPrinter(indent=4)print "************ final JSON schema **************"pp.pprint(json_schema)
— Reply to this email directly or view it on GitHubhttps://github.com/Pylons/colander/issues/112#issuecomment-22815218 .
Just a quick answer from what i can see from diff and from my knowledge of colander.
Colander does not output json from serializer(even if you do json.dumps), it outputs some tree structure with every leaf node value type equal to string. There's a bug request open for that and patches provided, but no decisions or expanation made. That's why to_json could be renamed to to_json_schema.
Secondly, the function that really outputs json data string with correct types should be made or serializer should be patched or added another flag to output json. I preffer to patch serializer to output json and add to_json function which would output json data. That probably also requires deform to be patched. On Aug 17, 2013 7:16 PM, "Jaka Hudoklin" [email protected] wrote:
Nice, thanks, i will look it up and test it as soon as possible for sure, and make contributions when time will allow. ColanderAlchemy+colander+json_schema makes a new world of possibilities.
I'm already sorry for not attending the sprint, but i had more important things to do. On Aug 17, 2013 6:47 PM, "Peter Lamut" [email protected] wrote:
I added basic JSON schema support (alpha) - plamut@5385f02https://github.com/plamut/colander/commit/5385f02603c6086fb32662a57ecaa60d5be8080dHowever, this is work is far from finished, there are several things that deserve a closer look:
JSON schema specification can be found in this drafthttp://tools.ietf.org/html/draft-zyp-json-schema-04. There wasn't enough time at the sprint to study it in detail, so I used this WIki example http://en.wikipedia.org/wiki/JSON#JSON_Schema as a reference as well. JSON schema as currently generated by colander might not be valid (e.g. there might be some properties/attributes which are not allowed by the draft).
There are no tests yet. It would also be beneficial to test JSON schema compatibility with existing form libraries which consume JSON schema as an input.
For some of the colander SchemaType objects it wasn't clear how to serialize them to JSON to retain all information. For example, a Tuple node is converted to JSON 'array' with a custom descriptive attribute * length*, specifying the number of items in the tuple. As mentioned before this might not be in accordance with the aforementioned draft.
I'm attaching a script I was using during the development, it might come handy. It basically defines a colander schema, dumps it to JSON and prints out the result.
from datetime import timeimport colanderimport json class Friend(colander.TupleSchema): rank = colander.SchemaNode(colander.Int(), validator=colander.Range(0, 9999)) name = colander.SchemaNode(colander.String()) class Phone(colander.MappingSchema): location = colander.SchemaNode(colander.String(), validator=colander.OneOf(['home', 'work'])) number = colander.SchemaNode(colander.String()) class Friends(colander.SequenceSchema): friend = Friend() class Phones(colander.SequenceSchema): phone = Phone() class NumberList(colander.SequenceSchema): number = colander.SchemaNode(colander.Int()) class OrderedList(colander.List): item = colander.SchemaNode(colander.String())
class Person(colander.MappingSchema): name = colander.SchemaNode(colander.String()) nickname = colander.SchemaNode(colander.String(), missing=colander.null) age = colander.SchemaNode(colander.Int(), validator=colander.Range(0, 200)) friends = Friends() phones = Phones() fav_numbers = NumberList() is_happy = colander.SchemaNode(colander.Bool()) ordered_item_list = colander.SchemaNode(OrderedList()) my_unique_items = colander.SchemaNode(colander.Set()) date_of_birth = colander.SchemaNode(colander.Date()) time_of_arrival = colander.SchemaNode(colander.DateTime()) wakes_up_at = colander.SchemaNode(colander.Time()) my_global_object = colander.SchemaNode(colander.GlobalObject(time))
schema = Person()json_schema = schema._to_json_dict()json_string = schema.to_json() assert(json_string == json.dumps(json_schema))print json_string import pprintpp = pprint.PrettyPrinter(indent=4)print "************ final JSON schema **************"pp.pprint(json_schema)
— Reply to this email directly or view it on GitHubhttps://github.com/Pylons/colander/issues/112#issuecomment-22815218 .
Hey, sorry for the late reply, I'm overwhelmed with work. I agree with the proposition, to_json can indeed be renamed to to_json_schema, it's more explicit on what it actually do.
I haven't touched serializer at the sprint, but yes, I thank that's the right place to add JSON support, since serializer already deals with data. And yes, the libraries that currently depend on colander's serialization would probably need to be patched as well (but can't really say, I don't know them).
One problem I see with serializing is how to define a mapping between colander and JSON types, the latter being rather limited. For instance a Tuple in colander could be serialized to JSON array, but would probably need additional metadata (such as number of elements, which is fixed for a Tuple) if we want to have a reverse transformation (JSON --> colander) unambiguously defined. Of course libraries dependent on the colander would most likely have to be aware of this metadata.
I like this idea, but I don't understand why generating JSON schemas (for JS form generation) should to be part of colander. Why not create a separate package, just as HTML form generation is done in a separate package.
I do think that serializing data to JSON is a reasonable enhancement for colander, but that's a different issue.
Does this do the trick? https://github.com/sbrauer/Audrey/blob/master/audrey/colanderutil.py
Sorry if not relevant. Did not take the time to investigate much.
BTW, is there something in the opposite direction (from json schema to colander), similar to dict2colander, but with JSON schema as input?
Looks like some work has been done on this already: https://pypi.python.org/simple/colander-jsonschema/
# -*- coding: utf-8 -*-
import colander
import colander.interfaces
__version__ = '0.2'
class ConversionError(Exception):
pass
class NoSuchConverter(ConversionError):
pass
def convert_length_validator_factory(max_key, min_key):
"""
:type max_key: str
:type min_key: str
"""
def validator_converter(schema_node, validator):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
converted = None
if isinstance(validator, colander.Length):
converted = {}
if validator.max is not None:
converted[max_key] = validator.max
if validator.min is not None:
converted[min_key] = validator.min
return converted
return validator_converter
def convert_oneof_validator_factory(null_values=(None,)):
"""
:type null_values: iter
"""
def validator_converter(schema_node, validator):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
converted = None
if isinstance(validator, colander.OneOf):
converted = {}
converted['enum'] = list(validator.choices)
if not schema_node.required:
converted['enum'].extend(list(null_values))
return converted
return validator_converter
def convert_range_validator(schema_node, validator):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
converted = None
if isinstance(validator, colander.Range):
converted = {}
if validator.max is not None:
converted['maximum'] = validator.max
if validator.min is not None:
converted['minimum'] = validator.min
return converted
def convert_regex_validator(schema_node, validator):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
converted = None
if isinstance(validator, colander.Regex):
converted = {}
if hasattr(colander, 'url') and validator is colander.url:
converted['format'] = 'uri'
elif isinstance(validator, colander.Email):
converted['format'] = 'email'
else:
converted['pattern'] = validator.match_object.pattern
return converted
class ValidatorConversionDispatcher(object):
def __init__(self, *converters):
self.converters = converters
def __call__(self, schema_node, validator=None):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
if validator is None:
validator = schema_node.validator
converted = {}
if validator is not None:
for converter in (self.convert_all_validator,) + self.converters:
ret = converter(schema_node, validator)
if ret is not None:
converted = ret
break
return converted
def convert_all_validator(self, schema_node, validator):
"""
:type schema_node: colander.SchemaNode
:type validator: colander.interfaces.Validator
:rtype: dict
"""
converted = None
if isinstance(validator, colander.All):
converted = {}
for v in validator.validators:
ret = self(schema_node, v)
converted.update(ret)
return converted
class TypeConverter(object):
type = ''
convert_validator = lambda self, schema_node: {}
def __init__(self, dispatcher):
"""
:type dispatcher: TypeConversionDispatcher
"""
self.dispatcher = dispatcher
def convert_type(self, schema_node, converted):
"""
:type schema_node: colander.SchemaNode
:type converted: dict
:rtype: dict
"""
converted['type'] = self.type
if not schema_node.required:
converted['type'] = [converted['type'], 'null']
if schema_node.title:
converted['title'] = schema_node.title
if schema_node.description:
converted['description'] = schema_node.description
if schema_node.default is not colander.null:
converted['default'] = schema_node.default
return converted
def __call__(self, schema_node, converted=None):
"""
:type schema_node: colander.SchemaNode
:type converted: dict
:rtype: dict
"""
if converted is None:
converted = {}
converted = self.convert_type(schema_node, converted)
converted.update(self.convert_validator(schema_node))
return converted
class BaseStringTypeConverter(TypeConverter):
type = 'string'
format = None
def convert_type(self, schema_node, converted):
"""
:type schema_node: colander.SchemaNode
:type converted: dict
:rtype: dict
"""
converted = super(BaseStringTypeConverter,
self).convert_type(schema_node, converted)
if schema_node.required:
converted['minLength'] = 1
if self.format is not None:
converted['format'] = self.format
return converted
class BooleanTypeConverter(TypeConverter):
type = 'boolean'
class DateTypeConverter(BaseStringTypeConverter):
format = 'date'
class DateTimeTypeConverter(BaseStringTypeConverter):
format = 'date-time'
class NumberTypeConverter(TypeConverter):
type = 'number'
convert_validator = ValidatorConversionDispatcher(
convert_range_validator,
convert_oneof_validator_factory(),
)
class IntegerTypeConverter(NumberTypeConverter):
type = 'integer'
class StringTypeConverter(BaseStringTypeConverter):
convert_validator = ValidatorConversionDispatcher(
convert_length_validator_factory('maxLength', 'minLength'),
convert_regex_validator,
convert_oneof_validator_factory(('', None)),
)
class TimeTypeConverter(BaseStringTypeConverter):
format = 'time'
class ObjectTypeConverter(TypeConverter):
type = 'object'
def convert_type(self, schema_node, converted):
"""
:type schema_node: colander.SchemaNode
:type converted: dict
:rtype: dict
"""
converted = super(ObjectTypeConverter,
self).convert_type(schema_node, converted)
properties = {}
required = []
for sub_node in schema_node.children:
properties[sub_node.name] = self.dispatcher(sub_node)
if sub_node.required:
required.append(sub_node.name)
converted['properties'] = properties
if len(required) > 0:
converted['required'] = required
return converted
class ArrayTypeConverter(TypeConverter):
type = 'array'
convert_validator = ValidatorConversionDispatcher(
convert_length_validator_factory('maxItems', 'minItems'),
)
def convert_type(self, schema_node, converted):
"""
:type schema_node: colander.SchemaNode
:type converted: dict
:rtype: dict
"""
converted = super(ArrayTypeConverter,
self).convert_type(schema_node, converted)
converted['items'] = self.dispatcher(schema_node.children[0])
return converted
class TypeConversionDispatcher(object):
converters = {
colander.Boolean: BooleanTypeConverter,
colander.Date: DateTypeConverter,
colander.DateTime: DateTimeTypeConverter,
colander.Float: NumberTypeConverter,
colander.Integer: IntegerTypeConverter,
colander.Mapping: ObjectTypeConverter,
colander.Sequence: ArrayTypeConverter,
colander.String: StringTypeConverter,
colander.Time: TimeTypeConverter,
}
def __init__(self, converters=None):
"""
:type converters: dict
"""
if converters is not None:
self.converters.update(converters)
def __call__(self, schema_node):
"""
:type schema_node: colander.SchemaNode
:rtype: dict
"""
schema_type = schema_node.typ
schema_type = type(schema_type)
converter_class = self.converters.get(schema_type)
if converter_class is None:
raise NoSuchConverter
converter = converter_class(self)
converted = converter(schema_node)
return converted
def finalize_conversion(converted):
"""
:type converted: dict
:rtype: dict
"""
converted['$schema'] = 'http://json-schema.org/draft-04/schema#'
return converted
def convert(schema_node, converters=None):
"""
:type schema_node: colander.SchemaNode
:type converters: dict
:rtype: dict
"""
dispatcher = TypeConversionDispatcher(converters)
converted = dispatcher(schema_node)
converted = finalize_conversion(converted)
return converted
… but hasn't been updated since 2014. This is a blocker, I will use Marshmallow instead.
