Support for PEP 526 - Use type hinting to express field types

[thedrow]

I really like Odin and I have used it before. There's one thing that is missing from the library that I would love to have. I'd like to be able to declare my models and mappings using type hints. pydantic provides a way to do so but it does not provide mappings. Resources can be defined like so:

from typing import Dict
import odin

class Author(odin.Resource):
    name: str

class Book(odin.Resource):
    title: str
    author: Dict[str, Author]
    genre: odin.String(max_length=255) # Only types with constraints have to be defined using custom types which are subclassed from the actual types
    num_pages: odin.Integer(min_value=1)

Mapping can be defined like so:

import odin

class CalendarEventToEventFrom(odin.Mapping):
  source: CalendarEvent
  target: CalendarEventFrom

  event_date: odin.From(CalendarEventFrom.start_date)

  # Mapping to multiple fields
  @odin.map_field(to_field=('event_hour', 'event_minute'))
  def start_date(self, v):
        # Return a tuple that is mapped to fields defined as to_fields
        return v.hour, v.minute

We can come up with more syntactic sugar to replace what we currently have. I'm aware that Odin still aims to support Python 2.7 so we'd have to find a way for this support to co-exist with the current implementation.

[timsavage]

I have already started to include type hinting in a smattering of places, however, this is something I am very much interested in adding support for. I made the decision a couple of weeks ago that release 2.x will drop support for Python < 3.5 which will allow me to fully embrace typing, to this end I recently started a 2.x branch (not in a workable state yet) to start the process of firstly removing all of the compatibility code before starting to introduce and taking advantage of Python 3 specific features.

There will likely be a couple more 1.x releases as I have a couple of features I want to introduce (namely a redesign of codecs to classes to make them more configurable/customisable).

Right now I'm working on the generic version of Baldr called OdinWeb, this provides a RESTful API built on top of Odin with a lot of improvements over Baldr including native support for Swagger with a built in Swagger UI and native integration with Flask, Bottle as well as Django.

The work on OdinWeb is letting me get a really good feel for type hints as I am using them extensively both directly and via Python 2.7 compatibility and is really being built with a Python 3 first mindset to allow Python 2.7 in the near future.

I still require Python 2.7 support for now as I need to support a couple of projects that I am not yet able to upgrade to Python 3.

[timsavage]

I've been thinking more about this particular feature and there is one glaring issue with this design.

Odin Resources allow for other attributes to be defined on a class that are not fields, a good example are resource specific constants. This presents a problem with inferring that all typed attributes are resource fields. I might need to use a construct like Field[str] to do this, however, this feels a bit like going back to the current design with little extra benefit. Another option could be an alternative Resource object that supports this, the current internal design would allow for this to be done.

[timsavage]

@thedrow Which python release you are using?

I'm considering if Odin 2.x will be based on Python 3.5 or 3.6. Seriously considering jumping straight to 3.6 to take advantage of type hints on variables, f-strings etc.

[thedrow]

3.6 is what we need. If we'd support 3.5 we'd have to drop class attributes annotations which is basically the whole feature.

[thedrow]

Has any progress been made on this issue?

[thedrow]

I've been thinking about this for a bit and maybe we could use attrs as infrastructure. It supports annotations and has many features odin lacks like automatic slots and automatic hash/cmp functions etc.

Odin will take care of the mapping and validation. What do you think?

[timsavage]

I don't really want to introduce new dependencies.

Automatic slots was a feature Odin did have in the early stages, but I removed it as it was more annoying than useful. Resources are more than just simple data elements. Automatic hash/cmp is also something else I had thought about adding several times, hashing, for example, could be all fields, or just selected fields (say keys).

I'd probably go with using meta flags to enable these features.

I've thought more about them in the refresh of Odin (odin3) where I will drop all support for older Pythons (looking like basing on 3.6+). I've just been short on time of late to dedicate the required time.

[thedrow]

At the very least, we should use Python 3.7's dataclasses. It has a backport for 3.6.

Anything that spares us a lot of code is a good idea. Anything that does so using the builtin modules is even a better idea.

[timsavage]

I've taken a very close look at dataclasses and been through all the code. One major improvement in the 3.0 version was to support validation of other objects and dataclasses would be one of those, essentially you just use the meta information against other objects or structures (another would be JSON).

These are all great ideas and I'd like to get my teeth into them. There is also some asyncio integration I want for use in OdinWeb when integrating with aiohttp.

Again the next couple of months I'm quite busy, beyond that I'll have more time to really dig into these problems

[timsavage]

I've been playing around with the concept of generating fields from dataclasses (in an interface for DynamoDB), I've gone with only supporting certain datatypes and it works quite well.

https://github.com/pyapp-org/pyapp.aiobotocore/blob/feature/dynamodb/pyapp_ext/aiobotocore/dynamodb/dataclasses.py

Is this the kind of support you were thinking of?

[thedrow]

Yes.

[timsavage]

Moving forward I'm thinking of migrating Odin away from its current design and instead basing it on pydantic (https://pydantic-docs.helpmanual.io/). That library handles much of the validation (using dataclass style declaration), and just focus on the mapping aspects. I'm struggling to find the time in my schedule to work on this project to move it beyond the current state.

[timsavage]

To expand on the previous comment. I'm working on a project that uses Marshmallow and comparing that, pydantic and Odin, they all cover common ground. However, the other two do not have a wide array of codecs or mapping features that Odin does. My employer is interested in doing some opensource work and maybe I can convince them this is a good use!

The other framework I've built pyApp makes extensive use of type attributes for various features so I would very much like to take advantage of them.

[timsavage]

Initial exploratory work is in the branch feature/typing-hinting-fields.

My thoughts at present are to allow definition using type hints but behind the scenes still generate the same metadata and field instances. This still presents some problems (how to define validation rules for example or field-specific options).

One option could be to simply keep it simple and only support a set of standard options (eg those of the base Field) and let the end-user use field classes if required just like the existing resources. This would give a best of both worlds approach.

[timsavage]

This is moving beyond just a proof of concept and into a working solution however, there are a couple of small caveats (these may not matter for your situation).

The order in which fields are defined is important for certain serialisation formats (CSV for example) in the existing odin this is handled by incrementing a counter each time a field is created, this presents some issues with the "new" resource as fields are not instantiated until later when type attributes are resolved.

With CPython 3.6 the order of items in a dictionary is based on the order they are entered (this also applies to attributes of a class) however this is considered an implementation detail and may not apply in other Python implementations that implement 3.6. As of CPython 3.7 however, the order of a Dict is defined as the order of addition as a language detail and any alternate implementation that claims to implement 3.7+ must also follow suit.

Some other details:

• ResourceOptions (a classes _meta instance) stay the same
• Resource, NewResource and AbstractResource classes all inherit off the same BaseResource
• getmeta will function as expected
• Existing fields can be used with NewResource and are unchanged for compatibility

In short, the only real change is how a resource is defined, mappings and usage remain unchanged.

[timsavage]

A basic implementation is working, however, it is missing some of the advanced generic types (List[], Dict[]).

I may introduce this as a beta feature in an upcoming release to start the feature getting used and find issues.

[timsavage]

AttributeResources are now completed. I will be releasing a pre-release of version 2.0 in the next couple days. Followed by a lot of testing and documentation to cover the new features.

This new release also drops python <3.8.

[timsavage]

Example of the new features:

class LibraryBase(odin.AnnotatedResource, abstract=True):
    class Meta:
        namespace = "library"

class Author(LibraryBase):
    name: str
    website: Optional[odin.Url]

class Book(LibraryBase):
    title: str
    isbn: str = odin.Options(max_length=32)
    num_pages: Optional[int]
    rrp: float = odin.Options(20.4)
    fiction: bool = True
    published: List[datetime.datetime]
    authors: List[Author]

[timsavage]

Released in 2.0