couchdb-python icon indicating copy to clipboard operation
couchdb-python copied to clipboard

Cannot add DictField or ListField to existing document schemas

Open djc opened this issue 10 years ago • 12 comments

From [email protected] on August 27, 2009 20:00:03

If a document is loaded, and the ListField/DictField specified in the schema class is not pre- existing, the Fields do not work. What steps will reproduce the problem? 1. Create and save a document 2. Add ListField/DictField to schema class and re-load the document 3. ListField/DictField are non-functional What is the expected output? What do you see instead? The ListField should be represented by a schema.Proxy object, instead it is a standard python list object. When storing the document, any values placed in the list will be ignored. What version of the product are you using? On what operating system? Python 2.5.2 CouchDB 0.9.1 Please provide any additional information below. Small app to demonstrate: """

!/usr/bin/env python

import couchdb from couchdb import schema

server = couchdb.Server('http://localhost:5984/' db = server['list-tests']

class TestDoc(schema.Document): title = schema.TextField()

class TestDocAlt(schema.Document): title = schema.TextField() test_list = schema.ListField(schema.TextField())

t1 = TestDoc() t1.title = 'Test' t1.store(db)

t2 = TestDocAlt.load(db, t1.id)

Will print "<type 'list'>" not "<class 'couchdb.schema.Proxy'>"

print type(t2.test_list)

t2.test_list.append('ham') t2.store(db)

t3 = db.get(t2.id)

Will print "False" becuase the list was never stored as part of t2 on line 25

print t3.has_key('test_list') """

Original issue: http://code.google.com/p/couchdb-python/issues/detail?id=88

djc avatar Jul 12 '14 14:07 djc

From [email protected] on September 08, 2009 05:30:54

I just wanted to report this as well. In the meantime something like this (before working with test_list) has worked for me:

if "test_list" not in t2: #Enhance document if it doesn't already have test_list t2["test_list"] = []

A proceeding store will save the list then.

djc avatar Jul 12 '14 14:07 djc

From [email protected] on March 10, 2010 23:11:13

I've actually had issues with this in the past and I think in general it's a good problem to solve. Especially with a doc-based DB like couch, the document structure could change all the time. My solution was to write a class function in my document classes to do update to the new schema through a recursion through the fields in the document. This should deal fine with addition and subtraction of fields, but it won't handle name changes since it populates fields with the same name.

In the above example, the way this would be used (if both TestDoc and TestDocAlt derive from MGDocument)

t1 = TestDoc() t1.title = 'Test' t1.store(db)

t2 = TestDocAlt.load(db, t1.id) t2 = TestDocAlt.update_schema(t2) print type(t2.test_list)

and then a

t2.store(db)

to persist back to the db.

class MGDocument(schema.Document): @classmethod def update_schema(cls, old_doc): """ Run this on an old document to generate the correct schema and update to the correct rev and ids. """ new_doc = cls() def update_field(field, old_field): try: for sub_field in field: if sub_field in old_field: try: update_field(field[sub_field], old_field[sub_field]) except TypeError: # Means this field is non-iterable, set with the old value field[sub_field] = old_field[sub_field] except TypeError: # this field is non-iterable, return to the calling function raise

    update_field(new_doc, old_doc)              

    if '_rev' in old_doc:
        new_doc['_rev'] = old_doc['_rev']                   
    new_doc._set_id(old_doc._get_id())                      
    return new_doc

djc avatar Jul 12 '14 14:07 djc

From kxepal on May 21, 2011 09:19:20

Almost three years for changing:

--- a/couchdb/mapping.py
+++ b/couchdb/mapping.py
@@ -175,7 +175,7 @@
     @classmethod
     def wrap(cls, data):
-        instance = cls()
-        instance._data = data
+        instance = cls(**data)
         return instance

     def _to_python(self, value):

However, this patch compromises wrap class method - why it should be exists then?(:

djc avatar Jul 12 '14 14:07 djc

From kxepal on May 21, 2011 09:43:47

Added patch with properly handle of private members and tests.

Attachment: mapping.patch

djc avatar Jul 12 '14 14:07 djc

From [email protected] on July 12, 2011 10:23:02

I tried the patch from kxepal with the current trunk version. When i run the tests against it I get some errors (errors are attached at the end of this comment). I solved the problem in a different way. I'm not sure if this is the right way but at least it succeeds all test conditions on my system (OS X, Python Python 2.6.1 and simplejson installed). Patch for my fix is attached. Any comments/improvements appreciated.

python init.py

...................................................................................................F....F................................................

FAIL: Doctest: couchdb.mapping

Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 2131, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for couchdb.mapping File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 8, in mapping


File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 32, in couchdb.mapping Failed example: person = Person.load(db, person.id) Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in **run compileflags, 1) in test.globs File "<doctest couchdb.mapping[8]>", line 1, in person = Person.load(db, person.id) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 365, in load return cls.wrap(doc) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 177, in wrap return cls(data) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 297, in __init Mapping.init(self, **values) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 137, in init setattr(self, attrname, values.pop(attrname)) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 104, in set value = self._to_json(value) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 494, in _to_json value = datetime.combine(value, time(0))

TypeError: combine() argument 1 must be datetime.date, not str

File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 51, in couchdb.mapping Failed example: person = Person.load(db, person.id) Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in **run compileflags, 1) in test.globs File "<doctest couchdb.mapping[15]>", line 1, in person = Person.load(db, person.id) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 365, in load return cls.wrap(doc) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 177, in wrap return cls(data) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 297, in __init Mapping.init(self, **values) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 137, in init setattr(self, attrname, values.pop(attrname)) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 104, in set value = self._to_json(value) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 494, in _to_json value = datetime.combine(value, time(0)) TypeError: combine() argument 1 must be datetime.date, not str

FAIL: Doctest: couchdb.mapping.ListField

Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 2131, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for couchdb.mapping.ListField File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 578, in ListField


File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 602, in couchdb.mapping.ListField Failed example: post = Post.load(db, post.id) Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in **run compileflags, 1) in test.globs File "<doctest couchdb.mapping.ListField[8]>", line 1, in post = Post.load(db, post.id) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 365, in load return cls.wrap(doc) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 177, in wrap return cls(data) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 297, in __init Mapping.init(self, **values) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 137, in init setattr(self, attrname, values.pop(attrname)) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 104, in set value = self._to_json(value) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 494, in _to_json value = datetime.combine(value, time(0))

TypeError: combine() argument 1 must be datetime.date, not str

File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 603, in couchdb.mapping.ListField Failed example: comment = post.comments[0] Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in run compileflags, 1) in test.globs File "<doctest couchdb.mapping.ListField[9]>", line 1, in comment = post.comments[0] File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 668, in __getitem ...

djc avatar Jul 12 '14 14:07 djc

From [email protected] on July 12, 2011 10:23:02

... return self.field._to_python(self.list[index]) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 568, in _to_python return self.mapping.wrap(value) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 177, in wrap return cls(**data) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 137, in init setattr(self, attrname, values.pop(attrname)) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 104, in set value = self._to_json(value) File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 494, in _to_json value = datetime.combine(value, time(0))

TypeError: combine() argument 1 must be datetime.date, not str

File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 604, in couchdb.mapping.ListField Failed example: comment['author'] Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in __run compileflags, 1) in test.globs File "<doctest couchdb.mapping.ListField[10]>", line 1, in comment['author']

NameError: name 'comment' is not defined

File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 606, in couchdb.mapping.ListField Failed example: comment['content'] Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in __run compileflags, 1) in test.globs File "<doctest couchdb.mapping.ListField[11]>", line 1, in comment['content']

NameError: name 'comment' is not defined

File "/Users/christian/devel/carrot/couchdb-python-vanilla/couchdb/mapping.py", line 608, in couchdb.mapping.ListField Failed example: comment['time'] #doctest: +ELLIPSIS Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1231, in __run compileflags, 1) in test.globs File "<doctest couchdb.mapping.ListField[12]>", line 1, in comment['time'] #doctest: +ELLIPSIS NameError: name 'comment' is not defined


Ran 153 tests in 14.219s

FAILED (failures=2)

Attachment: schemachangefix.patch

djc avatar Jul 12 '14 14:07 djc

From [email protected] on July 12, 2011 10:54:57

As already mentioned same problem occurs when updating schema with a DictField. Attached patch (testcase and fix) for this bug. Fixes are also in my clone https://code.google.com/r/christianhaintz-couchdb-python/

Attachment: schemachangefixdict.patch

djc avatar Jul 12 '14 14:07 djc

From [email protected] on December 15, 2011 16:08:36

I've applied the last to patches on top of the current HEAD. Does not work for me.

import couchdb from couchdb.mapping import *

class Alice(Document): field = ListField(TextField(), name='doesnotexist')

class Bob(Document): field = DictField(name='doesnotexist')

db = couchdb.Database(' http://tom.iriscouch.com/test' ) docid = "0578a01e-4ddc-45a7-8e15-a450dd97c213"

a = Alice.load(db, docid) b = Bob.load(db, docid)

type(a.field)

list

type(b.field)

dict

a.field.append("foo")

a = []

b.field['x'] = 'x'

b = {}

djc avatar Jul 12 '14 14:07 djc

From [email protected] on December 15, 2011 16:09:56

Of course I mean "# a.field == []" and "# b.field == {}" in the last two comments.

djc avatar Jul 12 '14 14:07 djc

This bug seems to have been around for a long time. I believe I have a fix. Changing the wraps class method to

@classmethod
def wrap(cls, data):
    # build the instance, this means defaults are set for any new fields
    instance = cls(**data)
    data.update(instance._data)
    # the data for the instance needs to be the same object that is passed in
    # so that when we edit it locally, the main parent document _data is also updated
    instance._data = data
    return instance

seems to work.

JonathanWylie avatar Mar 07 '16 06:03 JonathanWylie

Would you be able to prepare a pull request that includes a test case for this? Thanks!

djc avatar Mar 11 '16 11:03 djc

OK I made the pull request: https://github.com/djc/couchdb-python/pull/275

JonathanWylie avatar Mar 15 '16 02:03 JonathanWylie