Fails to serialize categories with icalendar 4.0.3

[jelmer]

Running icalendar 4.0.3 and todoman 3.5.0 I get lots of errors like this when adding new files:

Failed to read entry /home/jelmer/.calendars/jelmer/foss/BLAH.ics.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/todoman/model.py", line 956, in update_cache
    self.cache.add_vtodo(component, entry_path)
  File "/usr/lib/python3/dist-packages/todoman/model.py", line 622, in add_vtodo
    self._serialize_categories(todo, 'categories'),
  File "/usr/lib/python3/dist-packages/todoman/model.py", line 570, in _serialize_categories
    return ','.join([str(category) for category in categories.cats])
AttributeError: 'list' object has no attribute 'cats'

The representation for categories in icalendar has changed in 4.0.3, possibly due to "Categories are comma separated not 1 per line #265. [cleder]" (https://pypi.org/project/icalendar/)

[WhyNotHugo]

todoman 3.5.0 should work fine with ilcalendar 4.0.3 (support was added in previous versions).

This looks pretty odd; can you provide the output of pip freeze, to confirm these are the actual versions?

[jelmer]

I'm running both from their Debian package:

% dpkg -l | grep -e "(icalendar|todoman)" ii python3-icalendar 4.0.3-2 all parser/generator of iCalendar files for use with Python3 ii todoman 3.5.0-1 amd64 Simple CalDAV-based todo manager

[WhyNotHugo]

I tried to reproduce this on a clean installation of buster:

# apt-get update
...
# apt-get install -y todoman
...
# todo
bash: todo: command not found
# todoman 
Traceback (most recent call last):
  File "/usr/bin/todoman", line 6, in <module>
    from pkg_resources import load_entry_point
ModuleNotFoundError: No module named 'pkg_resources'

Looks like there's something broken with the debian package, and it won't work out of the box. I can't further debug this, sorry. You should report this to the debian package maintainers.

FWIW, support for icalendar 4.0.3 was included in todoman 3.5.0. I've just validated that it works as expected by installing todoman on buster via pip. Filtering via categories, etc, works as expected.

This is either a problem with your installation, or with the debian package itself, but, regrettably, the debian package won't work for me, so I can't really look into it.

[jelmer]

I can also reproduce this with the versions in pip. pip3 freeze::

agate==1.6.0 agate-dbf==0.2.0 agate-excel==0.2.1 agate-sql==0.5.2 aiodns==1.1.1 aiohttp==3.1.3 aiowsgi==0.7 alabaster==0.7.8 asn1crypto==0.24.0 async-timeout==3.0.0 atomicwrites==1.2.1 attrs==18.2.0 Babel==2.6.0 bcrypt==3.1.4 beautifulsoup4==4.6.3 bleach==2.1.3 blinker==1.4 bottle==0.12.13 breezy==3.0a2 bsddb3==6.2.6 byoc==2.1.1 byot==2.0.1 certifi==2018.8.24 cffi==1.11.5 chardet==3.0.4 Click==7.0 click-log==0.3.2 click-threading==0.4.4 cmarkgfm==0.4.2 colorama==0.3.7 configobj==5.0.6 cov-core==1.15.0 coverage==4.5.1 cryptography==2.3 cssutils==1.0.2 csvkit==1.0.2 cupshelpers==1.0 Cython==0.28.4 dbfread==2.0.7 decorator==4.3.0 defusedxml==0.5.0 devscripts==2.18.9 distro==1.3.0 distro-info==0.20 dnspython==1.15.0 docutils==0.14 dput==1.21 dulwich==0.19.9 entrypoints==0.2.3.post3 et-xmlfile==1.0.1 extras==1.0.0 fasteners==0.12.0 fastimport==0.9.8 fixtures==3.0.0 flake8==3.6.0 Flask==1.0.2 Flask-Assets==0.12 Flask-Bcrypt==0.7.1 Flask-Cache==0.13.1 Flask-Login==0.4.1 Flask-OAuthlib==0.9.5 Flask-Script==2.0.6 Flask-SQLAlchemy==2.1 Flask-Testing==0.4.2 Flask-WTF==0.14.2 freezegun==0.3.10 future==0.17.1 gajim==1.1.0 gbp==0.9.10+nmu1 gevent==1.3.7 ghdiff==0.4 google-api-python-client==1.5.5 gpg==1.12.0 gramps==5.0.0 greenlet==0.4.15 gunicorn==19.9.0 html5lib==1.0.1 httplib2==0.11.3 humanize==0.5.1 hypothesis==3.71.11 icalendar==4.0.3 idna==2.6 imagesize==1.0.0 isodate==0.6.0 isso==0.11.1 itsdangerous==0.24 jdcal==1.0 Jinja2==2.10 keyring==16.1.0 keyrings.alt==3.1 khard==0.12.2 klaus==1.3.0 launchpadlib==1.10.6 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 leather==0.3.3 linecache2==1.0.0 lintian-brush==0.1 loggerhead==1.20.0 lxml==4.2.5 Markdown==2.6.11 MarkupSafe==1.0 mccabe==0.6.1 meld==3.18.2 meson==0.48.2 milksnake==0.1.5 misaka==1.0.2 mock==2.0.0 monotonic==1.0 more-itertools==4.2.0 multi-key-dict==2.0.3 multidict==4.3.1 nbxmpp==0.6.8 netifaces==0.10.4 nose==1.3.7 nose2==0.7.4 oauth2client==4.1.2 oauthlib==2.0.6 olefile==0.46 openpyxl==2.4.9 packaging==18.0 paramiko==2.4.0 parsedatetime==2.4 Paste==3.0.4 PasteDeploy==1.5.2 PasteScript==2.0.2 pbr==4.2.0 pep8==1.7.1 Pillow==5.3.0 pkginfo==1.4.2 pluggy==0.8.0 ply==3.11 precis-i18n==1.0.0 prometheus-client==0.3.0 prometheus-xmpp-alerts==0.1 protobuf==3.6.1 psutil==5.4.8 psycopg2==2.7.5 py==1.7.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycairo==1.16.2 pycares==2.1.1 PyChromecast==2.3.0 pycodestyle==2.4.0 pycparser==2.19 pycrypto==2.6.1 pycups==1.9.73 pycurl==7.43.0.1 pyflakes==2.0.0 PyGithub==1.40 Pygments==2.2.0 PyGObject==3.30.2 PyICU==2.2 pyinotify==0.9.6 PyJWT==1.6.4 pykerberos==1.1.14 PyNaCl==1.3.0 pyOpenSSL==18.0.0 pyparsing==2.2.0 pysmbc==1.0.15.6 PySocks==1.6.8 PyStemmer==1.3.0 pytest==3.10.1 pytest-cov==2.6.0 pytest-localserver==0.3.7 pytest-runner==2.11.1 python-apt==1.7.0 python-axolotl==0.1.42 python-axolotl-curve25519==0.4.1.post2 python-dateutil==2.7.5 python-debian==0.1.33 python-debianbts==2.7.2 python-gitlab==1.6.0 python-gnupg==0.4.3 python-jenkins==0.4.16 python-ldap==3.1.0 python-magic==0.4.16 python-mimeparse==0.1.4 python-slugify==1.2.5 python-subunit==1.3.0 pytimeparse==1.1.5 pytz==2018.7 pyxattr==0.6.1 pyxdg==0.26 PyYAML==3.13 qrcode==6.0 readme-renderer==20.0 reportbug==7.5.1 requests==2.20.0 requests-oauthlib==0.8.0 requests-toolbelt==0.8.0 roman==2.0.0 rsa==4.0 ruamel.yaml==0.15.34 SecretStorage==2.3.1 setproctitle==1.1.10 setuptools-scm==3.1.0 simple-cdd==0.6.5 simplejson==3.15.0 SimpleTAL==5.2 six==1.11.0 sleekxmpp==1.3.3 slixmpp==1.4.0 Sphinx==1.7.9 sphinx-autorun==1.1.0 sphinx-rtd-theme==0.4.2 SQLAlchemy==1.2.8 tabulate==0.8.2 Tempita==0.5.2 testresources==2.0.0 testscenarios==0.5.0 testtools==2.3.0 todoman==3.5.0 toot==0.19.0 tox==2.5.0 tqdm==4.23.4 traceback2==1.4.0 twine==1.12.1 tzlocal==1.5.1 unattended-upgrades==0.1 Unidecode==1.0.22 unidiff==0.5.4 unittest2==1.1.0 uritemplate==0.6 urllib3==1.24 urwid==2.0.1 validictory==0.8.3 vdirsyncer==0.16.2 virtualenv==15.1.0 vobject==0.9.6.1 wadllib==1.3.3 waitress==1.1.0 webassets==0.12.1 webencodings==0.5 WebOb==1.8.2 WebTest==2.0.28 Werkzeug==0.14.1 WSGIProxy2==0.4.5 WTForms==2.2.1 xlrd==1.1.0 yarl==1.2.4 youtube-dl==2018.11.7 zeroconf==0.19.1 zope.exceptions==4.0.8 zope.interface==4.3.2 zope.testrunner==4.4.9

Failed to read entry /home/jelmer/.calendars/jelmer/foss/73bdee7b-1952-47c5-b835-043df389123a.ics. Traceback (most recent call last): File "/home/jelmer/.local/lib/python3.7/site-packages/todoman/model.py", line 956, in update_cache self.cache.add_vtodo(component, entry_path) File "/home/jelmer/.local/lib/python3.7/site-packages/todoman/model.py", line 622, in add_vtodo self._serialize_categories(todo, 'categories'), File "/home/jelmer/.local/lib/python3.7/site-packages/todoman/model.py", line 570, in _serialize_categories return ','.join([str(category) for category in categories.cats]) AttributeError: 'list' object has no attribute 'cats'

Problematic file: BEGIN:VCALENDAR BEGIN:VTODO CATEGORIES:documentation CATEGORIES:git CLASS:PUBLIC CREATED:20181123T133148Z DESCRIPTION:blah blah STATUS:NEEDS-ACTION SUMMARY:blah blah UID:some-uid URL:https://URL/blah END:VTODO END:VCALENDAR

[WhyNotHugo]

More than one category can be included by comma-separating them in a single line. I don't think that having multiple Categories entries in the iCalendar file is valid.

At least that's what's confusing python-icalendar and making it return the wrong type.

Can you try replacing:

CATEGORIES:documentation
CATEGORIES:git

with the following:

CATEGORIES:documentation,git

[jelmer]

It is valid for multiple CATEGORIES lines to appear. See https://tools.ietf.org/html/rfc5545#section-3.6.2:

 ; The following are OPTIONAL,
 ; and MAY occur more than once.
 ;
 attach / attendee / categories / comment / contact /
 exdate / rstatus / related / resources /
 rdate / x-prop / iana-prop
 ;

python3-icalendar also handles this fine, but it'll include a list for the categories property rather than a single vCategory object.

[WhyNotHugo]

Huh, I failed to find that reference. So, basically, there are multiple ways to express categories, and icalendar's API changes depending on what's present.

I'll try to address this soon; it'll probably be some ugly hack 🙁

[jelmer]

On Mon, Nov 26, 2018 at 12:36:17PM -0800, Hugo Osvaldo Barrera wrote:

Huh, I failed to find that reference. So, basically, there are multiple ways to express categories, and icalendar's API changes depending on what's present.

I'll try to address this soon; it'll probably be some ugly hack 🙁 Yep :( The only reason I know about this is because I ran into the same issue in Xandikos...

-- Jelmer Vernooij [email protected] PGP Key: https://www.jelmer.uk/D729A457.asc

[natema]

Any update on this? :(