Compatibility checking in test suite

[tobixen]

I was promising this before the summer, but suddenly other things started happening in my life preventing me from focusing on the caldav project. Anyway, now I've released the caldav client library version 2.1 and version 0.1 of my caldav-server-tester project - so it should now be possible to utilize it in the Radicale test suite. I think it's nice as it may prevent compatibility regression issues. It may also uncover other compatibility issues in future releases of the caldav-server-tester ... which may be nice or an annoyance, dependent on your point of view :-)

To call the caldav-server-tester from the Radicale test suite, it's needed to:

• add the caldav-server-tester and caldav to the optional dependencies of the project
• construct a client instance of the caldav.davclient.DAVClient class that can talk with the calendar system. This either means:
  • Starting up a Radicale server on localhost
  • Mock up the client.request-method so it will call on the appropriate radicale methods without having to go via IP.

The actual test code would look something like this (see also testCheckCompatibility in caldav/tests/test_caldav.py):

from caldav_server_tester import ServerQuirkChecker
checker = ServerQuirkChecker(self.caldav, debug_mode='assert')
checker.check_all()
observed = checker.features_checked.dotted_feature_set_list(compact=True)
expected = ...
assert(observed == expected)

The testCheckCompatibility in the caldav project also has quite some logic to smoothen out minor deviations between the observed and the expected, not sure if the Radicale project would need something like that.

If you want me to, I may try to write a pull request for this test, but possibly I may need some help with it.

Currently, the expected compatibility issues with Radicale looks like this:

{
    "search.category.fullstring": {"support": "unsupported"},
    "search.recurrences.includes-implicit.todo.pending": {"support": "unsupported"},
    "search.recurrences.expanded.todo": {"support": "unsupported"},
    "search.recurrences.expanded.exception": {"support": "unsupported"},
 }

I do have an old list of other known issues with Radicale, but they are not covered by the caldav-server-tester yet.

The search.category.fullstring-feature holds true if an event with CATEGORIES: foo,bar,zoo will be returned when doing a search for foo,bar,zoo.

search.recurrences.includes-implicit.todo.pending is a weird bug that I've found only with Radicale. Give a recurring todo (say, yearly event starting in year 2024) with STATUS:NEEDS-ACTION, and then make a query asking for tasks for 2025, and the task will be returned as it should. Make a query asking for tasks for 2024 with status NEEDS-ACTION and the task will be returned. So far all good - but then make a query for tasks for 2025 with status NEEDS-ACTION, and the event will not be returned. I will try to get time to create a separate issue for this, complete with the icalendar data and the XML used in the report-query.

Apparently server-side expanding of events work, while server-side expanding of tasks does not work. And server-side expanding of events with exceptions (a VEVENT with the same UID and a RECURRENCE-ID) also doesn't seem to work. Please tell if you want me to file separate issues for those.

Are those compatibility issues listed above important? No, most likely not:

• search.category.fullstring is unimportant. There is not much point in searching for events with foo,bar,zoo - an event with CATEGORIES:bar,zoo,foo should be considered to be equivalent as far as I understand the icalendar RFC.
• I have a feeling that my plann project is the only real world user of "complicated server-side caldav searches". While the bug above does affect the plann project, there exists a workaround now in version 2.1 of the caldav library.
• If I remember correct, the RFC says that expanding SHOULD work, but many caldav servers doesn't support expanding at all, it's probably another caldav feature that isn't much used in the wild. My caldav library is by now by default doing the expand work client-side. so I'm not affected by it.

[pbiering]

Thank you for analysis and comments, looks like a contributor is required to implement the tester and also work on test cases which currently fail to your comments and fix them.