Connection with iCloud

[coreice]

First, thanks for making available this script.

Did someone already tried making a CalDAV connection to the iCloud? I've been trying but without succes so far, calcurse-caldav is giving 400 back. If I change the app-specific password the error code changes to 401, so it seems it can log in into iCloud.

warning: Dry run; nothing is imported/exported. Add "DryRun = No" to the
warning: [General] section in the configuration file to enable synchronization.
Connecting to caldav.icloud.com...
Removing all local calcurse objects...
error: The server at caldav.icloud.com replied with HTTP status code 400 (Bad
error: Request) while trying to access //.

The config file:

# If you want to synchronize calcurse with a CalDAV server using
# calcurse-caldav, create a new directory ~/.calcurse/caldav/, copy this file
# to ~/.calcurse/caldav/config and adjust the configuration below.

[General]
# Path to the calcurse binary that is used for importing/exporting items.
Binary = calcurse

# Host name of the server that hosts CalDAV.
Hostname = caldav.icloud.com

# Path to the CalDAV calendar on the host specified above.
Path =       

# Enable this if you want to skip SSL certificate checks.
InsecureSSL = No 

# Disable this option to actually enable synchronization. If it is enabled,
# nothing is actually written to the server or to the local data files. If you
# combine DryRun = Yes with Verbose = Yes, you get a log of what would have
# happened with this option disabled.
DryRun = Yes

# Enable this if you want detailed logs written to stdout.
Verbose = Yes

# Credentials for HTTP Basic Authentication. Leave this commented out if you do
# not want to use authentication.
[Auth]
Username = <..>
Password = <..>

# Optionally specify additional HTTP headers here.
#[CustomHeaders]
#User-Agent = Mac_OS_X/10.9.2 (13C64) CalendarAgent/176

[lfos]

What version of the script are you using? Did you try current master? Does running the script with --debug show anything helpful?

[ghost]

Hi, I'm also playing with icloud right now. The issue you have is that you did not provide a path. Finding the path is tricky, as you need the user ID.

The full path would be something like this:

https://caldav.icloud.com/USER_ID/calendars/CALENDAR_ID

This tool might help you with that: https://github.com/muhlba91/icloud

Currently, I'm as far as that calcurse-caldav syncs with this URL:

https://caldav.icloud.com/USER_ID/calendars/

But I have 5 calendar set up, so without the CALENDAR_ID, it hardly can know what to sync. But when I add the calendar id, i'll get an error 500.

Debug output without calendar ID:

~ $ calcurse-caldav --init=keep-remote --debug                                                                                                      
warning: Dry run; nothing is imported/exported. Add "DryRun = No" to the
warning: [General] section in the configuration file to enable synchronization.
Connecting to caldav.icloud.com...
Removing all local calcurse objects...
> REPORT /1365501036/calendars/
> Headers: {'user-agent': 'Mac_OS_X/10.9.2 (13C64) CalendarAgent/176', 'Authorization': 'Basic c2*************************************4eS1qb21n', 'Content-Type': 'application/xml; charset=utf-8'}
> <?xml version="1.0" encoding="utf-8" ?><C:calendar-query xmlns:D="DAV:"                   xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><D:getetag /></D:prop><C:filter><C:comp-filter name="VCALENDAR" /></C:filter></C:calendar-query>

< Headers: [('Server', 'AppleHttpServer/2f080fc0'), ('Date', 'Mon, 30 Oct 2017 15:39:00 GMT'), ('Content-Type', 'text/plain; charset=UTF-8'), ('Content-Length', '67'), ('Connection', 'keep-alive'), ('X-Apple-Jingle-Correlation-Key', 'KGYK************MXKQ3KIZU'), ('apple-seq', '0'), ('apple-tk', 'false'), ('Apple-Originating-System', 'UnknownOriginatingSystem'), ('X-Responding-Instance', 'caldavj:35000201:mr26p50ic-zteg03150801:8501:17G79:cef7b286'), ('DAV', '1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-audit, caldavserver-supports-telephone, calendar-managed-attachments, calendarserver-sharing, calendarserver-subscribed, calendarserver-home-sync'), ('Strict-Transport-Security', 'max-age=31536000; includeSubDomains'), ('via', 'icloudedge:fr02p00ic-ztde010917:7401:17RC143:Frankfurt'), ('X-Apple-Request-UUID', '51b0a300-*****-*****-*****-432ea86d****'), ('access-control-expose-headers', 'X-Apple-Request-UUID'), ('access-control-expose-headers', 'Via')]
< <?xml version='1.0' encoding='UTF-8' ?>
< <multistatus xmlns='DAV:'/>

Saving synchronization database to /home/sdk/.calcurse/caldav/sync.db...
0 items imported, 0 items removed locally.
0 items exported, 0 items removed from the server.

Debug output with calendar ID:

~ $ calcurse-caldav --init=keep-remote --debug                                 
warning: Dry run; nothing is imported/exported. Add "DryRun = No" to the
warning: [General] section in the configuration file to enable synchronization.
Connecting to caldav.icloud.com...
Removing all local calcurse objects...
> REPORT /1365501036/calendars/M2CD-2-2-*******-*******-*******-*****-********5C2DF5C/
> Headers: {'user-agent': 'Mac_OS_X/10.9.2 (13C64) CalendarAgent/176', 'Authorization': 'Basic c2lsZW*****************************************em94eS1qb21n', 'Content-Type': 'application/xml; charset=utf-8'}
> <?xml version="1.0" encoding="utf-8" ?><C:calendar-query xmlns:D="DAV:"                   xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><D:getetag /></D:prop><C:filter><C:comp-filter name="VCALENDAR" /></C:filter></C:calendar-query>

< Headers: [('Server', 'AppleHttpServer/2f080fc0'), ('Date', 'Mon, 30 Oct 2017 15:42:09 GMT'), ('Content-Length', '0'), ('Connection', 'keep-alive'), ('X-Responding-Instance', 'caldavj:35001501:mr26p50ic-zteg03081301:8501:17G79:cef7b286'), ('X-Apple-Jingle-Correlation-Key', 'EUW6P2E2OBE5HHYTJFDKSVNPPM'), ('apple-seq', '0'), ('apple-tk', 'false'), ('Apple-Originating-System', 'UnknownOriginatingSystem'), ('Strict-Transport-Security', 'max-age=31536000; includeSubDomains'), ('via', 'icloudedge:fr02p01ic-ztde011004:7401:17RC143:Frankfurt'), ('X-Apple-Request-UUID', '252de7e8-*****-*****-9f13-********55af7b'), ('access-control-expose-headers', 'X-Apple-Request-UUID'), ('access-control-expose-headers', 'Via')]

error: The server at caldav.icloud.com replied with HTTP status code 500
error: (Internal Server Error) while trying to access
error: /136*****36/calendars/M2CD-2-2-**********-*****-*****-*******-******5C2DF5C/.

I edited in some stars to not reveal my IDs.

Actually I extracted all the paths from the Android App DavDroid, which syncs just fine with icloud. You can find the URLs in it's debug log.

Let me know if you make any progress in getting calcurse to sync with icloud.

Thanks, Stefan

[ghost]

I played a bit with curl. If I understand the debug output correctly, calcurse sends the following data to retrieve calendar entries:

<?xml version="1.0" encoding="utf-8" ?><C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><D:getetag /></D:prop><C:filter><C:comp-filter name="VCALENDAR" /></C:filter></C:calendar-query>

When I do this with curl, I get the following response:

Improperly formed XML encountered, unexpected root node

If I send something like this:

<propfind xmlns='DAV:'><prop><calendar-data xmlns='urn:ietf:params:xml:ns:caldav'/></prop></propfind>

Icloud responds correctly.

At least I got the URL right: URL: https://pXX-caldav.icloud.com/USERID/calendars/LONG-UUID/

[ghost]

Here is a bit more information about icloud caldav requests: https://stackoverflow.com/questions/42319487/icloud-calendar-requests

[lfos]

It would be interesting to find out what exactly the iCloud CalDAV server does not like about our request.

Does it work if you strip the XML declaration, i.e. <?xml version="1.0" encoding="utf-8" ?>? Does it work you do not use XML namespace, i.e. strip all the C: and D:?

[ghost]

Hi ifos, unfortunately this is not making a difference. Without declaration and with/without namespace declaration, the error stays the same. Do you, by any chance, have an icloud account to play with?

[lfos]

Unfortunately not. This [1] comment suggests that the query should work, though... Maybe you can experiment a bit.

[1] https://stackoverflow.com/a/41996982

[ghost]

Hi ifos, yes the query in the link works. I made a mistake before. The XML query needs to be sent with method REPORT instead of PROPFIND.

I was able to track the issue down to a missing <C:comp-filter name="VEVENT"></C:comp-filter> component. This query works:

<?xml version="1.0" encoding="utf-8" ?>
<C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
    <D:prop>
        <D:getetag />
    </D:prop>
    <C:filter>
        <C:comp-filter name="VCALENDAR">
            <C:comp-filter name="VEVENT">
            </C:comp-filter>
        </C:comp-filter>
    </C:filter>
</C:calendar-query>

[ghost]

I have patched this change into calcurse-caldav and it indeed starts syncing. So we're one step further.

However, some ical data seems to be incompatible:

New sync database entry: /1365501036/calendars/M2CD-2-2-43DB4CA6-FDB9-4636-B859-BC42A5C2DF5C/E449558C-7B88-4534-8F81-272E83B1F6B9.ics C=23686@U=dad598fd-a359-481d-8d5b-1813195b8f57 aae63fb64bd24cf8809afa501a28b09a9c4a7896
Importing new object C=23100@U=dad598fd-a359-481d-8d5b-1813195b8f57.
New sync database entry: /1365501036/calendars/M2CD-2-2-43DB4CA6-FDB9-4636-B859-BC42A5C2DF5C/5E4F5679-6FDF-441B-8F4A-DF75C400C347.ics C=23100@U=dad598fd-a359-481d-8d5b-1813195b8f57 a22399c5299814295376f7d04298a298313a69b7
Importing new object C=136@U=dad598fd-a359-481d-8d5b-1813195b8f57.
New sync database entry: /1365501036/calendars/M2CD-2-2-43DB4CA6-FDB9-4636-B859-BC42A5C2DF5C/D012B2BB-3FA6-4078-807B-F250F8DD44EF.ics C=136@U=dad598fd-a359-481d-8d5b-1813195b8f57 9b4f406effae286178bc21e3db581a154a35980c
Importing new object C=13@U=dad598fd-a359-481d-8d5b-1813195b8f57.
Traceback (most recent call last):
  File "./calcurse-caldav", line 553, in <module>
    local_new = pull_objects(missing, modified, conn, syncdb, etagdict)
  File "./calcurse-caldav", line 353, in pull_objects
    objhash = calcurse_import(cdata)
  File "./calcurse-caldav", line 55, in calcurse_import
    return p.communicate(icaldata.encode('utf-8'))[0].decode('utf-8').rstrip()
AttributeError: 'NoneType' object has no attribute 'encode'
ical.c: 1123: Warning: ical header malformed or wrong version number. Aborting...

The failing ical looks like this:

BEGIN:VCALENDAR
PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
VERSION:2.0
BEGIN:VEVENT
UID:D012B2BB-3FA6-4078-807B-F250F8DD44EF
SUMMARY:Teddy bei Jessi
LOCATION:
DTSTART;VALUE=DATE:20110202
DTEND;VALUE=DATE:20110203
SEQUENCE:0
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:Event reminder
X-WR-ALARMUID:B00AC119-3BEE-47EC-83BD-5E1A341393F6
TRIGGER;VALUE=DURATION:-PT15M
END:VALARM
END:VEVENT
END:VCALENDAR

On a side note: It synced a lot of entries successfully!

[ghost]

The failed files can be imported via calcurse -i. This leads me to belive that calcurse-caldav is not handing the data over correctly. And indeed, in line 55 calcurse_import(icaldata), icaldata is empty.

I didn't yet figure out why.

[ghost]

After line 339 in calcurse-caldav:

• cdatanode is <Element '{urn:ietf:params:xml:ns:caldav}calendar-data' at 0x80639c4f8>
• cdatanode.text is None

[ghost]

Now it is getting interesting. I have built in a check and print a message and continue if cdatanode.text is empty. After doing a few runs I can see that the issue is happening at random. So response payloads with empty cdatanode.text at one time, work just fine at another run. I've logged the payload. The response XML for the failed entries is looking fine. I was not able to spot a difference to the working response body parts.

Code:

cdatanode = node.find("./D:propstat/D:prop/C:calendar-data",
                                  namespaces=nsmap)

Reponse XML: response.xml

Looks good to me... but aparently doesn't find the calendar-data.

[lfos]

Progress! Thanks for debugging this.

Is the XML you pasted the full response? Looks like some of the tags are not closed at the end of the document.

[ghost]

Yeah there were a few lines in this instance missing. I've updated the file.

[lfos]

Hard to say with incomplete data then. Can you load the (complete) logged XML using xml.etree and play around with node.find() to figure out what exactly makes it fail? Maybe also print the XML of a successful call and use diff(1) to find any differences?

[ghost]

I figured out how to properly decode and print the etree node... there is a 404 in there.

[ghost]

Okay, so I think I checked the wrong ID and the --debug log only prints successful IDs.

I was wondering why no .ics file was referenced in the screenshot above. It looks like icloud it indeed sending this path:

Note the white highlight. No ics file. All other paths end in an ics file:

[ghost]

I have 261 entries to sync. It happen exactly once, that the URL without ics file is in the payload. To be honest, I think the solution would be to just ignore this entry. They payload order is random, thats why it looks like a randomly occuring issue, but it's not.

[lfos]

Maybe we should simply ignore all directory hrefs, or even all hrefs not having an .ics extension?

[lfos]

I pushed two patches to pu: One of them makes the CalDAV script skip directories, the other one makes the request more explicit (required as you pointed out above). Could you try again with both patches applied? :)

[ghost]

I've tested the updated version.

• The href check works fine
• The query with the added VTODO filter leads to nothing being returned. Without the VTODO it works fine.

I think this behavior is correct, as there are no items of type VEVENT AND VTODO According to this specification the comp-filter are connected with AND

Does this work with google? In icloud, the todos are on a different path anyway.

Maybe restrict it to VEVENT for now. The real solution would be multiple calendar support where you can specify a sync type for a given path (todo, calendar).

[lfos]

Restricting it to VEVENT would mean that synchronization of todo items no longer works where it did before. If we cannot combine VEVENT and VTODO within a single query, we probably need to execute two separate requests and merge the results.

[lfos]

I amended the commit and added test="anyof" to the comp-filter. Could you give it another try, please?

[ghost]

No, test=anyof did not fix this.

I solved the issue by switching from REPORT to PROPFIND in commit https://github.com/xkpd3/calcurse/commit/2957962d42d444dcab895788af6fb6978e9385d8

The initial sync works fine now. However, there are invalid entries in sync.db now. Entries with an objhash but no href and no etag.

• sync.db

[ghost]

I've added support for multiple entries in one ical in commit https://github.com/xkpd3/calcurse/commit/34a2a13927c35e76b39a17ec1f262dd1b0735626

This entry: 3apps.ics contains 3 apps when I import it with calcurse -i. In calcurse webdav it returns 3 obj hashes when imported. This leads to the entries without hrefand etag in the the DB.

I fixed this by looping at the hashes and saving them all with the same href and etag.

Now I'm running into the next problem. After syncing with --init=keep-remote I would assume the next run without parameter would not sync anything.

However, I get this output:

Connecting to p50-caldav.icloud.com...
Loading synchronization database from /home/sdk/.calcurse/caldav/sync.db...
Pushing new object 0bafcf86901ba281b9cd3cd155cc93453cc8a7ee to the server.
error: The server at p50-caldav.icloud.com replied with HTTP status code 400
error: (Bad Request) while trying to access https://p50-caldav.icloud.com/136550
error: 1036/calendars/M2CD-2-2-43DB4CA6-FDB9-4636-B859-BC42A5C2DF5C/.

I'm not able to find 0bafcf86901ba281b9cd3cd155cc93453cc8a7ee in sync.db.

[ghost]

Hmm. https://github.com/xkpd3/calcurse/commit/34a2a13927c35e76b39a17ec1f262dd1b0735626 is not working correctly. The key field href is not unique anymore, which leads to wrong assumptions later on.

Actually syncdb_add() updates href entries instead of creating multiple ones. So only the last one survives.

[ghost]

I have added support for multiple events by "encoding" the object hashes as a comma separated list (without whitespace). When syncing, I "decode" the list.

The sync payload detection seems to work fine now... I'm not sure if I broke compatibility to the google calendar along the way, though.

The upload still does not work. I can see a "magic" duplication of VEVENTS in one reqest. But other requests with only one VEVENT also do not succeed.

-> https://github.com/xkpd3/calcurse/commit/d5b69efd349f787d8504cfcd2b7389452d730e29

[ghost]

I figured out that iCloud does accept this UID:[email protected]

but not this: UID:fa98ed11befaa37ea1099c244d1b4d900768d4bf not this: UID:20010712T182145Z-123401 not this: UID:[email protected]

*sigh*

It also demands DTSTART and DTEND, but not one without the other.

[lfos]

Thanks for putting so much work into this. The UID issue does not make a lot of sense to me, though... The UID is determined by the client and as far as I know, there is no specific format this UID has to have. The server just has to accept it the way it comes in. Does iCloud only support Apple clients (which might always use this specific UID format)?

I will add some more comments on your pull request later.

[ghost]

I'm confused by this as well. I'm using the icloud calendar on android and it syncs just fine with davdroid.

https://github.com/bitfireAT/davdroid

The UID format that works is the example from the rfc.

Apple is using Calendarserver: https://github.com/apple/ccs-calendarserver/tags