Use Microsoft Graph API and get rid of not-supported onedrivesdk

[derrix060]

trafficstars

The current code is using a mix of old MS APIs.

This repo could authenticate using the new api, so it's something doable.

[derrix060]

Instead of re-write all the onedrive sdk and use it only on this project, I decided to fork the oficial sdk and make the changes there. Feel free to help me on this: https://github.com/OneDriveSDK/onedrive-sdk-python

[modelmat]

I'll help where I can, but I'm back to school next week so I won't do much.

[derrix060]

I was working on it yesterday, and almost all code changes are ready. I already get some information from Graph API in the code, I just need to make sure I can get everything.

[derrix060]

known issues:

• Also needs to support old accounts that were using onedrivesdk
• I had to tweak a little bit onedrivesdk, so I need to put the changes here and find a good way to install this custom package
• Subscriptions are not supported because Graph requires to have a HTTPS connection to the redirect_url. Testing with the default (NGROK) configuration didn't work.
• Maybe there will be some api calls that I couldn't test yet that will fail.

[modelmat]

Since onedrivesdk is now graph-sdk-python, should we rename it throughout that repo and for the pypi package?

Also I'm not sure if this is relevant, but should we maybe call OneDriveSDK organisation something else and have onedriveClient in that organisation?

[modelmat]

Additionally, with the HTTPS connection we could: a) Use https://github.com/derrix060/onedriveClient/issues/4#issuecomment-425824705 or, b) Serve up a temporary local web server which can handle the token requests (this should also stop token input errors) > there is an example given here

[derrix060]

Since onedrivesdk is now graph-sdk-python, should we rename it throughout that repo and for the pypi package?

I've already created an issue to rename to graph-sdk-python, feel free to work on that. We still need to update the pypi package

Also I'm not sure if this is relevant, but should we maybe call OneDriveSDK organisation something else and have onedriveClient in that organisation?

I liked the idea to move onedriveClient to the organization and I'm okay renaming it... Any suggestion for the "new" name?

Additionally, with the HTTPS connection we could: a) Use #4 (comment)

This wouldn't change the problem because the problem is not to have the token, but is to receive the notification from the subscription. I've reopened the issue anyway.

b) Serve up a temporary local web server which can handle the token requests (this should also stop token input errors) > there is an example given here

That's the idea of NGROK. I got 404 when trying to access the link...

[modelmat]

Oh, okay. That seems strange. I might take a look sometime, if the changes ever get pushed to the repo (or a branch on it).

[derrix060]

I've pushed the changes here

[derrix060]

Working fine locally. I've also tested with docker and is okay.

The key is to install the package using:

pip install .
pip install graph-sdk-python/

Please make sure to use a non-existent directory at this point, to avoid any conflict with the files.

@Modelmat can you double check to see if it's working, and if so, I'll submit a PR.

[modelmat]

Ok, on it. Does this have tests?

[derrix060]

Only the normal tests, the problem to test it, is I don't have a business account only for testing, and this won't be reliable at all.

[modelmat]

Okay, I was having issues with a keyerror, I believe it may be caused by a broken config - I'll try and delete it.

I got slightly sidetracked, so I'm not able to go any further at the moment. Sorry :P

[modelmat]

Starting onedrive-client ... OK
[2019-01-09 10:55:30,884] DEBUG: MainThread: Starting new HTTPS connection (1): login.microsoftonline.com
[2019-01-09 10:55:31,426] DEBUG: MainThread: https://login.microsoftonline.com:443 "POST /common/oauth2/v2.0/token HTTP/1.1" 200 3075
[2019-01-09 10:55:31,497] DEBUG: MainThread: Starting new HTTPS connection (1): graph.microsoft.com
[2019-01-09 10:55:32,634] DEBUG: MainThread: https://graph.microsoft.com:443 "GET /v1.0/me/drives HTTP/1.1" 200 None
[2019-01-09 10:55:32,638] DEBUG: MainThread: Refreshing repository session.
[2019-01-09 10:55:32,639] DEBUG: MainThread: Starting new HTTPS connection (1): login.microsoftonline.com
[2019-01-09 10:55:33,141] DEBUG: MainThread: https://login.microsoftonline.com:443 "POST /common/oauth2/v2.0/token HTTP/1.1" 200 3075
[2019-01-09 10:55:33,183] INFO: MainThread: Session for account 0137ANOXN6Y2GOVW7725BZO354PWSELRRZ will expire in 3599 seconds.
[2019-01-09 10:55:33,183] DEBUG: MainThread: Will refresh session in 3479 seconds.
[2019-01-09 10:55:33,184] INFO: MainThread: Sweeping onedrive_client temporary files from local repositories.
[2019-01-09 10:55:33,189] DEBUG: Worker-0: Started.
[2019-01-09 10:55:33,189] DEBUG: Worker-1: Started.
[2019-01-09 10:55:33,189] DEBUG: Worker-2: Started.
[2019-01-09 10:55:33,190] DEBUG: Worker-3: Started.
[2019-01-09 10:55:33,190] DEBUG: Worker-4: Started.
[2019-01-09 10:55:33,190] DEBUG: Worker-5: Started.
[2019-01-09 10:55:33,190] DEBUG: Worker-6: Started.
[2019-01-09 10:55:33,190] DEBUG: Worker-7: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-8: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-9: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-10: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-11: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-12: Started.
[2019-01-09 10:55:33,191] DEBUG: Worker-13: Started.
[2019-01-09 10:55:33,192] DEBUG: Worker-14: Started.
[2019-01-09 10:55:33,192] DEBUG: Worker-15: Started.
[2019-01-09 10:55:34,204] INFO: MainThread: Local ngrok API url: http://127.0.0.1:4040/api
[2019-01-09 10:55:34,216] DEBUG: MainThread: Starting new HTTP connection (1): 127.0.0.1
[2019-01-09 10:55:34,219] DEBUG: MainThread: http://127.0.0.1:4040 "GET /api/tunnels HTTP/1.1" 200 36
[2019-01-09 10:55:34,221] ERROR: MainThread: Error reading ngrok API: ngrok API did not return any tunnel.. Retry in 1sec.
[2019-01-09 10:55:35,227] DEBUG: MainThread: Starting new HTTP connection (1): 127.0.0.1
[2019-01-09 10:55:35,229] DEBUG: MainThread: http://127.0.0.1:4040 "GET /api/tunnels HTTP/1.1" 200 389
[2019-01-09 10:55:35,232] DEBUG: WebhookWorker: Started.
[2019-01-09 10:55:35,232] INFO: Webhook: Local webhook server listening on port 34263.
[2019-01-09 10:55:35,233] INFO: Webhook: Webhook server listening on https://88a8f1c8.ngrok.io/x1zIMAceBEG.
[2019-01-09 10:55:35,238] DEBUG: MainThread: Adding task StartRepositoryTask(drive=b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb)...
[2019-01-09 10:55:35,238] INFO: MainThread: Scheduled sync task for Drive b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb of account 0137ANOXN6Y2GOVW7725BZO354PWSELRRZ.
[2019-01-09 10:55:35,239] INFO: MainThread: Updating webhook for Drive b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb.
[2019-01-09 10:55:35,241] DEBUG: Worker-0: Got task StartRepositoryTask(drive=b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb).
[2019-01-09 10:55:35,242] DEBUG: MainThread: Starting new HTTPS connection (1): graph.microsoft.com
[2019-01-09 10:55:35,242] DEBUG: Worker-0: Adding task MergeDirectoryTask(/home/modelmat/OneDrive, deep=True, remote_unchanged=False, parent_remote_unchanged=False)...
[2019-01-09 10:55:35,243] DEBUG: Worker-0: Got task MergeDirectoryTask(/home/modelmat/OneDrive, deep=True, remote_unchanged=False, parent_remote_unchanged=False).
[2019-01-09 10:55:35,243] DEBUG: Worker-0: Removing watcher for "/home/modelmat/OneDrive"
[2019-01-09 10:55:35,249] DEBUG: Worker-0: Starting new HTTPS connection (1): graph.microsoft.com
[2019-01-09 10:55:35,701] DEBUG: Worker-0: https://graph.microsoft.com:443 "GET /v1.0/me/drives/b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb/root:/:/children HTTP/1.1" 400 None
[2019-01-09 10:55:35,703] ERROR: Worker-0: Encountered API Error: BadRequest - Resource not found for the segment 'root:'..
[2019-01-09 10:55:35,703] ERROR: Worker-0: Encountered API Error: BadRequest - Resource not found for the segment 'root:'.. Skip directory "".
[2019-01-09 10:55:35,798] DEBUG: MainThread: https://graph.microsoft.com:443 "POST /v1.0/me/drives/b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb/root:/:/subscriptions HTTP/1.1" 400 None
[2019-01-09 10:55:35,800] ERROR: MainThread: Error: BadRequest - Resource not found for the segment 'root:'.
[2019-01-09 10:55:35,802] WARNING: MainThread: Failed to create webhook. Will deep sync again in 21600 sec.
[2019-01-09 10:55:35,806] WARNING: MainThread: Executing <Handle gen_start_repo_tasks({'0137ANOXN6Y2...ZO354PWSELRRZ': [<onedrive_cli...x7f8039c1eda0>]}) at /usr/lib/python3.7/site-packages/onedriveClient-2.0.1-py3.7.egg/onedrive_client/od_main.py:129 created at /usr/lib/python3.7/site-packages/onedriveClient-2.0.1-py3.7.egg/onedrive_client/od_main.py:207> took 0.565 seconds
^C

finally got round to it, this error occurs.

[modelmat]

I just set this up after clearing all the config directories. Output of onedrive-client-pref drives list:

 10:57:09  ~  onedrive-client-pref drive list
Reading drives information from OneDrive server...

All available Drives of authorized accounts:

  #  Account Email                       Drive ID                                                            Type      Endpoint                Quota
---  ----------------------------------  ------------------------------------------------------------------  --------  ----------------------  ------------------
  0  [email protected]  b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb  business  Documents               9.9 GB/1.0 TB Used
  1  [email protected]  b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbxl7fi0NSxASKw9r6p1z9IF  business  EduOneNoteAppDocuments  0.0 B/1.0 TB Used

Drives that have been set up:

 #0 - Drive "b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb":
   Account:     [email protected] (0137ANOXN6Y2GOVW7725BZO354PWSELRRZ)
   Local root:  /home/modelmat/OneDrive
   Ignore file: /home/modelmat/.config/onedrive_client/ignore_v2.txt```

[derrix060]

The above error is because was using a wrong backend.

So is it working now? Can I merge this -break- change?

[modelmat]

The log above was an error? I don't think it is working - I couldn't find a for business command flag so I assumed both were the same?

[derrix060]

@Modelmat correct, both are the same.

I was referring to the longs that you put, more specifically to

[2019-01-09 10:55:35,798] DEBUG: MainThread: https://graph.microsoft.com:443 "POST /v1.0/me/drives/b!D7UiT0fOkUCFDKGEOWReIjCkcaYkNEpHrBCgM6IhYbwQZfhyXdaNSaKrLDD5UWXb/root:/:/subscriptions HTTP/1.1" 400 None
[2019-01-09 10:55:35,800] ERROR: MainThread: Error: BadRequest - Resource not found for the segment 'root:'.

[modelmat]

So, is this not an issue? So it's still syncing from this?