uppy
uppy copied to clipboard
transloadit
Companion OneDrive documentation - No Session/Body attached
Hi,
I am looking to add onedrive support to our companion server and found that there is very little documentation about showing how to set-up onedrive/companion, and even less for the errors you can get during the process.
I think it would be very helpful to have a single page that runs through the minimal required set-up of onedrive (and the other providers) to get things to work. Even if these are links to more general documentation by those providers on how to set them up, it would be invaluable.
My current experience has been a headache:
The documentation https://uppy.io/docs/companion/ simply suggests that a key and secret key should be provided to companion. The example Kubernetes setup https://github.com/transloadit/uppy/blob/main/packages/%40uppy/companion/KUBERNETES.md suspiciously doesn't provide the Onedrive environment variables, and I imagine that even if they were, it would be something cute along the lines of "YOUR ONEDRIVE KEY" so you'd have no idea what the key actually was meant to look like/how long etc...
After registering my companion app on onedrive as a web app (and worrying that they seem to refer to everything as ID rather than KEY), I added the redirect URL and created a secret whose secret ID and value I've used for the ONEDRIVE_KEY and ONEDRIVE_SECRET. I've gotten the following output while trying to connect.
companion: 2022-07-12T11:22:03.395Z [info] companion.client.version uppy client version @uppy/provider-views=2.1.2
companion: 2022-07-12T11:22:03.403Z [debug] null No session/body attached to req object. Exiting dispatcher.
And the only issue that comes up is https://github.com/transloadit/uppy-server/issues/60 which suggests that is a requirement for another package expression-session. Unfortunately, I can't find anything anywhere describing what that is or why it is an issue, and this issue is very old.
I can see that uppy is making a request to the companion server at the following address: https://{my companion hostname}/onedrive/connect?state={TOKEN}&uppyVersions=%40uppy%2Fprovider-views%3D2.1.2
I can see in the token that only the site origin is being passed, and there is no payload to the get request being made so I am really unsure where to go from here. I am not sure whether this is a client-side or onedrive authentication issue since I've set it up according to the documentation, but, the documentation doesn't cover what the expected payloads should look like.
Issues like the below would likely really benefit from some guidance for provider set-up just so that can isolate the problem faster.
- https://github.com/transloadit/uppy/issues/3663
- https://github.com/transloadit/uppy/issues/3741
Hi, I'm not sure what went wrong. But we're redoing the docs (for instance for Companion: https://github.com/transloadit/uppy.io/pull/5) and I agree this should be better explained. Would be nice if you could create a bullet list of questions of things that are unclear so I can tackle them in the docs.
In the meantime, perhaps @mifi knows whats going on.
That is good to hear! I feel like there are lots of small gotchas that can be very frustrating.
In general, something spelt out like https://filemanagerpro.io/article/how-can-i-get-my-microsoft-account-client-id-and-client-secret-key/ would likely be a great start. I have done the steps in the link above so the paranoid sets that there must be something subtle in the default settings that I'm missing.
Ultimately:
- Is the application ID == client id == key from companions perspective?
- Over defining the redirect URL, are any other routes required? is a signout URL required? (which it says it wants in azure)
- Are there permissions/roles that need to be defined?
- For the Implicit grant and hybrid flows sections, should Companion be using either access token or id token or both or none?
- What account types need to be given access in the Supported account types section? I have opened it as broadly as I could.
So using the application id and including the express_session package has worked and I can now go through the authentication path, giving access to the app to read and list one drive. So setting up onedrive it is as simple as the guide I posted, no need to turn on the access/id token. Not using the express_sessions was pretty silly on my behalf, this is something I missed in the plugging into an existing express server code snippet
I also found the following issue that takes through another persons experience and their solution, which could be useful in the documentation.
- https://github.com/transloadit/uppy/issues/2314
Unfortunately, this hasn't completed resolved the connection issues, companion produced the following log line when one of us tried to upload.
companion: 2022-07-13T10:51:53.670Z [debug] null Did not receive access token for provider onedrive
companion: 2022-07-13T10:51:53.671Z [debug] callback.oauth.resp {
error: {
error: 'invalid_client',
error_description: "AADSTS650051: Using application 'desc-uploader' is currently not supported for your organization *** because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of *** before the application desc-uploader can be provisioned.\r\n" +
'Trace ID: 81fa011e-ce1c-4797-982a-aa18b3c71800\r\n' +
'Correlation ID: 6e0dba8e-48f4-465f-96ea-4d0f84e08df8\r\n' +
'Timestamp: 2022-07-13 10:51:53Z'
}
}
I personally don't see these error messages when I try to upload from a personal account or the organisation account hosting the app.
Though I do get the following
companion: 2022-07-13T10:57:35.232Z [debug] null Instantiating uploader.
companion: 2022-07-13T10:57:35.233Z [debug] null Starting download stream.
companion: 2022-07-13T10:57:35.669Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-13T10:57:35.669Z [debug] db551332 uploader.socket.wait waiting for socket connection
companion: 2022-07-13T10:57:36.013Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-13T10:58:35.671Z [error] Error: Timed out waiting for socket connection
at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
It is very unclear why the timeout is occurring. I have set the COMPANION_CLIENT_SOCKET_CONNECT_TIMEOUT to 600000 (as its not clear if its seconds or microseconds, and its not listed in the options in the documentation page) as suggested in the following issue but, it's had no effect and the upload doesn't seem to have started in the first place. The test files I am working with are small in size.
- https://github.com/transloadit/uppy/issues/3640
Hi @Murderlon, sorry to be a bother. I've not been able to make any progress uploading files from one drive - do you have any idea of something I could try to get it working?
Hi, sorry to hear this is taking so long. I personally never worked with OneDrive (I don't even have an account) and these errors are not clear to me either. @mifi generally is most knowledgeable here but he is off.
Everything works on our example page and the only thing we do is deploy this Dockerfile and set COMPANION_ONEDRIVE_DOMAIN_VALIDATION, COMPANION_ONEDRIVE_KEY, and COMPANION_ONEDRIVE_SECRET in Heroku. We don't manually set the timeout (which should be milliseconds btw).
Are you on latest versions of Uppy and Companion?
Just saw this isn't documented and I'm not sure why 🙈
https://github.com/transloadit/uppy/blob/43fb673372a9166e8b97463b68fd1c96b47da53b/packages/%40uppy/companion/src/standalone/index.js#L175-L195
Amazing! this looks like we are on for a winner!
per your last message, I am in the middle of re-installing and building the companion server and I will add this in as well. I will let you know how that gets on.
BTW the package.json looks
{
"dependencies": {
"@uppy/companion": "^3.6.0",
"express-session": "^1.17.3"
},
"devDependencies": {
"@tsconfig/node16": "^1.0.3",
"@types/node": "^18.0.0",
"browserify": "^17.0.0",
"terser": "^5.14.1",
"typescript": "^4.7.4"
},
}
and the companion server looks
import express from 'express'
import bodyParser from 'body-parser'
import companion from '@uppy/companion'
import express_session from 'express-session'
const app = express()
// Companion requires body-parser and express-session middleware.
// You can add it like this if you use those throughout your app.
//
// If you are using something else in your app, you can add these
// middlewares in the same subpath as Companion instead.
app.use(bodyParser.json())
app.use(express_session({
secret: process.env.COMPANION_SECRET,
cookie: { maxAge: 43200 }
}))
// Construct the s3_prefix for the data
var s3_prefix = process.env.S3_DATA_PREFIX || ''
if (s3_prefix[s3_prefix.length-1] != '/'){
s3_prefix = s3_prefix + '/'
}
console.log("S3 Prefix: ", s3_prefix)
const options = {
secret: process.env.COMPANION_SECRET,
filePath: process.env.COMPANION_DATADIR,
server: {
protocol: 'https',
host: process.env.COMPANION_DOMAIN
},
sendSelfEndpoint: process.env.COMPANION_DOMAIN,
uploadUrls: [...],
allowLocalUrls: true,
providerOptions: {
s3: {
key: process.env.COMPANION_AWS_KEY,
secret: process.env.COMPANION_AWS_SECRET,
bucket: process.env.COMPANION_AWS_BUCKET,
region: process.env.COMPANION_AWS_REGION,
useAccelerateEndpoint: false, // default: false,
expires: 3600, // default: 300 (5 minutes)
acl: process.env.COMPANION_AWS_ACL, // default: public-read
getKey: (req, filename, metadata) => {
console.log(req, filename, metadata)
var filepath = `${s3_prefix}${metadata.site_id}/${metadata.camera_id}/${filename}`
console.log("File path: ", filepath)
return filepath
}
},
onedrive: {
key: process.env.COMPANION_ONEDRIVE_KEY,
secret: process.env.COMPANION_ONEDRIVE_SECRET
}
}
}
console.log('Options: ', options)
app.use('/', companion.app(options))
const server = app.listen(3020)
companion.socket(server)
Unfortunately, that doesn't seem to have done the trick. I've logged in to pod to confirm that the settings are there. I am still getting the following
companion: 2022-07-15T10:55:28.024Z [info] companion.client.version uppy client version @uppy/companion-client=2.2.1
companion: 2022-07-15T10:55:28.191Z [debug] null Instantiating uploader.
companion: 2022-07-15T10:55:28.194Z [debug] null Starting download stream.
companion: 2022-07-15T10:55:28.618Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-15T10:55:28.618Z [debug] 24687a6e uploader.socket.wait waiting for socket connection
companion: 2022-07-15T10:55:28.711Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-15T10:56:28.621Z [error] Error: Timed out waiting for socket connection
at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
I did notice that going to the endpoint referenced in the snippet returns
Cannot GET /.well-known/microsoft-identity-association.json
Potentially this is the issue? and the headers never get set as a consequence of the get method failing?
I did notice that going to the endpoint referenced in the snippet returns
Cannot GET /.well-known/microsoft-identity-association.json
Interesting, we should look into this.
companion: 2022-07-15T10:55:28.711Z [info] companion.client.version uppy client version 1.0.0
Are you on latest Uppy or on v1?
I believe I have version 3.6.0 though I understand how that 1.0.0 looks suspicious
from the package-lock.json
@uppy/companion 3.6.0
node_modules/@uppy/companion 3.6.0
node_modules/@uppy/companion/node_modules/cookie 0.4.0
node_modules/@uppy/companion/node_modules/depd 2.0.0
node_modules/@uppy/companion/node_modules/express-session 1.17.1
node_modules/@uppy/companion/node_modules/safe-buffer 5.2.0
That is @uppy/companion, can you post the version of @uppy/core and @uppy/onedrive
Oh, I didn't realise that I had to install them - I've only installed @uppy/companion and those packaged where all the packages with 'uppy' in their path.
My bad - will install them now
sorry to report that the problem persists
"@uppy/companion": {
"version": "3.7.0",
"resolved": "https://registry.npmjs.org/@uppy/companion/-/companion-3.7.0.tgz",
"integrity": "sha512-s2Lwlkd8VSiQcY880322Mfk9CJ6tylj9s/BIj7ZqYgP1R1uqAJuQ/3soflMmngrONRIgGje/2vVsWcoWj4Gz7Q==",
"requires": {
"@purest/providers": "1.0.1",
"atob": "2.1.2",
"aws-sdk": "^2.1038.0",
--
"@uppy/companion-client": {
"version": "2.2.1",
"resolved": "https://registry.npmjs.org/@uppy/companion-client/-/companion-client-2.2.1.tgz",
"integrity": "sha512-Y3E10NJLMfp/wjgthNhx3gJtT67fzFCPNPFwpNNRs5iJsW6PANhJ420eyMUFzfmEZ56ZzGYxr5pzJZx8YxHICQ==",
"requires": {
"@uppy/utils": "^4.1.0",
"namespace-emitter": "^2.0.1"
}
},
"@uppy/core": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/@uppy/core/-/core-2.3.1.tgz",
"integrity": "sha512-KV04X7ueYbYX1p37/i3QsoQSw8IDP8Yb+Bh9KNN0X2Vcun6K2VnNjhVtPmPXtyjDZooK7lVIqhRX8TZWcSfgSQ==",
"requires": {
"@transloadit/prettier-bytes": "0.0.7",
"@uppy/store-default": "^2.1.0",
"@uppy/utils": "^4.1.0",
"lodash.throttle": "^4.1.1",
"mime-match": "^1.0.2",
"namespace-emitter": "^2.0.1",
"nanoid": "^3.1.25",
"preact": "^10.5.13"
--
"@uppy/onedrive": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/@uppy/onedrive/-/onedrive-2.1.1.tgz",
"integrity": "sha512-2XMP3/v82raymJMRMIfGXamdnEC2zMDqMGN+VHNHwZ/emup0tWZHvf/QmtA0RvoChIBs21uKeuQsO4xmHFRoRQ==",
"requires": {
"@uppy/companion-client": "^2.2.0",
"@uppy/provider-views": "^2.1.1",
"@uppy/utils": "^4.1.0",
"preact": "^10.5.13"
}
},
Can you confirm it doesn't work when you try to reach you Companion instance in this setup?
https://codesandbox.io/s/uppy-dashboard-xpxuhd?file=/src/index.js
I've given it a go with no luck
const UPLOADER = "s3-multipart";
const COMPANION_URL = "https://companion.desc.dev.ngenius.ai";
companion: 2022-07-15T12:22:26.841Z [info] companion.client.version uppy client version @uppy/companion-client=2.2.1
companion: 2022-07-15T12:22:27.004Z [debug] null Instantiating uploader.
companion: 2022-07-15T12:22:27.005Z [debug] null Starting download stream.
companion: 2022-07-15T12:22:27.518Z [debug] null Waiting for socket connection before beginning remote download/upload.
companion: 2022-07-15T12:22:27.519Z [debug] 7f808863 uploader.socket.wait waiting for socket connection
companion: 2022-07-15T12:22:27.606Z [info] companion.client.version uppy client version 1.0.0
companion: 2022-07-15T12:23:27.518Z [error] Error: Timed out waiting for socket connection
at Timeout.<anonymous> (/app/node_modules/@uppy/companion/lib/server/Uploader.js:370:32)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
Hi!
- Are you running a single instance of companion or multiple?
- Are you using something in front, like a reverse proxy?
- Do you see any websocket request being sent from your browser in your developer tools when you start the upload? is it sending to the correct host/post (the companion server's?)
- Do other providers like for example google drive work?
Could you try to create a new project just with companion and one-drive.
or checkout the uppy repo, and then try to run it in development mode, to see if that works. Follow steps from here:
https://uppy.io/docs/contributing.html
then create a .env with the needed content (see .env.example), then run:
yarn run dev:with-companion
This is just to make sure that it fails with the same error without any other variables, and to make sure all packages are up to date.
Hi,
- Companion is running as a Kubernetes deployment with a single pod on EKS.
- The deployment is sitting behind a classic load balancer (who is responsible for handling the SSL certificate)
- When the upload is 'started', I can see two requests being made to the companion server address. The requests are: a 200 successful POST+preflight with the following data
Query Parameter
driveId: b!GLuy5kLKGUuZCP_5jwsaJbdC-jgYs9VFiJAq0tui--D_eSliqKaDQ6byKbTE999H
Payload
{"fileId":"01ZJEIQDESANF6PWMNTNCILRP2UJBD6Z5G","protocol":"s3-multipart","size":128,"metadata":{"name":"details.txt","type":"text/plain","organisation_id":"***","site_id":"***","camera_id":"***"}}
Response
{"token":"81bcc1f4-f3bf-4902-8c10-5aa1e7d81980"}
And then a 204 Options request with just the query data Query Parameter
driveId: b!GLuy5kLKGUuZCP_5jwsaJbdC-jgYs9VFiJAq0tui--D_eSliqKaDQ6byKbTE999H
- I have not yet tried the other providers, local uploads are working correctly.
I will try creating a new project and giving it a go, though I will not likely be today sorry.
Thanks. what about the websocket though?
In your companion log, after this line:
companion: 2022-07-24T06:25:37.477Z [debug] e8274265 uploader.socket.wait waiting for socket connection
...you're supposed to see:
companion: 2022-07-24T06:25:37.520Z [info] socket.connect connection received from e8274265-1ec4-4a94-a416-0b8da8f7769e
companion: 2022-07-24T06:25:37.521Z [debug] e8274265 uploader.socket.wait socket connection received
so it would be interesting to see whether the browser sends the websocket request and what happens to it. I suspect that it gets lost in the proxy.
in chrome after downloading a file it's supposed to look like this:
(the first request is the websocket connection)
Hi @mifi
I'm very sorry for not getting back to you - I haven't been able to focus until this week.
You were bang on the money - the websocket was not being established because the AWS classic ELB doesn't support websockets. Once updating the cluster to use Network load balancers everything started to work correctly.
Sorry again for the delay and thanks for your help!