remotestorage.js
remotestorage.js copied to clipboard
remotestorage
Added Solid support
Hello, I have implemented the Solid backend. These changes are compatible with the widget. I am creating another pull request for the widget as well.
The test code and documentation updates are also there...
Thank you :-)
Great to see a PR for another open protocol!
Feel free to also notify people on the community forums about these PRs: https://community.remotestorage.io/t/adding-solid-as-a-backend/828
Maybe you could also add some information for people new to SOLID (like myself e.g.) about how to test this end-to-end (i.e. which server implementation and.or popular provider/hoster to use).
- Have you had a chance to run the API test suite against this?
The API test suite is irrelevant here, as it tests the compliance of RS servers with the RS specification. However, this PR is for clients connecting to Solid servers, as an alternative to RS servers.
Another note for tests: there is a new Mocha suite for the RemoteStorage class in this open PR: https://github.com/remotestorage/remotestorage.js/pull/1318
- Have you had a chance to run the API test suite against this?
The API test suite is irrelevant here, as it tests the compliance of RS servers with the RS specification. However, this PR is for clients connecting to Solid servers, as an alternative to RS servers.
Right you are! I've been wrestling with Armadietto for half a year. :-S
When testing with a real app, does calling setAuthURL() eliminate the need for the updated Widget? And what object has that method? dropbox-and-googledrive.md is somewhat vague.
More generally, when I pull this version of remotestoragejs into a real app, what do I need to do to connect to a pod? (I have a pod on Inrupt)
- Do you know if the Solid library is using fetch or XHR? We've been supporting environments with either one; if Solid only uses fetch we'll need to document that.
I am almost certain that it only uses fetch. I can do a through inspection and report back if necessary.
- I see you've written tests using Jaribu. Unfortunately, that has been deprecated and doesn't run under recent versions of Node.js. New tests are being written using Mocha. I know it's a pain, but could you please add Mocha tests for Solid?
I've tried to follow the pattern of the rest of the code. But I've missed the recent updates. I'll try to match and let you know.
Let me elaborate on the pod-not-selected state. Each Solid account can have multiple storages/pods connected to it or even none, by the protocol. But in practice there is always only one storage connected to each account.
After a user logs in into their Solid account, their gonna have to select which pod they want to use. So pod-not-selected is not essentially an error state but just another step in the connection process. The the logic for the widget automatically passes this event if there is only one pod. If there is none, throws an error. And if there is multiple, asks the user to choose one from the list.
My line of logic is that, if the developer is using the widget, they don't need to worry about anything. The Solid option does not even show up if they don't put it in the widget's configuration. They need to worry about it only if they are bypassing the widget and intent to add support for Solid. It doesn't affect the existing apps.
As for the setAuthURL() function, we have a similar situation. Solid does not have a specific backend as it is only a protocol. So it needs to be selected. But on the other hand we don't need an API key or other configurations like Google drive and Dropbox do.
So before asking the Solid backend to connect, it needs to be told what to connect to. The widget shows some options and also an input. If bypassing the widget, then we assume that the developer already has one in mind or asked the user via some other process.
We call it "authURL" on rs. Solid calls it "Identity provider".
Btw thank you guys for attending to this PR so quickly. This is very refreshing and welcoming 😊
Maybe I need to add a full documentation on how to directly incorporate the Solid backend without using the widget?
When testing with a real app, does calling setAuthURL() eliminate the need for the updated Widget? And what object has that method?
It does. The object that has the method is Solid. Accessible via remoteStorage.solid.setAuthURL(). But we would still have the pod-not-selected step which needs to be called if the developer is not using the widget.
dropbox-and-googledrive.mdis somewhat vague.
I'll fix it. I believe I renamed the file to dropbox-googledrive-solid.md.
More generally, when I pull this version of remotestoragejs into a real app, what do I need to do to connect to a pod? (I have a pod on Inrupt)
const connectTask = setTimeout(() => {
remoteStorage.solid.setAuthURL('your-solid-identity-provider'); // i.e. https://login.inrupt.com
remoteStorage.solid.connect();
// Calling 'connect()' will immediately redirect to your identity provider website.
}, 1000);
remoteStorage.on('pod-not-selected', () => {
clearTimeout(connectTask);
const podURLs = remoteStorage.solid.getPodURLs();
// Choose one. Maybe there is even 0?
remoteStorage.solid.setPodURL(podURLs[0]);
// That's it. 'connected' is fired immediately.
};
remoteStorage.on('connected', () => {
// We are connected.
clearTimeout(connectTask);
// We arrive here either through calling 'setPodURL' on the `pod-not-selected` event or
// on page getting loaded. Because: Inrupt Solid library prefers to always redirect to the
// identity provider to get the session tokens. I spent a lot of time on this. If we want to
// change it, we'll have to make fundamental changes to Inrupt's library and make a PR.
});
Calling connect() redirects the page. And we get back to our homepage with a payload containing the session data. So whenever our home page is loaded, we are either not connected at all or in the process i.e. we get a session but have not selected the pod yet, or a pod was already selected and we immediately jump to the connected state.
My line of logic is that, if the developer is using the widget, they don't need to worry about anything. The Solid option does not even show up if they don't put it in the widget's configuration. They need to worry about it only if they are bypassing the widget and intent to add support for Solid. It doesn't affect the existing apps.
Great! Please add a comment in the code and update the event documentation, to that effect.
If you have an app that uses this, it would help if we could look at it.
I am almost certain that it only uses fetch. I can do a through inspection and report back if necessary.
I believe the vast majority of browsers in use support fetch, but the API didn't fully stabilize in Node.js until v21. Security support for v20 ends on 30 Apr 2026 (in 1 year and 8 months) If I understand correctly, the implementation in v18 and v20 doesn't support streams fully.
So, we'll need to fully test this under Node 18 and 20, to see if this works properly with them. If it doesn't, I'm okay with documenting that this version of remotestorage.js when used in Node.js, should only be used with v21 and later.
According to our docs, fetch is available from node.js 18 onwards, and we already ask to add a polyfill for when it's not. So the usage of fetch in a new back-end shouldn't change anything for node.js users or in our docs.
https://remotestorage.io/rs.js/docs/nodejs.html#caveats
I've updated the documentation and added mocha tests. This should be it as far as I followed the conversation. What do you guys think?
I haven't been able to use this in a real app yet — the app I've been testing with is throwing up issues that don't appear to be caused by this. I'll have to try with a different app.
Hi there, I don't know much about remoteStorage; but I've been working on Solid Apps and something crucial to Solid is using RDF for interoperability. Seeing this PR, I understand that this adds support for a Solid backend as a "file storage", right?
I wonder how would anyone use this library to make a Solid application using RDF. Would the developers need to write RDF manually? What about updates to documents?
Anyways, I'm just mentioning this in case you weren't aware. If that's already the intention, using Solid as a file storage rather than a store of RDF documents, that's fine. But I think using a Solid POD like that is missing a lot of the potential. I guess it's better than nothing though, if it adds "Solid support" to some of the existing apps implemented using remoteStorage. But I don't think I'd recommend anyone making a Solid App from scratch to follow this approach.
By the way, in case it's useful for anyone, some years ago we had a discussion comparing remoteStorage, Fission, and Solid. And this was one of the points that came up: https://vimeo.com/779640162
Anyways, I'm just mentioning this in case you weren't aware. If that's already the intention, using Solid as a file storage rather than a store of RDF documents, that's fine. But I think using a Solid POD like that is missing a lot of the potential. I guess it's better than nothing though, if it adds "Solid support" to some of the existing apps implemented using remoteStorage. But I don't think I'd recommend anyone making a Solid App from scratch to follow this approach.
Yes, you won't get many of the benefits of a native Solid app. But this is an important step in moving people to storage independent of apps. With this, someone who has a Solid pod can also use the same storage for Apps written for remoteStorage.
Wouldn't that be something that could be supported in the data modules for this library?
Wouldn't that be something that could be supported in the data modules for this library?
This was the idea when we were considering the task. The data module receives the Client instance which probably exposes the RemoteStorage hence the Solid backend which then you can get the raw Solid Session from it.
So a data module can implement a specific behavior for dealing with Solid the way it is recommended. The downside is that in order to be Solid friendly is that the different data modules need to have a check for the back-end type and do different thins.
I've tried adding this and the updated widget to a simpler app, but no option for Solid storage appears in the list. Is there an example of this running in a real app?
I'm trying this out in https://github.com/remotestorage/myfavoritedrinks/pull/15
There seems to be a bug related to folder listing. Storing 'beer' as my favorite drink works, but then it repeats GET 'https://michielbdejong.solidcommunity.net/myfavoritedrinks/' -H 'accept: text/turtle' in a loop and 'synchronizing' spins indefinitely. This is the response body. Is it supposed to fetch Turtle, or was the intention to fetch the folder listing as JSON-LD?
@prefix : <#>.
@prefix dct: <http://purl.org/dc/terms/>.
@prefix ldp: <http://www.w3.org/ns/ldp#>.
@prefix stat: <http://www.w3.org/ns/posix/stat#>.
@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.
@prefix myf: <>.
@prefix json: <http://www.w3.org/ns/iana/media-types/application/json#>.
myf:
a ldp:BasicContainer, ldp:Container;
dct:modified "2024-12-11T13:42:10Z"^^xsd:dateTime;
ldp:contains myf:beer;
stat:mtime 1733924530.183;
stat:size 4096 .
myf:beer
a json:Resource, ldp:Resource;
dct:modified "2024-12-11T13:42:10Z"^^xsd:dateTime;
stat:mtime 1733924530.183;
stat:size 88 .
I found a bug in this PR. The code that produces folder item maps is not including ETags (lines 414 and 419 of src/solid.ts) But sync considers an items map without ETags as corrupt. If the server doesn't give ETag headers, I think we can just use the last-modified dates as etags.
@DougReeder
Is there an example of this running in a real app?
That was a good question! With a few changes I made, it's working in https://github.com/remotestorage/myfavoritedrinks/pull/15 now.
So what should be our next step?
I'll get back to this in May in preparation for https://www.localfirstconf.com/ where I can hopefully give a presentation about it I also mentioned it in https://www.youtube.com/watch?v=i66WxqUz0u8&t=1990s with a shout-out to @yasharpm for this awesome work :)
Is there anything that you might need assistance with to merge this update? I'm also really interested in seeing this in RemoteStorage 😄
all we need is documentation of 'pod-not-selected', and anything else that a app programmer needs to know when integrating this.
On an unrelated note, I'm also not entirely comfortable with adding large new dependency chains to the library. But I don't know enough about SOLID to gauge if it's necessary or not.