Tested on s3 backed by localstack

[anudeepreddy]

Hi,

I have tested this with a little modifications on the rsapi side to get it to work with s3 backed by localstack and I think it's working. I am able to see objects created in the s3 bucket.

[bcspragu]

Thanks for testing it out! I hadn't even considered using localstack to test. The telltale signs that it's working are:

1. Logseq shows the green dot on the cloud sync icon
2. Your S3 bucket is populated at <user-id>/<graph-id>/e.<random hex identifiers>, those should be the encrypted files

eg

(looking at that screenshot, the filetypes being inferred as random strings is probably harmless, but also not great, filed #4)

[anudeepreddy]

The main motive to test with localstack is to use it during development instead of deploying an s3 bucket on AWS.

And I do see the signs you mentioned.

Thank you for working on this

[bcspragu]

I have tested this with a little modifications on the rsapi side to get it to work with s3 backed by localstack

For posterity, what other changes did you make to the rsapi package? I'm trying to keep track of all the necessary upstream changes, since those will need to be exposed via some interface in Logseq's settings at some point

[anudeepreddy]

since amazonaws.com is hardcoded in s3-presign crate, I have directly used the presigned_url function from the crate which accepts url as input to generate a presigned url.

I just got a chance to look at your fork of rsapi, May I know the reason to use rust-s3 crate?

[bcspragu]

May I know the reason to use rust-s3 crate?

The short answer is that I was debugging some signing issues. The longer answer is that I believe that the s3-presign does indeed fail under specific (still unknown) circumstances, see the upstream bug for all the details.

[enigma]

Thank you @bcspragu and @anudeepreddy for working on this!

I'm dipping my toe and managed to make my local electron logseq talk to my local logseq-sync server. At some point though the electron app tries to reach the sync server via a hardcoded url, causing [0].

@anudeepreddy do you happen to have your rsapi/localstack work published anywhere?

[0]

› update remote files: reqwest error: HTTP status client error (404 Not Found) for url (https://foiv3ndsvlkkas.frp.bsprague.com/file-sync/get_temp_credential)
Error occurred in handler for 'main': [Error: reqwest error: HTTP status client error (404 Not Found) for url (https://foiv3ndsvlkkas.frp.bsprague.com/file-sync/get_temp_credential)] {

[bcspragu]

frp.bsprague.com is my personal frp instance. The reason you're hitting that is because I've built my own rsapi packages (here), and those point to my own server. You're likely using one of my forks of the Logseq codebase that updated the packages.json to use my rsapi forks.

The short term solution is to build your own rsapi fork with URLs to your instance (e.g. some localhost-like domain, but I think it needs HTTPS unless you update that too), the longer term solution is that Logseq passes in the URL as configuration to the rsapi package, so that these things are not hard-coded everywhere.

[enigma]

Yes, I did replicate most of your fork by hand just to get a sense of where the hooks are and made my logseq point to your rsapi package.

I was curious about whether anyone already had published a version of rsapi that points to localhost. If not I'll go ahead and do it as it might be useful to share for development.

I agree that, as you suggest, longer term there should be a way to specify the endpoint from the call site. I haven't digged enough to see how many points that has to touch within logseq codebase.

but I think it needs HTTPS unless you update that too

I did have to disable CORS in logseq-sync to let logseq web (on :3001) be able to call logseq-sync (on :8000), but it turned out to be necessary only when using logseq on a browser and not via electron. Later I discovered sync only works on electron/mobile, rsapi is not called from web, I'll check what happens for rsapi.