feat: add uploadthing driver

[juliusmarminge]

🔗 Linked issue

#389

❓ Type of change

• [ ] 📖 Documentation (updates to the documentation, readme, or JSdoc annotations)
• [ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
• [ ] 👌 Enhancement (improving an existing functionality like performance)
• [x] ✨ New feature (a non-breaking change that adds functionality)
• [ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
• [ ] ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

📝 Checklist

• [x] I have linked an issue or discussion.
• [x] I have updated the documentation accordingly.

[juliusmarminge]

@pi0 just tried to test this out and we stop usage of the utapi from client (mostly cause they shouldn't leak their api keys there)

got any other good way to test this?

---

oh right and our api also doesn't enable cors so it cannot be used from browser anyways :)

[pi0]

We have unit tests you can run with pnpm run vitest (unstoage also targets server). We mainly need to mock endpoints (responding from mobile if couldn’t find it please tell me to push commit)

[juliusmarminge]

I just realized that in order to support this we'd need to allow setting the key manually. right now uploadthing generates the key for you

[juliusmarminge]

I just realized that in order to support this we'd need to allow setting the key manually. right now uploadthing generates the key for you

@pi0 is there a concept of mapping keys inside unstorage? For example when we setItem we get the key from UT, that we could then map to the unstorage key, i.e:

storage.setItem("foo")
// internally map foo -> KEY_FROM_UT
storage.getItem("foo")
// key = map["foo"], fetch(utfs.io/f/key)

[pi0]

I think we mainly need a storage for keys which is usually same storage layer.

If your API is designed to always return random names, I am up to consider a new change that setItemRaw (setItem is mainly for text values) returns the random name. It is a non breaking change and i hope to unblock this presets also any other preset that need this.

(in the meantime, i guess it would be also nice if you consider supporting custom names. let's see which comes first haha)

[markflorkowski]

(in the meantime, I guess it would be also nice if you consider supporting custom names. let's see which comes first haha)

The reason we don't is because names must be URL-safe, and have a length limit.

[pi0]

Hmm, I see. What if we make the driver make sure that input is url-safe and respects the limit like any fs does (so it throws an error if it doesn't comply) I think users of the driver, would understandably follow this and probably can benefit more from such control vs random ids. Thoughts?

[juliusmarminge]

So I did a thing to track internal vs uploadthing keys - not sure if it aligns with what this project aims to do though - I'll let you be the judge.

Currently passing all tests except clear since our API throws a 500 when you try to delete 0 files currently (will fix that momentarily)

[markflorkowski]

Currently passing all tests except clear since our API throws a 500 when you try to delete 0 files currently (will fix that momentarily)

Merged fix for this on the API side

[juliusmarminge]

current approach only works for a long-running server since it depends on that in-memory map. we're thinking about how to support user-provided keys

[juliusmarminge]

Now just gonna have to get the mocking right 😅

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 66.05%. Comparing base (f6841df) to head (6444a42). Report is 39 commits behind head on main.

:exclamation: Current head 6444a42 differs from pull request most recent head a54c648. Consider uploading reports for the commit a54c648 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
+ Coverage   64.79%   66.05%   +1.25%     
==========================================
  Files          41       41              
  Lines        4071     4186     +115     
  Branches      489      520      +31     
==========================================
+ Hits         2638     2765     +127     
+ Misses       1422     1411      -11     
+ Partials       11       10       -1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[juliusmarminge]

struggling getting msw to intercept the requests 🤔

[pi0]

yeah. msw is not the best in class nor supports ofetch which I am aware of and also the fix but still hesitated because by doing it we add an overhead to every user of it only because of msw.

I will try to check it ASAP.

[nuxt-studio[bot]]

✅ Live Preview ready!

Name	Edit	Preview	Latest Commit
unstorage	Edit on Studio ↗︎	View Live Preview	4d55f54aefc989042ae04d6a08305a608ec4004a

[juliusmarminge]

@pi0 updated to use native custom ids now.

there are some tests failing however that's expected, since they relate to deletion of files.

UT is not a KV store, so when a file is deleted in UT, it is not instantly deleted but it may take up to 30 seconds before it is deleted at the storage provider, and then deleted from our db. some of the tests here assume that calling hasItem(key) directly after deleteItem(key) should return false, which is not the case for uploadthing.

idk how you wanna pursue given that knowledge. alter the tests? custom test suite for UT?

---

cc @markflorkowski we could, delete the customId field when a file is marked for deletion, so that a new file may be uploaded with that same customId even before the file is removed. idk if that would make sense for us to do, but it would be a workaround to fix this issue

another thing that we could do is to filter out the files marked for deletion in the getFileUrl endpoint. this would be a breaking change to the API but I don't see why they shouldn't be filtered out in the first place...

[pi0]

Exciting updates 🚀 Sorry i'm quite busy still will try to come to this PR as soon as could also for failing tests.