Find a way other than chrome.storage.sync to store user data

[xcv58]

[Vermeille]

Hi.

Why? I was about to ask "Can you expose chrome.storage.sync in the scripts so that they can store data?", but I guess you'll refuse, given this issue.

So, out of curiosity, why do you want to get rid of it? (and how could we store the scripts' data?)

Thanks :)

[xcv58]

First of all. when I said data I mean the user's script not something user will use to store the actual data.

The reason to look this direction is that the chrome.storage.sync has some limitations, i.e. the quota bytes per item (the scripts size), max items (how many websites you can use). More in https://developer.chrome.com/apps/storage#type-StorageArea

And there are already several users directly report this limitation is really annoying. That's the motivation to look for other options can store and sync user's scripts without such small limitation.

For your question:

"Can you expose chrome.storage.sync in the scripts so that they can store data?"

It's definitely NO because above reason. In addition, the user may corrupt their own scripts by misuse the API because there is no separation in chrome.storage.sync.

I'm looking for several solutions:

1. https://scaphold.io/, it supports GraphQL query and really easy to use
2. https://firebase.google.com/, it provides nice library to integrate
3. http://www.graph.cool/cloud

All of above have a free plan and the free trial quota is quite large compares to chrome.storage.sync.

chrome.storage.sync has 100KB limit per user, consider current users 10, 000 -> 1GB. The free plan is enough and with a scale-up option.

[xcv58]

But chrome.storage.sync does have some benefits:

1. Offline support
2. Seamlessly sync
3. Isomorphic in both webpage and background process, which means no need to worry about CORS issue

[Vermeille]

Thank you for your quick and detailed answer!

FYI, to store user data (ie, not the scripts, but the data they may want to store and sync), I considered https://keyvalue.xyz/ which could be used by a 5yo. Though, given CORS and stuff (I'm not super familiar with JS), I might not be able to use it until it's in the manifest.

Btw, how would you load an external library without being killed by the policy enforcement? I'd really love to use html2canvas, but didn't find a way to load the script. EDIT: Nevermind, I found the "load external libraries", though you may want to fix the comment or the code. The comment says "one per line", while the code expect them to be semi-colon separated.

[Vermeille]

Why just not using GitHub actually?

[xcv58]

Couple reasons:

1. Privacy, some users want their scripts not public. I know it's doable via private repo (paid) or private gist (free but has scalability issue).
2. Scalability reason as I mentioned above. In addition, users may store scripts for many websites, we need a way to ensure loading one website's script without loading all scripts. My intuition is that GitHub would be hard for this.

But GitHub does have some pros:

1. GraphQL API endpoints ready
2. Built-in version control
3. Free

And I do know some projects using GitHub as storage mean. So I'll evaluate GitHub definitely. Thank you!

[lulu5239]

J'ai trouvé la solution : 1- Aller dans votre dossier avec votre nom, "AppData", "Local", "Google", "Chrome", "User Data", "Default", "Extensions", "ddbjnfjiigjmcpcpkmhogomapikjbjdk" puis chercher dans les dossiers un fichier "manifest.json". 2- Ajouter dans la liste "permission" "unlimitedStorage". Fini !

[xcv58]

@lulu5239 could you please use English?

[lulu5239]

TRANSLATION OF GOOGLE TRANSLATION: I found the solution : 1- Go to your folder with your name, "AppData", "Local", "Google", "Chrome", "User Data", "Default", "Extensions", "ddbjnfjiigjmcpcpkmhogomapikjbjdk" then search in folders a file " manifest.json. " 2- Add to the list "permission" "unlimitedStorage". Finished !

[xcv58]

@lulu5239 Thanks.

The unlimitedStorage is not a valid solution.

Provides an unlimited quota for storing HTML5 client-side data, such as databases and local storage files. Without this permission, the extension or app is limited to 5 MB of local storage.

https://developer.chrome.com/extensions/declare_permissions

Our limitations is Google chrome sync. https://developer.chrome.com/apps/storage

[lulu5239]

Alors la dernière solution est de dupliquer l'extension puis de la modifier.

So the last solution is to duplicate the extension and then modify it.

[xcv58]

@lulu5239 Sorry I don't understand what you said.

[lulu5239]

It is necessary to copy the file of the extension and to modify it.

[RobKohr]

It would be nice if the editing box alerted you to when the limit is hit so you can reduce the size of things to fit in the limit (rather than finding out when you try to save)

[xcv58]

Thanks for suggestion @RobKohr

I was considering the same thing. But alerts are annoying. How about change the color and put some warning in the bottom of the size indicator?

[askrenes]

Might consider adding a JS minification function. Won't solve the problem of additional storage, but it would help shrink the bytes needed to be stored.

I also like the GitHub storage option.

[xcv58]

Thanks, @askrenes As you said minification wouldn't solve the problem because it can only save limited space and it's one-way function, the user can't edit or even read the code after minification.

Beside GitHub, I also consider the Dropbox as file storage solution.

[Vermeille]

GitHub is so much better :)

You might even consider automatically adding a header with some keywords so that a search on github will reveal extensions written for CJS2 :)

[adamtaylor13]

Hey this limit is pretty... well.. limiting.. while I try to build a fairly large script. Any work arounds for the time being or is this still an active consideration to use GitHub repos to store scripts?

[xcv58]

@adamtaylor13 you can use this workaround for now: https://github.com/xcv58/Custom-JavaScript-for-Websites-2/issues/24#issuecomment-349041736

[adamtaylor13]

@xcv58 Wouldn't that get stopped by CSRF stuff?

[xcv58]

@adamtaylor13 the host website has to support CORS request.

[adamtaylor13]

Following up here: Have you considered using IndexedDB for this? I've used it for a recent extension I was developing and it's very fast + easy to use. Would also remove any kind of size limitation.

[xcv58]

Thanks @adamtaylor13 . There are two issues with IndexedDB.

It's unique to an origin:

Each IndexedDB database is unique to an origin (typically, this is the site domain or subdomain), meaning it cannot access or be accessed by any other origin. https://developers.google.com/web/ilt/pwa/working-with-indexeddb#what_is_indexeddb

It has no way to sync the scripts across machines.

[adamtaylor13]

@xcv58 What about some sort of AWS solution that allowed larger scripts to be uploaded to some sort of S3 bucket? Then folks who need/want bigger scripts could just pay a small fee for "Pro" version which would cover costs for the S3 and possibly make you a bit of money on the side. ¯_(ツ)_/¯

Just an idea.

[xcv58]

Yup, S3 is a good solution for immutable scripts. It may not works well for frequently updating scripts.

[xcv58]

Possible candidates:

• AWS AppSync, https://aws.amazon.com/appsync/
• Dropbox, https://github.com/dropbox/dropbox-sdk-js
• Prisma, https://www.prisma.io
• Firebase, https://firebase.google.com
• 8base, https://www.8base.com

Something we missed from the native chrome sync:

• Offline support
• Seamlessly sync
• Isomorphic in both webpage and background process, which means no need to worry about CORS issue
• Seamlessly Authentication & Authorization

[xul8tr]

What about using a javascript based text/object compess/decompress? This won't solve the issue but it will give you much more available space

[xcv58]

It can save some space but still limited by the chrome.sync limitation.

[xul8tr]

It should save up to 90% space, I wouldn't call "some". Plain text is very compressible. I did a test with lzwCompress.js and I was around 90% compressed. So though it doesn't solve the issue, it does give way more space than before, and something is always better than nothing. Should be fairly easy to implement, basically, you just compress on save and decompress when you retrieve it.