Memex
Memex copied to clipboard
WorldBrain
Fix disk full, lost all data #Bug #Help
macOS, 10.15.3 (19D76) Vivaldi, 3.0.1874.23 (Stable channel) (64-bit)
If reporting a bug:
- Can you describe the problem and bug in more detail?
I woke my machine today and all my (browser) data is gone (i.e. all bookmarks)
My last manual backup was 2020-03-04 so I've lost almost two months of data
After further research it appears to be linked with the low disk space of my machine which happened overnight while I was away
I also understand there's some type of warning related to quota/disk space - how does this work because I've never seen it? If it's generated by the backup helper (desktop) then I have this turned off until I need it for backups (i.e. once per week/month) so, if they're not already then, warnings must be generated by the extension (too). Also, in case this feature is in a newer release of the backup helper, how do I discover which version/release of the backup helper I'm running?:
I inspected the local (browser) storage (i.e. IndexedDB) and no data
I inspected the backup helper (desktop) but couldn't find any useful data - if I'm able to [salvage] anything then please provide instructions?
Where are the log files to tell me what went wrong here?
FYI I freed up some disk space, restarted my machine, reinstalled the extension and it's functioning as expected now, albeit with new data
- How can we replicate the issue?
Max out a disk/partition/container
- Expected behavior (i.e. solution)
Keep my data safe
- Error stack (from extension crash page)
Nothing crashed but please direct me to logs if you think they're relevant
- Other comments
I'll keep it simple because I don't know the story behind the decision but manual backups f#cking suck and paywalling auto backups leaves a sour taste in my mouth and doesn't make me want to give you money. Paywall >1 device and/or mobile like every other paid app, where there's a legitimate reason to cloud data and cover your costs. But get it running CONCRETELY on a single desktop machine first so I have trust and confidence in your brand.
There's still no simple structured data export (e.g. JSON) and nothing on roadmap - why, is this going to be part of your API?! Usually when I start using an app and there's no suitable portability features I stop using it but I regrettably took a chance on memex. Never again.
FYI the only reason I'm posting this is so it doesn't happen to anyone else but you've lost my trust and I'm currently looking for alternative software
Hope to hear back
Sincerely
Related:
https://github.com/WorldBrain/Memex/issues/675 https://github.com/WorldBrain/Memex/issues/783 https://github.com/WorldBrain/Memex/issues/806 https://github.com/WorldBrain/Memex/issues/816 https://github.com/WorldBrain/Memex/issues/970
[salvage]:
ldexterldesign@Lewiss-MacBook-Pro ~ % tree /Users/ldexterldesign/Library/Application\ Support/memex-backup-helper
/Users/ldexterldesign/Library/Application\ Support/memex-backup-helper
├── App\ Shim\ Socket -> /var/folders/vp/9hbhc9jj2894cv04smb3n1l40000gn/T/chrome-qto1x7/Socket
├── BrowserMetrics-spare.pma
├── Crashpad
│ ├── completed
│ │ ├── 3fd6cb7c-3396-457c-b87a-ca14ec81d5d7.dmp
│ │ ├── e610ae62-7632-401c-a4c4-fb980239fb62.dmp
│ │ └── f15b579e-2589-45ba-b725-47b64cddfcdf.dmp
│ ├── new
│ ├── pending
│ └── settings.dat
├── CrashpadMetrics-active.pma
├── Default
│ ├── 000101.log
│ ├── CURRENT
│ ├── Cookies
│ ├── Cookies-journal
│ ├── Extension\ Rules
│ │ ├── 000003.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ └── MANIFEST-000001
│ ├── Extension\ State
│ │ ├── 000003.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ ├── LOG.old
│ │ └── MANIFEST-000001
│ ├── Favicons
│ ├── Favicons-journal
│ ├── GPUCache
│ │ ├── data_0
│ │ ├── data_1
│ │ ├── data_2
│ │ ├── data_3
│ │ └── index
│ ├── History
│ ├── History\ Provider\ Cache
│ ├── History-journal
│ ├── LOCK
│ ├── LOG
│ ├── LOG.old
│ ├── Login\ Data
│ ├── Login\ Data-journal
│ ├── MANIFEST-000100
│ ├── Network\ Action\ Predictor
│ ├── Network\ Action\ Predictor-journal
│ ├── Network\ Persistent\ State
│ ├── Preferences
│ ├── README
│ ├── Search\ Logos
│ ├── Secure\ Preferences
│ ├── Shortcuts
│ ├── Shortcuts-journal
│ ├── Site\ Characteristics\ Database
│ │ ├── 000004.ldb
│ │ ├── 000006.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ ├── LOG.old
│ │ ├── MANIFEST-000001
│ │ └── lost
│ │ ├── 000003.log
│ │ └── MANIFEST-000001
│ ├── Sync\ Data
│ │ └── LevelDB
│ │ ├── 000003.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ ├── LOG.old
│ │ └── MANIFEST-000001
│ ├── Thumbnails
│ │ ├── 000003.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ ├── LOG.old
│ │ └── MANIFEST-000001
│ ├── Top\ Sites
│ ├── Top\ Sites-journal
│ ├── TransportSecurity
│ ├── Visited\ Links
│ ├── Web\ Data
│ ├── Web\ Data-journal
│ ├── blob_storage
│ │ └── 89ffc05d-8a02-45a6-ac51-5c8c4138b16e
│ ├── data_reduction_proxy_leveldb
│ │ ├── 000101.log
│ │ ├── CURRENT
│ │ ├── LOCK
│ │ ├── LOG
│ │ ├── LOG.old
│ │ └── MANIFEST-000100
│ ├── page_load_capping_opt_out.db
│ ├── page_load_capping_opt_out.db-journal
│ ├── previews_opt_out.db
│ └── previews_opt_out.db-journal
├── First\ Run
├── GrShaderCache
│ └── GPUCache
│ ├── data_0
│ ├── data_1
│ ├── data_2
│ ├── data_3
│ └── index
├── Local\ State
├── NativeMessagingHosts
├── RunningChromeVersion -> 71.0.3578.98
├── ShaderCache
│ └── GPUCache
│ ├── data_0
│ ├── data_1
│ ├── data_2
│ ├── data_3
│ └── index
├── SingletonCookie -> 10952549031799627200
├── SingletonLock -> Lewiss-MacBook-Pro.local-2205
├── SingletonSocket -> /var/folders/vp/9hbhc9jj2894cv04smb3n1l40000gn/T/.io.github.nwjs.memex-backup-helper.J0w6as/SingletonSocket
└── WidevineCdm
└── 4.10.1440.18
├── LICENSE
├── _metadata
│ └── verified_contents.json
├── _platform_specific
│ └── mac_x64
│ ├── libwidevinecdm.dylib
│ └── libwidevinecdm.dylib.sig
├── manifest.fingerprint
└── manifest.json
27 directories, 105 files
I read another issue here where someone asked why you don't have a useful export (e.g. JSON/XML) and so created a JS script to salvage some useful export data but can't find it now...
OK, so I have another extension with lost IndexedDB data too so I think that this problem can be attributed to Vivaldi
I still think it's a nightmare use case to consider remedying
Telling a user / potential customer who's lost data that they should have a) not let their disk fill up and/or manually backed up more often and/or paid for auto backups is not gonna fly
Sincerely
NB because I value my privacy I'm deleting the following every time I close my browser:
Usually the fact I'm not deleting "storage" means my extension data stays intact but if my machine has low disk space is there a conceivable way memex data could be lost through "Applications cache" and/or "Cache" deletion?
Sincerely
FYI https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria
I think the moral here is this highlights the reliability (or lack of) browser storage
Sincerely
Thanks @ldexterldesign for all this information about your case and I am terribly sorry this happened to you.
We actually have a notification on low disk storage implemented, which works on Chrome, Brave and Firefox. Vivaldi is not officially supported/tested so I don't know if it fails there.
Indeed the browser storage is not reliable, that is why we started building StorexHub our API and application platform. With it you can write your own export plugins already. We will do it eventually ourselves though. In the mid-run all the database and application logic of Memex will be running there - one of the reasons that there is no database cap anymore.
Besides that I agree that the auto-backup function is due to be a free feature. I added it to our ASAP tasks - maybe already going out with the next release. Other than that I'll add a section in the on-boarding process making clear about the recommendation to enable the backup feature.
Seen the issue I opened for Opera was referenced here. From my side I have to tell that I had a decent amount of disk space left and the data still seemed to have been wiped clean. It was a one-time occurrence out of the blue, since then it seems Memex is working fine. Not an expert on browsers, but my wild wild guess would be lack of consistency in the way browser storage is handled across different browsers. I am eager to see a version that does not use browser storage this way (I'll probably jump on a subscription at the same time then).
Besides that I agree that the auto-backup function is due to be a free feature. I added it to our ASAP tasks - maybe already going out with the next release.
🙏
Hi @blackforestboi,
Besides that I agree that the auto-backup function is due to be a free feature. I added it to our ASAP tasks - maybe already going out with the next release.
Can I start using Memex safely yet?
Sincerely
We have no control over the eviction policy of the browsers, so no the OPs bug is not solved (as in it is not a "bug").
However I suspect that since you didn't get the warning about low disc space, your data eviction came from another place. That is that sometimes during schema upgrades something goes wrong and this causes the browser to evict data too. There has been a fix introduced by Dexie.js, our underlying storage tech, that has solved that particular issue.
So we assume it has been fixed, but we are not 100% sure yet. Either way it is highly recommended to still do the backups via the backup helper.
Hi @blackforestboi,
Thanks for reply
Are you making the auto-backup function a free feature?
Sincerely