Sefaria-Project
Sefaria-Project copied to clipboard
Sefaria
There should be a desktop app
Would be great if there was a "desktop" app using electron/chrome apps for offline for laptop use. It could download all the books and ideally allow local saving of sheets/notes, etc until online for syncing.
You can install our Android app on Windows using BlueStacks. It has an offline library, but no support for source sheets yet
Android emulators are heavy... would be a lot better to have a Chrome App or a real software (I'm on Mac). Always better to learn offline
Okay I have confirmed that using https://github.com/jiahaog/nativefier the endpoint http://www.sefaria.org/texts can be packaged as an electron desktop app no problem.
In terms of offline functionality, the issue is that the data it would need to copy is in a mongodb behind a server. In order for this offline app to be effective it needs to just be installed and thats it. We cannot require an end user to install all of these dependencies (python, mongodb, etc.) that are required. Because of this we would need to copy the nosql database to a local serverless datastore, something like https://unqlite.org/.
Then, I think Electron uses nodejs to access the file system and Electron can also intercept urls, so theoretically the calls to the api can be intercepted and redirected to some code that attempts to get the resource from the umbedded nosql store.
@nassan What if we made a read-only Mongo datastore available by url? Would that work?
I don't think so. The client (for both web and electron) speak to api endpoints. Those endpoints are on the django server I believe? That is what it seemed like from reading through the source code. Although wasn't immediately obvious to me the path from request>router>read mongo>respond with json. Either way, this python code queries mongo for and returns back the correct json.
So having the mongo datastore available on a url doesn't necessarily help for 2 reasons:
- The client doesn't speak to mongo, it speaks to the django server
- the url would have to be localhost for an offline app, and also require running mongo on the machine running the electron app.
Do you know the request>router>read mongo>respond with json works with the Sefaria backend? If I could understand that, I could possibly port it to Nodejs and read from an embedded version of the mongo datastore.
Thanks for looking into this. I've thought about this a little bit for my own project - here are some notes, references. BTW, it's not practical to have an offline mongodb instance unless you want to bundle a VM - but it's far too heavy and clunky for the masses and won't deploy nicely as an app.
The most ideal approach is allow an offline approach which has the ability to make changes and sync, whatever could sync. This is more work, but the best option for the users because it would increase the changes that people that are working offline can not just consume your content, but help create content (whether that is source sheets, translations, links, etc..).
For offline with sync - you can look at https://strongloop.com/strongblog/node-js-api-offline-sync-replication/ for some approaches. Also couchdb supports this with pouchdb, but as that's not your backend, you'd have to consider if it's worth it (but take a look at Mango.)
If you want to have a read-only mongodb (it's not really mongo.. but looks like mongo) you can look into NeDB or linvodb3. The advantage of this is that you're already familiar with the queries need to get the data you want - the problem is that this code currently exists in python on the server so you'd have to port it to js and maintain it just for the desktop app - which is not good. If the API was nodejs-driven already, you could reuse that code for an electron app - but that's not the case.
I don't think it's a terrible idea to have a different storage mechanism between the a web and desktop client. I haven't looked into your client-side code, but you'll want to have the resource abstracted out so you're not referring to urls all over your code - just called some objects which will use an adapter for how to translate the object request into a data request - eg into a URL or localdb query, etc..
Other links I found while writing this: https://github.com/forbesmyester/SyncIt http://stackoverflow.com/questions/22957902/best-practice-for-on-off-line-data-synchronization-using-angularjs-and-symfony-2 http://stackoverflow.com/questions/24384803/using-pouchdb-with-mongodb https://github.com/mWater/minimongo
Now that you're mentioning offline with sync, the way our iOS app works is that it downloads the library as a set of zip files. As you need to read a zip file it unzips it. The app is written in React Native which I assume cannot run as a desktop app, but all the basic structure of downloading our library as separate files and syncing is already built.
We also have a SQL export of our MongoDB which we use on the Android app. I assume that can much more easily be built into a desktop app: https://github.com/Sefaria/Sefaria-SQL
Now that you're mentioning offline with sync
just so it's clear there are no surprises here.. I mentioned in the description
It could download all the books and ideally allow local saving of sheets/notes, etc until online for syncing.
As I said, it's more difficult and might not be the first step. But I think it has a lot of value, especially for this content.
React native can work with destop apps, but it's not the main target: https://github.com/Microsoft/react-native-windows http://www.reactnative.com/build-os-x-desktop-apps-using-react-native/ But, you can also just build it using electron - you don't need to use react native (just react) (one example - there are many)
If you have a process that works for the iOS or Android apps, it seems a natural place to start. As I said, syncing add complications so you don't need to start there. And it's possible, that you don't even ever really need to fully support it, but it would be great to have offline actions be pushed to the server when they come online (not exactly a sync - but has certain similarities..)
Sorry @yehosef , you're right that syncing was clearly in the OP. I just realized that this problem has been mainly solved in our apps.
If you do plan on pursuing this project, I would recommend going the route of converting the iOS React Native app into a desktop app. It already has syncing support in it and the zip files should work on all platforms out of the box. I'd be happy to help if need support understanding the iOS code to get this done.
I'm not going to work on this for Sefaria. I'm saying that I think there is a big audience of sefaria users that would be interested in an offline-able desktop app. The idea of syncing is an extra bonus - but there is still value without this.
But it's encouraging that it's doesn't seem complicated and should work out of the box. I hope sefaria is able to give it some attention.
On Sun, May 21, 2017 at 4:17 PM, Noah Santacruz [email protected] wrote:
Sorry @yehosef https://github.com/yehosef , you're right that syncing was clearly in the OP. I just realized that this problem has been mainly solved in our apps.
If you do plan on pursuing this project, I would recommend going the route of converting the iOS React Native app into a desktop app. It already has syncing support in it and the zip files should work on all platforms out of the box. I'd be happy to help if need support understanding the iOS code to get this done.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sefaria/Sefaria-Project/issues/294#issuecomment-302936170, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJHBdOQEwpa6VVLub7NnzyiK4_yICJDks5r8DlUgaJpZM4L6GbR .
I would be interested in taking a look at this project. I could do electron, but how does a PWA/native offline support sound?
@EliezerIsrael If you or someone could briefly outline what this project requires, i would love to start on it. Also, concern: there's a ton of data. would the user need to download all of it? Maybe something like mongodb atlas or firebase would work.
Count me in as someone who would really appreciate this. If someone wants to take the lead, I could help out.
@b3u @mikecg I'll try to summarize this conversation this it's been going on a while. Sefaria doesn't see this project as a high enough priority to devote time to. That being said, we are happy if other developers want to take on this project and we are here to answer specific questions they have.
The easiest way to accomplish this project seems to be using React Native for Windows https://github.com/Microsoft/react-native-windows. The reason I say this is that we already have a mobile app written in React Native which has offline capabilities (https://github.com/Sefaria/Sefaria-iOS)
So the project would be to clone the Sefaria-iOS repo and modify it to work with React Native Windows.
@nsantacruz Thanks. I would rather go with Electron for Unix support, but that repo should be a great starting place. Would this project be under the sefaria org, or would I work on it under my personal acct?
@b3u
I would rather go with Electron for Unix support
Fine with me. I know nothing about Electron so won't be able to give specific advice. I have no idea if React Native is compatible with Electron since Electron seems to be aimed at web apps. You may want to base your app off of this repo (https://github.com/Sefaria/Sefaria-Project) which is the code for our web app. That may lead to other issues since our web app depends on a running instance of MongoDB
Would this project be under the sefaria org, or would I work on it under my personal acct
Your personal account for now
@mikecg Where do you propose we discuss this matter further? Edit: I made a gitter chat room at https://gitter.im/sefaria-desktop . Feel free to join.
@nsantacruz For the design - should I make it up as I go along, or make a proposal, etc.? Edit: I see from the ios app that a design system exists. Maybe I should access that?
@b3u This is your project from beginning to end. We're here to provide technical help regarding our software.
You can access all of our CSS styles from this repo (or simply by inspecting our site from your browser).
https://neutralino.js.org/ is a lighter-weight alternative to Electron - just FYI.
Great to see some interest/movement here.
Hey I would like to re-open this thread, what about simply packaging with something like electron ? I would really like taking this to desktop. @nassan , @yehosef @jeremieberrebi who's in ? where is it standing today ?
can anyone give a rough estimation of how much data this represents (maybe a couple GB, lets not forget its all text)
If we manage to package this in an offline App, many Kollelman could use it with Kosher Phones
@YonathanMeguira it's about 3-4GB with just the texts and links. That number comes from adding up the Mongo collection sizes. Sefaria already has an offline app for phones on both iOS and Android. You need internet to download the texts initially though.
@nsantacruz - I'm assuming that's for all the texts and versions of the texts, which is probably not needed.
If I'm not mistaken, the Sefaria app downloads the text into an SQLite database. Instead, you could bundle that (especially a trimmed-down version) into the app. You could also have a hybrid in which the app starts with a preloaded SQL database of the most needed and used texts and the user/app could download the rest if desired and internet-capable. I would guestimate that you could get the 95% use case with less than 200MB of data. (find . -type f -name 'merged.json' -exec du -ch {} + in the json dir of the export repo gives around 460MB.)
I'm still primarily interested in a desktop app - I often want to learn in a Beis Midrash where I don't have internet access. I don't mind downloading it when I'm online if that were an option.
Same for me. Interested in a desktop app. I think it needs to be built using https://www.electronjs.org/
The Android and iOS apps already have offline access
For the moment, I'm using the Android App with offline access on my Mac using BlueStacks but it's really not ideal
- Jeremie Berrebi
ᐧ
On Tue, Jun 8, 2021 at 2:22 PM yehosef @.***> wrote:
@nsantacruz https://github.com/nsantacruz - I'm assuming that's for all the texts and versions of the texts, which is probably not needed.
If I'm not mistaken, the Sefaria app downloads the text into an SQLite database. Instead, you could bundle that (especially a trimmed-down version) into the app. You could also have a hybrid in which the app starts with a preloaded SQL database of the most needed and used texts and the user/app could download the rest if desired and internet-capable. I would guestimate that you could get the 95% use case with less than 200MB of data. (find . -type f -name 'merged.json' -exec du -ch {} + in the json dir of the export repo gives around 460MB.)
I'm still primarily interested in a desktop app - I often want to learn in a Beis Midrash where I don't have internet access. I don't mind downloading it when I'm online if that were an option.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sefaria/Sefaria-Project/issues/294#issuecomment-856679444, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAFR7DJZH4IQCVCOXC22XTTRX4QXANCNFSM4C7IM3IQ .
@yehosef the current Android and iOS apps use JSON files to store the text and links that come out to about 900MB. There's duplication there to make the app run at a reasonable speed. People are welcome to try to optimize the size further, but I feel that's not that important once you're talking about a desktop app where there should be ample space.
Reposting this to remind people what Sefaria thinks is the way to go here:
@b3u @mikecg I'll try to summarize this conversation this it's been going on a while. Sefaria doesn't see this project as a high enough priority to devote time to. That being said, we are happy if other developers want to take on this project and we are here to answer specific questions they have.
The easiest way to accomplish this project seems to be using React Native for Windows https://github.com/Microsoft/react-native-windows. The reason I say this is that we already have a mobile app written in React Native which has offline capabilities (https://github.com/Sefaria/Sefaria-iOS)
So the project would be to clone the Sefaria-iOS repo and modify it to work with React Native Windows.
@jeremieberrebi there is now a MAC version available.
Btw I saw what you wrote on chatgpt on mac. I got it on whatsapp. In case you would like to try it. 058 628 8454
Indeed. I'm using the iOS Edition on Mac already for a few months. Thanks
About ChatGPT on Whatsapp, I'm not using Whatsapp. Thanks :)
- Jeremie Berrebi
On Sun, Mar 26, 2023 at 12:27 AM, Yonathan Meguira < @.*** > wrote:
@ jeremieberrebi ( https://github.com/jeremieberrebi ) there is now a MAC version available.
Btw I saw what you wrote on chatgpt on mac. I got it on whatsapp. In case you would like to try it. 058 628 8454
— Reply to this email directly, view it on GitHub ( https://github.com/Sefaria/Sefaria-Project/issues/294#issuecomment-1483925881 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/ABAFR7ALQIU6RHCBG5FTH23W55PKTANCNFSM4C7IM3IQ ). You are receiving this because you were mentioned. Message ID: <Sefaria/Sefaria-Project/issues/294/1483925881 @ github. com>