Offline CRUD and sync to server

[johnpitcairn]

I'm tasked with developing an app for use by remote workers that will wind up running without a data connection some of the time. I'm going to need to allow users to create, update and delete content while they are offline, then sync changes back to Drupal when they have a data connection.

Initial examination of drupalgap suggests there is no local entity data model, so I'd wind up building data models with backbone.js anyway. If so, I might as well go fully offline-first, forget about drupalgap, and handle sync to Drupal myself (possibly Drupal 8).

Or am I missing something in drupalgap that would help me?

[signalpoint]

@johnpitcairn Out of the box, DrupalGap does not have this feature. This has been requested a few times, and I know of one person who successfully implemented it. Private message me on d.o. and I can put you in touch with her.

Alternatively, I really would like to turn this into a contrib module for DrupalGap. Essentially, the module just needs to use a hook_form_alter() within the DG FAPI and attach a custom submit handler, that in turns check's for offline, then stores items in local storage, etc, etc.

I wouldn't mind assisting in writing this module, just let me know if you're interested. If you're going to DrupalCon Austin, we could sit down and sprint something together I think.

UPDATE: here is a rough overview of what needs to be implemented for this fix:

Issue Tasks

• [ ] Use hook_services_* to save entity field and instance to local storage after the initial system connect call
• [ ] When offline, all forms need a custom submit handler attached (probably pre pended or over write any existing submit handlers). This submit handler must then save the form state values into a local storage queue, along with the original submit handler. (Except user login and registration).
• [ ] When offline, all forms must check to see if any form state values have been queued up for synchronization, if they have been, place them back into the form
• [ ] When the device comes back online, it should run the form state queue and submit each form that was queued up
• [ ] We need same way to detect when a device goes on/off offline, maybe a background service is needed. At minimum, the jDrupal services API needs to check for a connection, and if it doesn't have one, the call should probably be queued up for later

[johnpitcairn]

Yes, I'm interested. I won't be at DrupalCon Austin (I live in New Zealand). I can start hacking something together. If the Drupal 8 effort is far enough along I'd also be very interested in working with that.

[signalpoint]

@johnpitcairn The D8 version still needs lots of love, it isn't functional at all at this point. I started an 8.x-1.x branch of the Drupal module, but didn't make it very far. Any help is much appreciated!

[johnpitcairn]

I think I'll back off on this (for now). A closer analysis indicates I really will need full-featured local entity data models and business logic, otherwise entity creation while offline will not work well (validation, permissions, relationships, etc). You can't simply queue an entity creation form, the user won't be able to work with the result until online again.

So my application needs to be offline-first, possibly with Drupal as the data store and admin-only back-end. Backbone.dualstorage here I come (I think).

[signalpoint]

Just to update everyone on this issue:

• it is the most requested feature
• it is very complex to implement
• it needs sponsors and/or contributors
• the successful implementation mentioned earlier never happened, it doesn't exist :P

www.drupalgap.org/contribute

Stop by #drupal-drupalgap on IRC if you're interested in implementing this solution, I have many thoughts rolling around in my head, so it'd be nice to chat with someone about what it takes to accomplish this.

Interesting read: http://coenraets.org/blog/2012/05/simple-offline-data-synchronization-for-mobile-web-and-phonegap-applications/

[ssoulless]

I would like to help to accomplish this. let me know

[signalpoint]

First we need to decide what storage mechanism(s) will be used: http://docs.phonegap.com/en/3.3.0/cordova_storage_storage.md.html#Storage

Right now DrupalGap uses local storage for everything, and has no api for any of the other mechanisms.

I guess for simplicity, going the local storage route may be appropriate. I don't know enough about all the mechanisms to weight the pros/cons of each.

[ssoulless]

in http://coenraets.org/blog/2012/05/simple-offline-data-synchronization-for-mobile-web-and-phonegap-applications/

it only supports unidirectional sync, but in the example there you cannot make changes in the client and update them to the server, that is the feature we need =/

[ssoulless]

I began to do some experiments with the code in https://github.com/ccoenraets/offline-sync

maybe there we can have some conclutions... good start point

[ssoulless]

Here http://docs.phonegap.com/en/3.3.0/cordova_file_file.md.html#File describes everything how to use local file system, is drupalgap using local file system?

and Im agreed with you, going the local storage route is appropriate, I gues drupal gap uses WebSQL, does not it?

[signalpoint]

@ssoulless Yes, DrupalGap uses Local Storage. DrupalGap does NOT use WebSQL or any other method.

UPDATE: I've made a list of the tasks that need to be implemented to get this feature working: https://github.com/signalpoint/DrupalGap/issues/258#issuecomment-43841685

[ssoulless]

Ok are we gonna manage this through a separate drupal module? or are we gonna work on the drupalgap core?

[signalpoint]

@ssoulless DrupalGap core will be used for this feature.

[samwilson]

Is there any news on this feature? I'd be interested in helping, if I can.

[signalpoint]

@samwilson Anyone interested in helping on this issue, please stop by #drupal-drupalgap on IRC to chat about it, thanks!

[unmeshsheth]

I am very interested in this feature as this is a single most requested feature in my line of work. Happy to see lot of work. However, I am confused with above thread! Is someone still working on it or is this issue dead?

[johnpitcairn]

I am not working on it.

On 27 February 2015 at 11:37, Unmesh Sheth [email protected] wrote:

I am very interested in this feature as this is a single most requested feature in my line of work. Happy to see lot of work. However, I am confused with above thread! Is someone still working on it or is this issue dead?

— Reply to this email directly or view it on GitHub https://github.com/signalpoint/DrupalGap/issues/258#issuecomment-76290244 .

John Pitcairn Opus Locus Auckland, New Zealand Tel: +64 9 815 5984 Mob: +64 27 474 8636

[hounded]

Where is this up to ?

I'm trying to build a phonegap app that can save form data when there is no connectivity and then create new nodes when connectivity is achieved.

I was looking at angularjs and localforage to do this, but as the ultimate goal is to integrate with a drupal website this looks like the logical answer.

What I have described, is this achievable with offline forms?

[signalpoint]

@metax45 There has been no progress on getting this into DrupalGap core. However, there has been much recent progress with a contributed DrupalGap module, called Local Forms: http://drupalgap.org/project/local_forms - I am currently building this for a project, so it will see much improvement over the next few weeks.

[chrisschaub]

This is cool stuff ...

http://aws.amazon.com/cognito/

and there are js/nodejs libraries to work with it.

• Chris

On Wed, Mar 18, 2015 at 4:06 PM, Tyler Frankenstein < [email protected]> wrote:

@metax45 https://github.com/metax45 There has been no progress on getting this into DrupalGap core. However, there has been much recent progress with a contributed DrupalGap module, called Local Forms: http://drupalgap.org/project/local_forms - I am currently building this for a project, so it will see much improvement over the next few weeks.

— Reply to this email directly or view it on GitHub https://github.com/signalpoint/DrupalGap/issues/258#issuecomment-83176921 .

Christopher Schaub http://chris.schaub.com

[joeyshidab]

@signalpoint how is the progress with this?

[signalpoint]

@joeyshidab All progress has, and will be, reported here. Folks, I understand this issue is in high demand, we need someone to step forth and help establish the most feasible plan to implement this solution (or throw money at the probelm). All parties interested, come chat about it: www.drupalgap.org/irc

EDITED

[eduoda]

Hi, I'm studying the local_forms and I think it's too generic, it tries to deal with all sort of form which leads to several issues that I'm not sure we will find a simple way to deal with, e.g, page redirects.

Maybe we should restrict the local_forms to requests that are made to a Drupal services endpoint. I made a fork, so I can test this idea.

If I'm right, then probably this feature should be part of jDrupal and not DrupalGap.

Oda

[signalpoint]

@eduoda Thanks for chiming in. Yes, the local_forms module is definitely generic, and will not be able to handle all situations.

That's an interesting idea that this offline/sync caching layer should/could be up in jDrupal, very interesting indeed.

[eduoda]

Please, check it out: https://github.com/eduoda/local_forms I also had to make to change jDrupal a little bit: https://github.com/eduoda/jDrupal

I think that this kind of offline behavior can not be transparent to other contrib modules. They should use hooks or callback to interact with local_forms for deal with redirects, reset forms, etc.

[eduoda]

btw, I think that the way the drupalgap.online is updated in my version of local_forms is a good idea. I'm considering to write a module that automatically gets you out of that "failed connection" page when you get connected.

[signalpoint]

@eduoda This is some great stuff you've implemented. Have you tried running the tests.html file to see if all the unit tests still pass in jDrupal with these changes. I now definitely see the advantage to your approach, so that custom forms can implement their own handlers, and we can just focus on the core forms.

@rjsteinert, I just wanted to ping you in on this thread, and to hopefully hear your thoughts about a local_forms layer in jDrupal to help with online/offline CRUD syncs.

[eduoda]

Hi Frank! I'm glad to help you with the awesome job you have done!

I've just commit another version of jDrupal that has this feature in core. I think it's pretty simple and backward compatible: https://github.com/eduoda/jDrupal I also made a slight modification to webform: https://github.com/eduoda/webform

Sorry, I was not aware about the tests.html, I'll run the tests as soon as I can.

[eduoda]

It seems fine:

Tests completed in 43004 milliseconds. 60 assertions of 60 passed, 0 failed.

[eduoda]

Hi @signalpoint Did you review the changes I made to jDrupal? I'm planning to work on this issue next week and it would be nice to know if I'm in the right direction.