Async loading

[vermiculus]

Migrated from #36 -- this text copied from @Silex's latest response:

I don't have a gitter account

No such thing; it uses GitHub. To each their own though; I'm fine like this.

Well, went there and it said "Sign in to start talking". I guess I can sign in with my github account, but I didn't quite see the advantage compared to discussing it here.

but I imagine something like magit-status-hook starts an async task that when done finds the magit-status buffer and adds the relevant section.

Harder than you might think – I'd have to hook into magit-section's internal data structure to make navigation work right…

Well, you know better than me but from a quick look at the code, it looks like you could change (add-hook 'magit-status-sections-hook #'magithub-issue--insert-section t) into something like (add-hook 'magit-status-sections-hook #'magithub-issue--insert-section-only-if-data-are-ready t) and have the async task refresh the magit status buffer when it is done.

Of course, refreshing the magit status buffer is maybe not desirable, because maybe it messes with what the user started doing while magithub was refreshing. I don't know how well magit preserves scrolling & sections, etc.

That or you always display the section with a text like "refreshing..." and when the async task is done you refresh the status buffer.

…but that could be a workable idea. We could use that 'refreshing…' text with a special marker for the sentinel to find it again. We still run into the problem of interfacing into magit-section, but at least we're already in our own section at that point and can't mess anything up

It also makes more sense interface wise, because we know we can expect something to appear there. Again, I'm not familiar enough with magit-section to really see how difficult it is... I just assume each section is refreshable independently, if you want better advice you should ask tarsius he's usually very helpful with guidance on how to do achieve certain things.

---

Both options are workable, but the first proposal does seem easier.

[rgrinberg]

How about something like only enabling manual refreshes until this is done? The random pauses when opening a magit buffer are quite annoying.

[humitos]

I start using magithub today and this was the first thing that I found was annoying.

I don't have experience on elisp, but I wrote a very hacky and stupid workaround for this:

diff --git a/magithub.el b/magithub.el
index cc1ba14..32776a4 100644
--- a/magithub.el
+++ b/magithub.el
@@ -57,6 +57,13 @@
 (require 'magithub-issue-post)
 (require 'magithub-issue-tricks)
 
+(defun magithub-force-refresh (&optional force)
+  "Refresh all GitHub data.  With a prefix argument, invalidate cache."
+  (interactive "P")
+  (setq magithub-cache 'expire)
+  (magit-refresh)
+  (setq magithub-cache t))
+
 (magit-define-popup magithub-dispatch-popup
   "Popup console for dispatching other Magithub popups."
   'magithub-commands
@@ -70,7 +77,7 @@
              (?O "Toggle online/offline" magithub-toggle-offline)
              "Meta"
              (?` "Toggle Magithub-Status integration" magithub-enabled-toggle)
-             (?g "Refresh all GitHub data" magithub-refresh)
+             (?g "Refresh all GitHub data" magithub-force-refresh)
              (?& "Request a feature or report a bug" magithub--meta-new-issue)
              (?h "Ask for help on Gitter" magithub--meta-help)))

and I disable by default the magithub-cache in my init.el with this:

;; go to OFFLINE mode
(setq magithub-cache t)

So, how this works? The idea is simple, at startup just go to OFFLINE mode for magithub. Then, when we open magit-status nothing will be retrieved from github. If we want to force it, we just hit H g from the magit status windows. The hacky function will be called, and this one will go to ONLINE mode, retrieve data from github, and go to OFFLINE mode again.

I would like to have a similar behavior without touching the code. Besides, instead of hitting H g from magit status I would like to just hit g. So,

1. calling magit-status will just work locally
2. hitting g at this point will
   • retrieve data from github if cache is disabled
   • check the cache expiration time first, and if it's expired will retrieve data

Next time I call magit-status magithub won't retrieve data from github even if the cache is expired and the previous flow will be considered.

If you guide me through the code (and point me something to read) I could help to develop this.

[tarsius]

@vermiculus I prefer to have such discussions on github instead of gitter.

From gitter:

Sean Allred @vermiculus 15:30 @tarsius what's the feasibility of, when building the magit-status buffer, putting a marker at a section to insert it asynchronously? I'd like to start thinking about ways to tackle vermiculus/magithub#37. My thought is if I can put markers where the github sections should be, I can come back to them later after making the API calls asynchronously.  Jonas Bernoulli @tarsius 18:29 @vermiculus I plan to support "asynchronous sections" out of the box. This is mentioned in #2982. And actually Magit is already updating some sections asynchronously: the sections in the process buffer. But there is a bug: sometimes a new sections becomes a child of another, while they actually should all we siblings. It's a heisenbug. Because of that and because nobody ever complained about it, I never got around to fixing it. It's fairly simple to do this actually, just make sure you do/don't inherit (as appropriate) properties at the position you are inserting a new section. The code that inserts text into the process buffer works by first inserting a stub section with no section content, and later inserting the text before a final empty line that was inserted on initial setup to prevent accidentally inheriting properties. But as I said, there's a bug somewhere, that newline doesn't seem to do the trick. On the other hand Magit also supports delaying the insertion of section bodies until a section is expanded (though that currently requires a bit too much boilerplate, and the section bodies are in that case still inserted synchronously - just later) and that does not have this bug. Related to that is #2768. So yes, you can do this now, and in the future it will be well supported even (I consider this one of the most important changes to be made), but if you want to do it right now, then you might have to investigate and fix the mentioned bug yourself.

Anyway... I recently made submodules faster, among other things using this commit (view in Magit for a meaningful diff). Doing the equivalent thing in your code should help make your code faster too (by not spending time on generating and/or formatting hidden information until it is not hidden anymore).