Overdr0ne
FYI: dogears.el
Hi there,
FYI, I recently uploaded this package, and I added Gumshoe to its readme, like you mentioned some alternatives in yours: https://github.com/alphapapa/dogears.el I will probably mention some more that you have mentioned, too. :)
BTW, if I may offer a little feedback, too: In trying to understand how Gumshoe works, I don't understand what the readme means by "scope." Obviously it doesn't mean the same as Lisp binding scopes, but it doesn't seem to explain exactly what it means, so I'm not sure what that feature is or how I would use it.
And I still love the photo at the top of the Gumshoe readme! :) I feel like I've seen the movie that's from, but I'm not sure--do you know which it is?
Thanks for pointing me to this. I'm not near my computer for another day or two, but I'll take a closer look when I am to integrate or differentiate things. And I'll definitely add it to the readme
BTW, if I may offer a little feedback, too: In trying to understand how Gumshoe works, I don't understand what the readme means by "scope." Obviously it doesn't mean the same as Lisp binding scopes, but it doesn't seem to explain exactly what it means, so I'm not sure what that feature is or how I would use it.
I am referring really to the 'scope' of each gumshoe log, using the 'make-variable-*-local' interface. So I have one global, one buffer-local, and one perspective-local gumshoe right now. But I can imagine extending it to have a window-local, frame-local or something, and as long as they use that interface for isolating their respective environments, it should be trivial to add another corresponding gumshoe. I guess you could have a gumshoe locally bound for standard lexical/dynamic scope if someone added that interface, but those contexts just aren't relevant to the user. Maybe it's an abuse of the term, but emacs seems to club them together in the manual, I dunno, I'm definitely open to suggestions...
Hm, I think I understand better what you mean. "Scope" seems like a reasonable term. I guess if you wanted to run with the theme, you could call them "beats" or "cases," but that might be too much. :)
It might be helpful to have a list view, sort of like dogears-list, that would show all of the places in the gumshoes, and show the scope of each one in a column. Then the user could see all of the places and sort them by scope. That would probably make for a good screenshot too, which could help users understand how the package works.
I like the idea of both packages, gumshoe and dogears. @alphapapa Maybe it makes sense to mention in the readme that the main technical difference is the use of markers vs bookmarks. Another difference seems to be how the "scope" of a tracked location is determined, where gumshoe seems to use an elaborate algorithm. I've seen you write about bookmarks, but it is not mentioned in the comparison. Furthermore can you elaborate on the advantages of bookmarks, since you don't make use of the most obvious advantage - bookmarks can be persisted!
I like the idea of both packages, gumshoe and dogears.
Yeah, I think both approaches can be useful.
@alphapapa Maybe it makes sense to mention in the readme that the main technical difference is the use of markers vs bookmarks.
Ah, I even didn't know that Gumshoe uses markers.
Another difference seems to be how the "scope" of a tracked location is determined, where gumshoe seems to use an elaborate algorithm.
Yes, while Dogears uses a fairly basic "relevance" categorization to determine a place's relevance to the place where the user currently is.
Furthermore can you elaborate on the advantages of bookmarks, since you don't make use of the most obvious advantage - bookmarks can be persisted!
Dogears isn't meant to replace Emacs's bookmarks system. The fact that it uses bookmark records internally is an implementation detail. The benefit of using them is that bookmark records can be used to jump to places in buffers that no longer exist, e.g. if a file's buffer has been killed. The bookmarks system is effectively used as an API to remember places and go to them. Rather than writing per-mode, per-buffer, per-file code, we get that functionality for free in any mode that supports the bookmark system, which is many of them. And we also benefit from packages like org-bookmark-heading, which improve the way the bookmark system works in Org buffers.
Does that answer your question?
Thanks. ~It doesn't answer my question entirely. But I try to answer it myself~ by using bookmarks (even if you don't persist the entire list), you still profit from the ability of bookmarks to restore buffers which have been killed. This is a crucial advantage of bookmarks. Otherwise markers as in gumshoe are the more lightweight approach, but gumshoe cannot restore closed buffers. Maybe mention this in the readmes?
(Edit: ah you already wrote that, but I read over it on my mobile. Sorry.)
I guess if you wanted to run with the theme, you could call them "beats" or "cases," but that might be too much.
Beats, love it. But I'll probably have to at least parenthisize scope.
And I was just about to come back after reading some of your code to comment on bookmarks. It seems like a much better data structure here, dunno why I didn't think of that. I've been doing the buffer switching explicitly lol, and purging marks in killed buffers. Plus persistence, yeah.
@Overdr0ne Thinking about it again, I also believe that bookmarks are the more powerful base abstraction here. Probably you went with markers because evil markers and the built-in Emacs mark rings work like this? Maybe you @Overdr0ne and @alphapapa could join forces on these packages? It seems that maybe the distance heuristic you are using in gumshoe is more sophisticated? Currently I am not using such a history package but I would like to, so I will keep an eye on both of your projects. I would like it very much if you keep the packages focused, narrow in scope and without additional dependencies, but this is of course only my personal preference :)
Yeah, when I realized that the bookmark functions could be used outside of actual Emacs bookmarks commands, and that that whole mode-specific infrastructure was available to other tools, it was an eye-opening moment. Even though I wrote org-bookmark-heading years ago, it didn't hit me until recently. It works great in burly, so it seemed natural to use it in dogears.
It seems that maybe the distance heuristic you are using in gumshoe is more sophisticated?
It sounds like it, but I haven't looked at the code. The way dogears-remember works is to construct a new bookmark record and then add it to dogears-list with cl-pushnew and a custom test function (which ends up comparing buffer positions if other parts of the two records are equal). I'm trying to avoid causing any performance problems when that code is run, so I limit the list to 100 entries by default, and it seems to work okay. I wouldn't be opposed to a smarter buffer-position-comparing test if it is fast enough, and if I can understand it. :)
Maybe you @Overdr0ne and @alphapapa could join forces on these packages?
Maybe. I think both approaches can be useful on their own, though, and maybe we could cross-pollinate some ideas, too. :)
I would like it very much if you keep the packages focused, narrow in scope and without additional dependencies, but this is of course only my personal preference :)
That's generally my intention, yes. I'd like it to be suitable for ELPA or NonGNU ELPA eventually, and I want to keep it simple and fast. Big, all-encompassing packages can be useful (see Emacs and Org, haha), but they're a lot harder to understand and maintain.
It sounds like it, but I haven't looked at the code. The way dogears-remember works is to construct a new bookmark record and then add it to dogears-list with cl-pushnew and a custom test function (which ends up comparing buffer positions if other parts of the two records are equal). I'm trying to avoid causing any performance problems when that code is run, so I limit the list to 100 entries by default, and it seems to work okay. I wouldn't be opposed to a smarter buffer-position-comparing test if it is fast enough, and if I can understand it. :)
The limit seems a bit too low (for my taste). Maybe you can use a more sophisticated tree datastructure to quickly lookup if an entry falls into an existing range. I certainly do appreciate the focus on performance. Unfortunately Emacs quickly has the tendency to get slow, I really like to avoid this. Just recently I used which-function-mode in combination with some custom imenu regexps and file opening got significantly slower. I've seen you use imenu to get some context information. This is a nice idea, I mean the idea to get context in general, but I am a bit worried about the performance of imenu.
Another difference seems to be how the "scope" of a tracked location is determined, where gumshoe seems to use an elaborate algorithm.
Maybe I should clarify, at any given time you can access any of currently 3 different logs, basically so you can just search for marks set within a buffer, or within a perspective, or whatever 'scope' independently. They are tracked independently in different logs.
Another way to get that result though might be to tag each location with the perspective, buffer, window, or whatever your scope is, then narrow from there with completion. You could then have commands with initial input for the current buffer, current perspective etc. Honestly, that's a more flexible way. Also more space efficient. We'd have to see if it's fast enough with lots of marks. With gumshoe they're basically presorted or presearched. Hmm...
The limit seems a bit too low (for my taste). Maybe you can use a more sophisticated tree datastructure to quickly lookup if an entry falls into an existing range. I certainly do appreciate the focus on performance. Unfortunately Emacs quickly has the tendency to get slow, I really like to avoid this. Just recently I used which-function-mode in combination with some custom imenu regexps and file opening got significantly slower. I've seen you use imenu to get some context information. This is a nice idea, I mean the idea to get context in general, but I am a bit worried about the performance of imenu.
You can increase the limit, and maybe it will work fine up to a much higher limit. I haven't tried, I just guessed that 100 would be more than enough to be useful and low enough to perform adequately. If you'd like to experiment and let me know what you find, that would be helpful.
Anyway, I don't think a more sophisticated data structure is needed. The buffer-position test is simply (<= (abs (- a-position b-position)) dogears-position-delta), and you can't get much faster than that.
Just recently I used which-function-mode in combination with some custom imenu regexps and file opening got significantly slower. I've seen you use imenu to get some context information. This is a nice idea, I mean the idea to get context in general, but I am a bit worried about the performance of imenu.
Yes, which-function and imenu seem like useful abstractions, but they aren't necessarily intended to be used for these purposes. You can see that the use of which-menu in dogears is optional, and you can plug in your own alternative function, or choose dogears's own, very simple fallback function, which should perform fine in nearly any circumstance.
Another way to get that result though might be to tag each location with the perspective, buffer, window, or whatever your scope is, then narrow from there with completion. You could then have commands with initial input for the current buffer, current perspective etc. Honestly, that's a more flexible way. Also more space efficient. We'd have to see if it's fast enough with lots of marks. With gumshoe they're basically presorted or presearched. Hmm...
Yeah, I've come to appreciate even more the power of simple, plain-text regexp matching for filtering. At least, as long as it performs well, just "format it all into lines of text and let Emacs filter with regexps from user input" can fulfill most needs, rather than writing purpose-specific filtering commands. Sometimes it's even more useful to not "silo" data from different sources and require the user to remember separate commands for them.
Alright, I have my computer again. My plan here is to basically try to get my gumshoe workflow with dogears. If I can get something equivalent or better, then I'll just paste my dogears config at the top of gumshoe and point my users to dogears. I'll see about adding some dogear features to gumshoe as well though. I feel like I should withdraw or somehow pause my MELPA pull until this is nailed down though. Not sure exactly how to handle that...
Edit: Gumshoe's actually already been merged I just realized. I'll just improve it as best I can. It would be nice to have this basic feature canonicalized in some form in Elpa like y'all were saying. I think there's still some room for differentiation between the packages, so I'll keep experimenting.
That's very kind and humble of you, to offer to defer to my package, but I don't think that's necessary. Gumshoe is a cool package and I think both approaches are worth pursuing. Maybe someday they will converge more, as we explore these problems and solutions. :)
That's very kind and humble of you, to offer to defer to my package, but I don't think that's necessary. Gumshoe is a cool package and I think both approaches are worth pursuing. Maybe someday they will converge more, as we explore these problems and solutions. :)
Hey, thank you for that. And I haven't forgotten about this. In fact I've done a fair bit of experimenting, it has just been a bit more complicated than I was expecting, and I took some wrong turns. But things seem to be coming together and I hope to have some interesting things to show you guys soon.
Hello again. I just noticed I hadn't yet referenced dogears in my related packages. It's in there now, my bad. Here's the new version I've been working on. https://github.com/Overdr0ne/gumshoe/tree/peruse
The main new features here are:
- Gumshoe now uses 'gumshoe--entries' in the backlog instead of marks or bookmarks. I tried to make bookmarks work for me, but found the bookmark-alist far too ingrained into their interface and ended up finding it easier and more extensible to just make my own abstraction. It contains basically all the same metadata as a bookmark, but users/developers may add whatever metadata they want, by inheriting from it. That's how I added the perspective field.
- peruse: this is my take on
dogears-list. It uses completing read to browse through the backlog. The display is customizable, allowing users to specify what and in what order gumshoe--entry fields are selected. - Noticing how similar backtracking is to isearch, I upgraded backtracking to also display 'footprints' which visually indicate mark positions while backtracking.
- Both backtracking and perusing be filtered programmatically by passing in a predicate function.
And after going through all that to unify the separate tracking, I can't help but notice some benefits of the old way of doing things, like, the length of logs in a scope can be kept constant and predictable, so you won't unintentionally wrap. Like, if I'm in a single perspective for a while, I will pretty much inevitably overwrite the backlog entries for another perspective, unless I make the max backlog length really long, but that might cause other problems with filtering latency and footprint clutter. But overall I like this new method and will probably end up merging it.
Hello again. I just noticed I hadn't yet referenced dogears in my related packages. It's in there now, my bad.
I hadn't noticed, but that's kind of you, thanks.
The changes sound interesting, though I confess I still don't understand all of it. :)
One note: you mentioned having tried to use the bookmark library and then decided against it due to bookmark-alist. Maybe you already know this, but in case you don't: you don't need to use bookmark-alist in order to benefit from the bookmark library. LIke dogears does, you can use the bookmark library functions to make and jump to bookmark records, while storing the records in your own list. This lets you benefit from mode-specific bookmark support, etc.
The changes sound interesting, though I confess I still don't understand all of it. :)
Uh oh, if you don't understand, then I don't think anyone will haha. Let me try again. The main change here is I started thinking of the Gumshoe log as a database. Perusing and backtracking are just different ways of searching that database: perusing just uses completing-read, and backtracking just moves sequentially backward in the log, sort of like an isearch. So the scopes I was using before are just different searches on that database. This allows me to accommodate any arbitrary context, or even combinations of contexts, so it's much more future-proof.
LIke dogears does, you can use the bookmark library functions to make and jump to bookmark records, while storing the records in your own list. This lets you benefit from mode-specific bookmark support, etc.
And yeah, I started with bookmarks, but found the API confusing, with lots of special cases. I really wanted to add my own context metadata, like perspective, window, phase of the moon, whatever, and allow my users to do the same. Classes just seem naturally suited to that kind of extension through inheritance(until we get traits or something better...), and I like that they allow individual fields to be documented. I wasn't aware of mode-specific bookmarks though. Is that documented somewhere?
The changes sound interesting, though I confess I still don't understand all of it. :)
Uh oh, if you don't understand, then I don't think anyone will haha. Let me try again. The main change here is I started thinking of the Gumshoe log as a database. Perusing and backtracking are just different ways of searching that database: perusing just uses completing-read, and backtracking just moves sequentially backward in the log, sort of like an isearch. So the scopes I was using before are just different searches on that database. This allows me to accommodate any arbitrary context, or even combinations of contexts, so it's much more future-proof.
I don't understand how that's different than it was before, or how/if it differs from dogears-list, but okay. :)
LIke dogears does, you can use the bookmark library functions to make and jump to bookmark records, while storing the records in your own list. This lets you benefit from mode-specific bookmark support, etc.
And yeah, I started with bookmarks, but found the API confusing, with lots of special cases. I really wanted to add my own context metadata, like perspective, window, phase of the moon, whatever, and allow my users to do the same. Classes just seem naturally suited to that kind of extension through inheritance(until we get traits or something better...), and I like that they allow individual fields to be documented
You can add any keys/fields you like to a bookmark record, because a bookmark record is just an alist. That's basically the ultimate in flexibility, because an alist can have any number of keys, and applications simply ignore ones they don't care about. It doesn't require defining classes, subclasses, etc.
. I wasn't aware of mode-specific bookmarks though. Is that documented somewhere?
See bookmark-make-record-function. Major modes can set that function buffer-locally, and bookmark records carry their handler function so they know how to activate themselves. I use bookmark records this way in burly and dogears, and I extend it for Org buffers in org-bookmark-heading, and it works well.
I don't understand how that's different than it was before, or how/if it differs from dogears-list, but okay. :)
So before I was just using simple marks. They don't contain any context information besides buffers. I was getting the perspective context by storing those marks as variables local to each perspective. I'd have had to do something similar if I wanted to add window local logging, or whatever. But yeah, it's totally the same idea as dogears-list, just a different implementation that doesn't use bookmarks. In particular, the interface allows for programmatic searches, which I needed to make backtracking work correctly. So I made my own, to make sure I had the control I needed to maintain backward compatibility.
You can add any keys/fields you like to a bookmark record, because a bookmark record is just an alist. That's basically the ultimate in flexibility, because an alist can have any number of keys, and applications simply ignore ones they don't care about.
That's a very good point. And I guess documentation for fields could all go in the doc string for the container. Hmm. Other than the ability to document fields individually, I use the :printer tag to specify how to print each field. I suppose I could just transform the alist for the same effect. I grew to like the ergonomics of classes, like oref, oset, and with-slots in particular are pretty handy and kept my code compact and readable I thought. The fact that you have to inherit to add fields definitely makes it less flexible, and I don't really like that. But I think the class does make the specification more clear, to humans, and to the interpreter when things break. I feel like I had trouble understanding bookmarks because the spec for what needed to be inside was spread out between the doc strings of the functions that used them, which lacks semantic highlighting. I guess classes were the tool I was somewhat familiar with, and that had stuff I needed built in, so that's what I reached for. I'll have to think about that...
See bookmark-make-record-function. Major modes can set that function buffer-locally, and bookmark records carry their handler function so they know how to activate themselves.
So you just use bookmark-make-record-default to get what the api needs, then add whatever other alist fields you might want on the fly I guess? I do like that, it's very flexible. Inheriting is kinda verbose, but it does let you specify clearly your particular configuration. cl-describe-type I think can be pretty handy when trying to understand how to extend something. But it does make things more rigid, but you can tune that rigidity with defclass tags like :type. Maybe those features could be added to alists, but then you might just end up with something very much like classes. It's hard for me to say which is absolutely better here, because alists are just so simple and elegant...
In particular, the interface allows for programmatic searches, which I needed to make backtracking work correctly.
Can you explain a little bit about what you mean there? I implemented forward/back navigation in Dogears, but it doesn't sound as complicated as that, so I'm not sure it's the same thing you mean.
Using EIEIO is certainly more powerful, and I guess if you plan to have more classes that inherit from a base one, that would make sense. Note that EIEIO isn't as efficient at runtime, because the byte-compiler can't specialize on struct slots--it has to inspect the type at runtime when you do oref or oset. So it's probably a bit slower than alists, too. But that's probably not an issue here; even Magit uses EIEIO, and it works with a lot more data than we are in these packages. I also used EIEIO in matrix-client, and it was fast enough.
So you just use bookmark-make-record-default to get what the api needs, then add whatever other alist fields you might want on the fly I guess?
Yes, using, e.g. (setf (alist-get .... I only add a few fields, and I don't need inheritance, because fields aren't added beyond the ones in the code.
cl-describe-type I think can be pretty handy when trying to understand how to extend something.
Thanks for reminding me about that. It is nice.
Can you explain a little bit about what you mean there? I implemented forward/back navigation in Dogears, but it doesn't sound as complicated as that, so I'm not sure it's the same thing you mean.
For example, when I do a backtrack local to my perspective, I pass gumshoe--in-current-buffer-p to gumshoe--backtrack called by gumshoe--backtrack-back/forward, and it filters(searches) for only entries that match the current perspective. Then subsequent gumshoe--backtrack-back/forward commands just use that filtered list until a different command is called. That was how those commands worked before, so I wanted to make sure that interface didn't change. gumshoe--peruse can be similiarly pre-filtered, though I could have just added initial-input to the minibuffer in that case for the same effect.
And as far as extending things, I planned on providing different ways to jump to entries. Like, whether or not to switch to a different perspective when jumping is a policy decision I'd like to leave to users. And I imagine similar nuances might be desired for things like windows or frames.