karthink
Media (and other) links.
Karthik, question/suggestion about links.
They are VERY convenient. Often more convenient than using context.
I understand that in chat buffers, one can include links to media files:
https://github.com/karthink/gptel?tab=readme-ov-file#including-media-images-documents-with-requests
-
Why is this limited to chat buffers only?
I understand I can use the context to include media files in other buffer types but why not also allow links in buffer of other types, for example org or markdown?
-
Why is this limited to media files types?
If a link is to a file that has a mime type that is not media known to the model, but it is text, why not include the contents of the file in the prompt at that point? That is, if it's a link to a txt, org, md or other text file type, just include it in the prompt.
-
Why do links need to be standalone?
In org mode buffers and chat buffers, why not support links however they appear, not just when they are standalone. Likewise, if a link is internal, to another heading in the same or a different file, why not allow the body text in that heading be included in the prompt. It would for example, make it very easy to include "boilerplate" or other context by simply including links in the prompt text.
[2024-11-23 Sat 23:04]
Question 1: Because users can have lots of links in regular Org buffers, which are almost always intended for purposes other than sending data to the LLM.
Question 3: To signal intent more clearly, and to make the links whose data will be included stand out and easier to spot. Making it a "standalone" link is used as a proxy for the @file: syntax that some other LLM clients use.
I can make both of these customizable, but I think it's easy to forget that you've included something in a link that you didn't mean to send, especially in a non-chat buffer that's being used for something else primarily. If you have more thoughts on this I'd be curious to hear them.
Question 2: I like the idea of using links to set up per-buffer context lists, as a solution to #475. It's also more transparent than an internally maintained context list that is only accessible via running a command/opening up the context buffer.
The reason this is limited to media file types right now is just mental compartmentalization -- I added media support and @daedsidog added context support at two different times, and it didn't occur to me to combine the two. I'll work on adding this feature.
It's still ill-defined because there are many ways to link to buffers in Org, including user-defined link types. It's also difficult to link to regions of buffers robustly. On the other hand Markdown does not have any built-in way to link to buffers, only files. So we'll have to think about how to do this carefully. If you can provide a spec to help nail down the feature it'll be easier to implement.
One thing I can do but wish to avoid is to define a new gptel-specific Org link-type.
See also #325.
One thing to be wary of is cases when you do not wish links to be treated as actual file links. For example, I often use an Org document as a reference for prose and content material, and would not want to have links translated into media. I don't se a way around that other than having a gptel-specific link type.
...On second thought, if the link translation is handled only in the chat buffer, it's fine.
For example, I often use an Org document as a reference for prose and content material, and would not want to have links translated into media
Yes, this is why link translation is currently limited to chat buffers.
Question 1: Because users can have lots of links in regular Org buffers, which are almost always intended for purposes other than sending data to the LLM.
Understood, but as an org user this seems unnecessarily limiting.
How about any links that appear in a region of text that is inside a drawer called :CONTEXT: for example, be sent to the LLM and no other links. That way one can manually construct context explicitly.
I like the idea of using Org/markdown links to add to the context, as discussed in https://github.com/karthink/gptel/issues/481, as the context set automatically becomes buffer-local, transparent and easily editable. But that has other issues, including that you can't have "live" buffer regions in the context.
What if links that are within a CONTEXT drawer, which have a label name, which is suffixed with two line numbers, separated by colon, which is used to define a region within a referenced context file. More or less as you must be doing already within context buffers. So something like this:
:CONTEXT:
[[id:a18cbfc7-231d-414f-8f0f-4b64f7dff504]]
[[file:/some/other/path]]
[[file:/path/to/file][my source code 124:581]]
[[id:a18cbfc7-231d-414f-8f0f-4b64f7dff504][some context 4:10]]
# etc
:END:
-
Any link that targets a headline in an org file means to include the body of that headline.
-
Any link that targets a file means to include the contents of that file.
-
Any link that targets a file where there is a colon separated number pair in the link description means to target that region of lines within the target file.
-
Likewise, regions can be targeted from any source, for example a headline identified by ID.
I can think of some corner cases here. For example, if the referenced region changes because the referenced file is modified, the limits in the link would need to be updated. As I imagine, you do now with context buffers.
Question 3: To signal intent more clearly, and to make the links whose data will be included stand out and easier to spot. Making it a "standalone" link is used as a proxy for the @file: syntax that some other LLM clients use.
Fair enough. However, if links appear inside a context draw for org mode users, then this should not be a requirement.
I can make both of these customizable, but I think it's easy to forget that you've included something in a link that you didn't mean to send, especially in a non-chat buffer that's being used for something else primarily. If you have more thoughts on this I'd be curious to hear them.
I prefer DWIM as far as possible and avoiding customizations where possible. I don't know about others and their use of gptel, but I tend to assume org mode or markdown syntax for all the text that I'm using, even in files that are not explicitly of that syntax. Org mode in particular offers powerful syntactic sugar to enable gptel to be integrated in ways I've suggested here.
Question 2: I like the idea of using links to set up per-buffer context lists, as a solution to #475. It's also more transparent than an internally maintained context list that is only accessible via running a command/opening up the context buffer.
I'm 100% with you on this one.
The reason this is limited to media file types right now is just mental compartmentalization -- I added media support and @daedsidog added context support at two different times, and it didn't occur to me to combine the two. I'll work on adding this feature.
Excellent. I think this will be a major workflow gain, at least for me :)
It's still ill-defined because there are many ways to link to buffers in Org, including user-defined link types. It's also difficult to link to regions of buffers robustly. On the other hand Markdown does not have any built-in way to link to buffers, only files. So we'll have to think about how to do this carefully.
Agreed. One needs to consider this carefully in the context of files that are not org mode.
If you can provide a spec to help nail down the feature it'll be easier to implement.
I think the solution I suggested above will work in markdown as well as in org mode. That is, you co-opt the label part of the link to have syntax that you interpret for gptel purposes if it appears in a :CONTEXT: drawer. The equivalent of the idea of a context draw in Markdown might be to use front matter?
One thing I can do but wish to avoid is to define a new gptel-specific Org link-type.
Agree 100%. You do not want to change anything org mode does with links. Including customised link types and all the rest. However, the link label is not touched by org mode and is therefore "open slather" for you with gptel, So long as it appears within a context drawer!
More ideas from as an org mode user: further to the :CONTEXT: drawer suggestion to isolate LLM links, perhaps when working in org mode with gptel-org-set-topic set, or gptel-org-branching-context enabled, any link, not just those within a :CONTEXT: drawer should be resolved to the LLM. In this case, links are likely to be explicitly intended for dereferencing. The gptel-track-media variable would still apply so one can turn this off. If not already saved, the status of track media should be included in the gptel-org-set-properties data set.
The more I think about it, I think the link metaphor to embed context in gptel will be a great addition. It's simple, explicit and easily managed. Especially in org mode because of the rich features set apropos links and what one can do with them. For example MCP servers https://github.com/karthink/gptel/issues/484 presenting a RESTful API on a URL fits right in. If if not on a URL end-point, perhaps one can shell out using shell: or ssh: (tramp) links in some creative way too. Food for thought.
Done.
Links to plain-text files in chat buffers are now followed, and their contents included in the request.
For example, suppose you have this text file named remap.yaml:
- JOB: intercept -g $DEVNODE | caps2esc -m1 | uinput -d $DEVNODE
DEVICE:
EVENTS:
EV_KEY: [KEY_CAPSLOCK, KEY_ESC]
Then in the chat buffer, you can have a regular Org link to it:
In this yaml file, I have some key-remapping configuration:
[[file:/path/to/remap.yaml]]
Could you explain what it does, and which program might be using it?
Then the query text sent will be the following:
In this yaml file, I have some key-remapping configuration:
In file `remap.yaml`:
```
- JOB: intercept -g $DEVNODE | caps2esc -m1 | uinput -d $DEVNODE
DEVICE:
EVENTS:
EV_KEY: [KEY_CAPSLOCK, KEY_ESC]
```
Could you explain what it does, and which program might be using it?
As usual, the link needs to be standalone, i.e. on a line by itself.
This hopefully provides an easy and intuitive way to get per-buffer context.
[!NOTE] To enable this feature, turn on
gptel-track-media. This option is not new, until now it was used to turn on image link parsing for supported models. Now it's more inclusive and applies to plain text links too. (You can turn this on via elisp, or by clicking on[Ignoring media]in the header bar, or via the transient menu.)
I have two questions about this feature.
- Should
gptel-track-mediabe renamed to something more appropriate, likegptel-track-links? - How should links to directories be handled? Currently they are ignored.
One quick idea in the directory case would be to include the contents of the corresponding Dired buffer.
More generally, is there any reasonable default way (without custom link types) to link to a buffer? I'm not aware of any, but if there were, that could be useful, e.g., like the following, if it worked:
I got the following error:
[[buffer:*project-shell*]]
Can you please help?
Thank you so much for this feature and all you work on this package!
As a package user, I think gptel-track-links is more intuitive now. However:
- Are all links supported? If use a TRAMP link or FTP, is that handled? If not, maybe
gptel-track-file-linkswould be more correct? - Are all file types linkable? Perhaps if I'm new, it would be surprising to find out that some of the file types linked to are not supported (e.g., if I link to an
mp4file).
My preference would be to add all files in the directory, which I think is what happens if done via the context submenu. Are there any edge cases from that implementation?
Thank you again, karthink!
- Are all links supported? If use a TRAMP link or FTP, is that handled?
Only file: links are handled, but all file: links are handled. So TRAMP links like
[[file:/ssh:user@server:/path/to/file]]
will work fine.
FTP links are not handled.
If not, maybe
gptel-track-file-linkswould be more correct?
It would be but I'm wary of the growing number of user options -- the current state of Org mode is my antipode here.
- Are all file types linkable? Perhaps if I'm new, it would be surprising to find out that some of the file types linked to are not supported (e.g., if I link to an
mp4file).
In principle all file links are checked. But whether the content is sent depends on what MIME types the model accepts. The Gemini models can accept videos, for example, but gpt-4.1 can't. You can check what is supported via (gptel--model-mimes gptel-model). If a link points to a file type that is unsupported the corresponding file contents are not included in the request. (gptel will issue a message at the time of the request about this.)
My preference would be to add all files in the directory, which I think is what happens if done via the context submenu. Are there any edge cases from that implementation?
No edge cases, it's just that the user might mistakenly include too much. It's easy to miss a link to a directory that you intended to be something else.
More generally, is there any reasonable default way (without custom link types) to link to a buffer? I'm not aware of any, but if there were, that could be useful, e.g., like the following, if it worked:
There isn't a built-in link type to refer to a buffer that I know of. The closest would be [[elisp:(pop-to-buffer "*project-shell*")]] or something like that.
While it's easy to introduce a bespoke Org link type for this,
- I prefer to stick to what Org mode already provides to contain the cognitive load of using gptel, and
- Markdown doesn't have anything like this.
This is also a slippery slope. There are many objects that can be linked to in Org that aren't files. With Magit/Forge for example you can link to a GitHub issue using [[orgit-topic:somehash123]]. Should gptel follow the link and include the contents of the issue buffer? Ditto for links to info pages, emails (Notmuch/Mu4e/Gnus), RSS entries etc.
Would it make sense for the context header to include the full path to the file rather than just the basename? Ie, a link like [[file:/path/to/remap.yaml]] would result in
In file `/path/to/remap.yaml`:
rather than
In file `remap.yaml`:
I saw the idea mentioned above, and wanted to add some support for it : having some kind of explicit @file marker in front of file link.
I am an org user, so this could look like:
@file [[/path/to/remap.yaml]]
I get that this adds additional syntax one has to remember, but it also makes things more explicit.
Let's say this "file cookie" has to be at the very start of the line: now I can easily add font-lock rules to highlight it in the chat buffer, making it very visible for me (which helps me ensure I don't forget about the files I added).
Also, it is easier for me to remember that one cookie, @file, than to remember that they have to be standalone links, with blank line above and blank line below -> I might by accident remove one of those blank lines and not know that file is not being added to context any more.
I guess one could make a bit stretched comparison to typed vs dynamic languages: same as types, this allows me to express my intention explicitly, reducing space for accidental changes that would violate it, and also get some feedback if I formatted the link wrong (not a file, not a link, ... -> while now that just means the link gets ignored without me being aware).
Similar system could then be used for dirs for example:
@dir [[/path/to/somedir]]
therefore avoiding the risk of user by accident providing dir where they wanted to provide a file, which you stated @karthink as one of the reasons not to support the dirs at the moment.
Note: I guess this would conflict with presets cookies (@preset-name) though. Solution might be to refactor preset cookies to have to start with preset: @preset:preset-name. Also explicit, which I like in general as an approach. If we want to go with backwards compatibility though, we could use different syntax for file/dir/... instead of @, although I like @ as it becomes a uniform way to communicate something to LLM from the chat, or we could say that if it is at the start of the line than more specific cookie wins over preset, but that is a rule that is too easy to forget about and I wouldn't do it.
Nice idea, @file, but it collides with other usage.
Marking links as context in some way is highly desirable IMHO. If we can do this it would mean the requirement that context and media links be explicit file links, on a line of their own, could be dropped.
Another way of doing this might be to define a org link alias for "file:" which perhaps could be called "gptel:".
It would do exactly the same thing as "file:" but explicitly tells GPTel that this link is a gptel context. Given that it is a custom link it could be given its own face to visually distinguish it from regular file links.
The general case would be to define a set of these links, perhaps via a new configuration variable for GPTel. I can imagine "gptel-file:" for links to local files, of any type the model supports, and "gptel-curl:" for context links to URLs and "gptel-id:" for links to org headings by their ID etc.
In all cases, each link alias can be given its own face for visual decoration, and they can be used embedded in text, with descriptions if, desired without any special formatting (which is as it stands quite fragile).
It would mean one could create, for example, a paragraph of text that is in effect a (system) prompt in which it has embedded context, which paragraph could be cut and paste anywhere one needed it to ensure the context required to engage with the model is comprehensive and complete.
When GPTel sees any of such links it presumes they are context to be retrieved.
What do you think @karthink ?
Nice idea, @file, but it collides with other usage.
With what usage does it collide? Presets, that I mentioned, or something else also? That could easily be solved by adding preset: in front, which anyway is probably helpful, to make it more explicit.
And the nice thing one gets than is that all the annotations for gptel in the text are the same: whatever starts with @. No special cases like custom links you described and similar.
Nice idea, @file, but it collides with other usage.
With what usage does it collide? Presets, that I mentioned, or something else also?
Yes, presets, but in general the ubiquity of @ for mentions in many platforms creates namespace pollution where the @ symbol gets overloaded for multiple purposes, leading to parsing ambiguity and user confusion even if there is nothing else (in org mode) that it collides with. Just my 2c.
Yes, presets, but in general the ubiquity of @ for mentions in many platforms creates namespace pollution where the @ symbol gets overloaded for multiple purposes, leading to parsing ambiguity and user confusion even if there is nothing else (in org mode) that it collides with. Just my 2c.
That makes sense, just @ is not very specific, even if not used for anything right now in org/markdown. It could easily be something else, @@file, or @gptel:file, or {!file}, what I was really suggesting here is not a specific prefix/notation, but a notation that can be used consistently to communicate explicit intention of something being a special piece of text to be understood and processed by the gptel.
Maybe there is no such notation that wouldn't conflict with something else? I doubt that though.
Another way of doing this might be to define a org link alias for "file:" which perhaps could be called "gptel:".
It would do exactly the same thing as "file:" but explicitly tells GPTel that this link is a gptel context. Given that it is a custom link it could be given its own face to visually distinguish it from regular file links.
The general case would be to define a set of these links, perhaps via a new configuration variable for GPTel. I can imagine "gptel-file:" for links to local files, of any type the model supports, and "gptel-curl:" for context links to URLs and "gptel-id:" for links to org headings by their ID etc.
...
What do you think @karthink ?
It's certainly possible to define a new Org link type (or multiple new link types) to indicate that gptel should follow the links and include their references. In fact it's pretty easy.
But every gptel-specific feature I add increases the learning curve and the cognitive load of using gptel. As you are aware, I usually try to reuse features provided by Emacs out of the box that users are already familiar with.
The new @preset cookie syntax was an exception, because it's a single piece of
syntax, and you can do many, many things with presets. Their potential has
barely been scratched, see for example the presets installed by the Macher
package.
A short tangent on presets:
Each @preset is essentially a family of behaviors, equivalent to an app.
In fact, you can write a preset that essentially implements this behavior --
i.e. it replaces links of the form [[gptel-file:/path/to/foo]] with its
reference content, in something like:
@links What do you make of [[gptel-file:/path/to/foo][this file]]?
Alternatively, the @links preset can replace all [[file:...]] links in the
prompt with their reference content, and not just standalone ones.
In contrast, a globally recognized gptel-foo: family of link types adds a lot
of new syntax for not nearly enough utility.
Also, any bespoke design for including links should also cover markdown-mode. Otherwise we have more Org-mode-only syntax, which makes gptel a little more confusing on the whole.
Can't font-lock highlight rules be added without this marker?
Yes, you can add a font-lock rule that adds a little check mark at the end of links that are recognized as context. The link will have to:
- be of type
file:,:attachmentor their Markdown equivalent, - be a standalone link (
gptel-org--link-standalone-porgptel--link-standalone-p), - correspond to a readable file,
- and have a MIME type supported by the active model.
See gptel--parse-media-links for details.
A PR for this is welcome, but the above checks will have to be quite optimized, and minimize both memory allocation/GC pressure and event loop load. Unfortunately, it is very easy to write font-lock rules that slow down Emacs. This is why I haven't added it yet.
That makes sense, just
@is not very specific, even if not used for anything right now in org/markdown. It could easily be something else,@@file, or@gptel:file, or{!file}, what I was really suggesting here is not a specific prefix/notation, but a notation that can be used consistently to communicate explicit intention of something being a special piece of text to be understood and processed by the gptel. Maybe there is no such notation that wouldn't conflict with something else? I doubt that though.
@Martinsos The current indicator of this intent is the placement of the link on a paragraph by itself, which is simple to do, requires no syntax, and does make it stand out:
Some text, along with a [[file:/path/to/foo][link]] that won't be included, and
[[id:...][another link]] that also won't be followed.
[[file:/path/to/foo][Link to be followed]]
Some more text here.
Essentially you are asking to replace this signal with some new special syntax.
For reasons mentioned above, I try not to introduce new syntax into gptel, or at
least do it very slowly. (@preset is the first bit of gptel-specific syntax
I've added in two years.)
Maybe I can make the link-validity check customizable, and you can write your own syntax?
That makes sense, just
@is not very specific, even if not used for anything right now in org/markdown. It could easily be something else,@@file, or@gptel:file, or{!file}, what I was really suggesting here is not a specific prefix/notation, but a notation that can be used consistently to communicate explicit intention of something being a special piece of text to be understood and processed by the gptel. Maybe there is no such notation that wouldn't conflict with something else? I doubt that though.@Martinsos The current indicator of this intent is the placement of the link on a paragraph by itself, which is simple to do, requires no syntax, and does make it stand out:
Some text, along with a [[file:/path/to/foo][link]] that won't be included, and [[id:...][another link]] that also won't be followed. [[file:/path/to/foo][Link to be followed]] Some more text here.Essentially you are asking to replace this signal with some new special syntax. For reasons mentioned above, I try not to introduce new syntax into gptel, or at least do it very slowly. (
@presetis the first bit of gptel-specific syntax I've added in two years.)Maybe I can make the link-validity check customizable, and you can write your own syntax?
I understand the reasoning and it makes sense, and I like the focus you are putting on reducing cognitive load -> ideally behaviour would be intuitive and explicit.
The issue I see with it is that it is not explicit: it is a rule in gptel that one has to first learn from the docs and then remember in order to be aware of this behaviour. If I don't know the rule, I could easily be sending a file to LLM by accident (if I have a standalone link and I have image sending enabled for some other image I added via context).
Even if I do know about the rule and I want to send the link, it is also hard to verify that link is indeed being sent: there is no visual indication of link being considered by gptel as link to be sent, meaning I have to myself know the exact gptel rule and run it in my head. Rule is quite complex, as you described it above, with the 4 bullet points, and the last bullet point, file being readable, I can't even check by just looking at it. Nor its MIME type. Additionally, rule might change with future versions of gptel, and I might not be aware of the change.
One solution is to visually show that that link is picked up by gptel, most likely by applying special face to it. But as you said yourself, it is not easy to implement such font-lock rule, due to how complex the actual rule is. Not to mention that if I, user, implement the font lock rule for it, it again might become incorrect once gptel updates the rule for links.
Also, what is missing is intention. I will use a bit of an edge case example, but let's say I open an older chat I saved, and I see that two links are being sent, two are not. Was that my original intention? Or did gptel update the rule in the meantime? Did I by accident delete that blank line above the link?
What having a special syntax like @file provides is a way to express intention in an explicit manner.
- I can put
@filenext to a link and if gptel realizes it is not a valid link (doesn't satisfy some of the rules: not readable, wrong mime type, wrong link type, ...) , it can throw me an error. - I don't have to remember any rules but
@filekeyword. - I won't get link sent by accident, without even knowing about this feature.
- Implementing font-lock rule for it is trivial.
- It is robust to gptel possibly changing the rule of what a valid link is (in the future gptel versions).
- It works the same way both in Markdown and Org.
- If I see
@filekeyword being used in an example chat or somewhere, I will understand that it is a feature even without knowing about it. - If I open an old chat, I don't have to guess what my intention was, I see which links I wanted to send.
- It allows for links not having to have blank lines above and below but they can be part of normal text. And
@fileis not as fragile as blank lines to accidentally messing it up.
I would say @file reduces cognitive load, instead of increasing it (i.e. no rules to remember, no concept of "standalone link", obvious even without knowing about the feature, discoverability (if completion)).
It gives the same benefit as a strict type system gives in a programming language -> way to express intention and get feedback from the compiler/interpreter on it.
I would even go further to suggest using @ keywords for all annotations is also cognitively easy because it gives a uniform way to do all annotations and makes it easy to scan visually for them. It could even support autocompletion -> I type @ and get the list of possible annotations by gptel (preset, file, ...), making it even easier to become aware of / remember gptel functionality.
As an example, imagine explaining to a friend how special behaviour in gptel chat buffer works.
- Right now: "You can specify presets with
@<preset-name>, it will be active for that message. If you want to add a link, you need to write a valid org/md link that is standalone: that means it needs one blank line above and one below. It can't be inline. It also needs to be valid: only file: links and I think attachments? It needs to be in those brackets, or in lesser bigger signs, can't be raw. Oh, how do I know if link is valid? Ha I guess you notice if LLM used it or not. Also, there might be some other rules, but I haven't read the whole docs yet." - With
@keywords: "You can annotate stuff with@<keyword>and gptel will pick it up. You can add a file with@file, set a preset for specific message with@preset:<preset_name>, and there is more I think, just type@and you will get suggestions. Oh yes, it will let you know if link is invalid, sure."
Uff I wrote a lot :D. I might be biased here somewhat by my appreciation of strictly typed prog languages, probably making me wish for this kind of "explicitness", but now that I explained the idea in detail I am even more convinced it is a good one hah. In any case I am interested in how you find this.
p.s. It doesn't have to be @ as prefix, it can be some other explicit type of annotation, I am just going with it for now.
If I don't know the rule, I could easily be sending a file to LLM by accident (if I have a standalone link and I have image sending enabled for some other
image I added via context).
I don't think this would happen by default -- you would either need to set gptel-track-media or use the header line in gptel-mode.
For me this suggestion is a little like having to type @confirm before any shell command that has rm with an asterix as an argument. We expect the user to be responsible for their input. I wonder if some kind of opt-in warning or confirmation might suffice to address the concern, though.
If I don't know the rule, I could easily be sending a file to LLM by accident (if I have a standalone link and I have image sending enabled for some other image I added via context).I don't think this would happen by default -- you would either need to set
gptel-track-mediaor use the header line ingptel-mode.
That is what I said above, in the parentheses: "... and I have image sending enabled for some other image I added via context".
I appreciate you taking the time to comment on my comment, but you haven't commented on most of the arguments I listed above, just a small part. Not that you are obliged in any way to read them, I did write quite a bit, but it is hard to argue further for me otherwise.
I am also not sure that the rm analogy really tracks, rm has its clear syntax and semantics that are not overloaded, and the whole problem in question is quite different.
EDIT: I just edited the original message to make it a bit easier to read, it is still a wall of text/paragraphs, but a bit less with some bullet points and bolded parts.
Would be nice to have id links be evaluated, too.
I start writing my documentation for an issue/feature from a new org node, and this node often links to many other "related" nodes (with org-super-links, but this is a plain org feature). Would be nice if each was added to the context, then the "sublinks" too, recursively.
Would be nice to have
idlinks be evaluated, too.I start writing my documentation for an issue/feature from a new org node, and this node often links to many other "related" nodes (with
org-super-links, but this is a plain org feature). Would be nice if each was added to the context, then the "sublinks" too, recursively.
@nvimtor I plan to make the link-as-context system more flexible, allowing for any org-link type to be followed and added as context. Will be a while before I can work on it though.
What having a special syntax like @file provides is a way to express intention in an explicit manner.
I can put @file next to a link and if gptel realizes it is not a valid link (doesn't satisfy some of the rules: not readable, wrong mime type, wrong link type, ...) , it can throw me an error. [...] It allows for links not having to have blank lines above and below but they can be part of normal text. And @file is not as fragile as blank lines to accidentally messing it up.I would say @file reduces cognitive load, instead of increasing it (i.e. no rules to remember, no concept of "standalone link", obvious even without knowing about the feature, discoverability (if completion)).
@martinsos I'm not working on the context-specification syntax right now, but I wanted to point out that this kind of thing is quite easy to do with a prompt transformation in gptel. You can try it out and let me know what you think.
The below adds syntax like
What is in this file
@file /path/to/file.yaml
and in this buffer?
@buffer *scratch*
;; Add a new prompt transformation that looks for @file
(add-hook 'gptel-prompt-transform-functions 'my/gptel-inject-files)
;; The transformation:
(defun my/gptel-inject-files (_)
"Search backward, injecting text files into context as needed.
Specify files as:
@file /path/to/file.ext"
(while (and (re-search-backward "^\\s-*@file\\b" nil t) ;look for @file
(not (get-char-property (point) 'gptel))) ;avoid LLM response regions
(goto-char (match-end 0))
(let ((file-name (string-trim ;grab file name
(buffer-substring-no-properties
(point) (line-end-position)))))
(delete-region (point) (line-beginning-position))
(cond
;; For directory: inject list of files
((file-directory-p file-name)
(insert (format "\nFiles in directory `%s`:\n\n```\n" file-name))
(dolist (f (directory-files-recursively file-name "." t t))
(when (file-readable-p f) (insert f "\n")))
(insert "```\n"))
;; For binary files: ignore for now
((gptel--file-binary-p file-name)
(message "Binary file \"%s\" not supported, ignoring @file"
file-name))
;; For file: inject file
((file-readable-p file-name)
(insert "\n")
(gptel--insert-file-string file-name))
(t (message "File \"%s\" not readable, ignoring @file"
file-name)))
(delete-region (point) (line-end-position)))))
Here's one for buffers:
;; Add a new prompt transformation that looks for @buffer
(add-hook 'gptel-prompt-transform-functions 'my/gptel-inject-buffers)
;; The transformation:
(defun my/gptel-inject-buffers (_)
"Search backward, injecting text files into context as needed.
Include buffers by name as:
@buffer *scratch*"
(while (and (re-search-backward "^\\s-*@buffer\\b" nil t) ;look for @buffer
(not (get-char-property (point) 'gptel))) ;avoid LLM response regions
(goto-char (match-end 0))
(delete-region (point) (line-beginning-position))
(let ((buf-name (string-trim
(buffer-substring-no-properties
(point) (line-end-position)))))
(if (not (buffer-live-p (get-buffer buf-name)))
(message "Buffer \"%s\" not live, ignoring @buffer"
buf-name)
(delete-region (point) (line-end-position))
(insert (format "\nIn buffer `%s`:\n\n```\n" buf-name))
(insert-buffer-substring-no-properties buf-name)
(insert "\n```\n")))))
Just a note, buffer-local context specification (add files or buffers to context) is now possible, see https://github.com/karthink/gptel/issues/475#issuecomment-3341392365
gptel will now annotate links, and let you know via a tooltip why a link path cannot be included with the prompt. Assuming you have link tracking turned on (with gptel-track-media), this is automatic in gptel buffers.
Links also don't have to be "standalone" any more, all links will be included. You can revert to the old behavior if required.