Additional options for transcluding text and source files

[overideal]

The newly added ability to tranclude lines from a source file seems to work well in my testing and has huge potential. It happens quite often that one writes some code and wants to note down additional documentation, explanations etc., which might contain latex expressions and be quite long. Consequently, org seems like a good candidate. But only using org source blocks makes it hard to actually write the code (no "IDE feel", hard to write multiple blocks that depend on each other). With this package, one can still write in the normal source file and integrate the desired parts into the org document. This also avoids the problem of syncing between the two versions. After all, there is only one, namely the code in the source file. Ingenious! I really think this form of "literate programming" could be incredibly useful.

To make such a workflow possible, at least two additional options should be provided:

• It should be possible to specify the end of the transclusion using a search string (e.g. regexp). With this option, one could for example use a regexp like "^\s-\n\s-\n" to match two consecutive empty lines and then the transclusion would end at the first position after the start position at which two empty lines are encountered.
• It would also be nice to be able to specify a negative number as the second :lines argument to indicate that the transclusion should end that many lines before its usual end. E.g. in the example above one could set it to -2 in order to not include the two empty lines in the transclusion. Not specifying :lines should default to the region from start to end, where start and end are determined by the two search strings.

With these two options, after the tranclude block is created, the user only needs to manually intervene if the search string of the start or end point is removed or reappears too early.

[nobiot]

Thank you for your comments and idea.

It's interesting. My original intent of this package is more for non-programming writing, but it's nice to see that some people find usage for programming, too.

I cannot assess how complex this will be off the top of my head; it might be worth trying out something. Let me see what I can do. Of course, if you have a sample code, PR, etc., that would be fantastic, too.

[nobiot]

I believe #98 provides the requested feature. Closing for now. Feel free to reopen if there is anything further to discuss. Thank you.

[overideal]

Hi, the implementation works well. Great work! However, in my testing it seemed that searching for regular expressions does not work, which is a bit unfortunate as that feature could be very useful (e.g. to search for two consecutive empty lines to mark the end of a code block). Maybe this would be worth implementing.

Another comment (which is much more subjective, so feel free to ignore it): It feels a bit "asymmetric" that if you supply :end, then only the first part of :lines has an effect. Maybe the second part should also have an effect: just remove that many lines from the end of the source block.

Thanks for your continuous work on the package, it is really a joy to use!

[nobiot]

Thank you!

searching for regular expressions does not work

I guess I didn’t implement it… I’m coming from a non-programming and I guess i’m just not familiar with how to implement regexp feature. Do you have advice or resource I could read?

then only the first part of :lines has an effect

Sorry I don’t quite understand what the “effect” you are referring to?

[nobiot]

For regexp support, the current implementation relies on Org mode’s built-in org-link-search. If I were to extend the logic to support regexp, I’d need a handy function / form to do this…

;; FIXME `org-link-search' does not return
;; postion when ::/regex/ and ;;number are
;; used
  (when (org-link-search end-search-op)

[overideal]

First about the effect of :lines: In the current implementation, if you supply :end, then only the first part of :lines has an effect, e.g. :lines 1- is the same as 1-1 and 1-2 etc. I propose to also assign a meaning to the second part: Namely, interpret the number as moving back that many lines from the end. For example, :lines 1-1 would mean to display everything from the line that matched the start to the line that matched the :end string. Similarly, :lines 1-2 would display the same except for the last line previously displayed, which now would be excluded. In other words, we just negate the second number and exclude one minus that many lines from the end of the matched region.

Now about the regexp: I read into this a bit and while I am by no means an expert for regexps either, here is what I found: If we keep our syntax similar to that of org mode links, then we detect regexps by checking if the string starts and ends with a forward slash (/). This can be done by checking for (string-match "^/.*/$" search-option) and then calling org-link-search like you currently do or calling (re-search-forward (substring search-option 1 -1) nil t) instead in case the given string was indeed surrounded by two slashes. I was mainly testing with the string "/^[:space:]\n[:space:]\n/" in order to match two consecutive empty lines. The problem is that after reading the string from the org buffer, the backslashes essentially "double" (see https://emacs.stackexchange.com/questions/55601/re-search-forward-regex-behavior-for-or-boolean.). Because of the reason explained at https://emacs.stackexchange.com/questions/45382/elisp-regex-newline-escape-confusion, '\\n' (backslash backslash n) does not match a newline, so the regular expression above does not work. To fix this, I used (string-replace "\\n" "\n" (substring end-search-op 1 -1)) instead of just (substring end-search-op 1 -1). With this it seems to work in my limited testing, but it would be nice if someone with more knowledge could investigate this phenomena.

[nobiot]

@overideal Thanks for the additional comments.

For the first comment about :lines -- I understand that you would like an offset from the end line. It is not implemented yet. I will not be able to implement it any time soon purely due to my limited capacity at the moment. I'd welcome a contribution.

One thing I am not in favour of is (I could be persuaded otherwise of course) to use the second number of the :line property for the purpose of offset. This property is designed to be analogous to the way the built-in #+include works, so :line x-y is supposed to be a range between x and y. The hyphen "-" in the middle does not indicate y is a negative number. For the offset, I would suggest another property should be used, for example :end-offset, which could take a positive or negative integer.

For the regexp, I think I lack the skill set to implement it properly unless there is a function from either Org or some other built-in library within Emacs.