Bothway Sync between modifyed tangled file and org file in common programming workflow

[reyman]

Hi

This is more a discussion than a question.

I suppose this is a common use case in dev team where people have different habits : multiple tools, multiple workflow, multiple os.

I'm very interested to develop Literate Programing in my works but i found this is very difficult to impose this LP habits to other people working with me.

For example, if i wrote (me/reyman) a software in python using powerfull org-mode, org-transclusion, i generate a myprogram.org document that weave/tangle in myprogram.py and myprogram.md before executing.

The function A need B to do the right thing correctly bla bla bla ...

#+NAME functionA
#+BEGIN_SRC python :tangle modules.py 
def myfunctionA():
    <<functionB>>
    ...
#+END_SRC

The function B order thing using a state of the art algorithm in machine learning bla bla bla ...

#+NAME functionB :tangle modules.py
#+BEGIN_SRC python
def myfunctionB():
    ... do something ...
#+END_SRC

Finally the program run this two very important function like this ... bla bla ...

#+NAME functionB
#+BEGIN_SRC python :tangle main.py
from module import *

def main():
   var result = myfunctionA()
#+END_SRC

My friends use directly myprogram.py file because like 90% of programmers they don't want LP in their workflow, and they use an IDE that don't understand anything about org file. So they directly modify the result tangled file.

So even with very small refactoring, if they decide to change the name of myfunctionA to myfunctionA1 or ... rename the source file that contain the function A and B to modulexx.py, my org file will be automatically broken and not in sync with these modification.

I think one way to limit interference will be to integrate unique uuid as code comment for each chunk during tangling of file, so even with little modification (like moving or renamming file) the uuid follow and it will be possible to re-sync the content after my friend modification.

Or, probably this is most complicated, and perhaps the only way to do that will be to use syncweb (https://github.com/aryx/syncweb) software ? I didn't found integration for this tool with org, do you know something about that ?

Do you think this is a discussion i need to move to org mailling liste ?

[nobiot]

Just very quickly (i’ll comeback), how about linking to a certain commit in Git?

[reyman]

Hum, I'm not sure to understand well the mechanism of git link but i think that solve only part of the problem no ?

In this case we have a desynchronisation between :

• myprogram.org i use to produce myprogram.py and module.py, tangled and stored into git repo.
• myprogram.py and module.py modified (ex. myfunctionA renamed to myfunctionA1 or module.py renamed to modulexx.py) and pushed on git by my colleague.

If i pull modification of my colleague, run the org doc, that overwrite myprogram.py, and recreate a module.py. I work on a codebase totally desynchronized.

One way was to run some sync tool to check if everything goes well using unique identifier :

• uuid of myFunctionA continue to exist in modulexx.py, name of :tangle need to evolve, and probably the snippet of code stored in org file.
• if uuid disapear into source code, that's probably because this function doesn't exist anymore in the current codebase. Or my colleague accidentaly delete the uuid comment associated to this function... User need to check manually

This is not magic i suppose, because if my colleagues decide to do some major refactoring that will probably break the logic in my org file;

I don't know if it's exist, but another way to think this problem will be to extract and display into org file one part only of the source code i want to describe. By using some uuid or specific noweb tag << ... my function A content >> the tangling process will only replace/sync this part of org file into the original source code file.

For example if myfunctionA is described between two tag that delimit a zone between line 35 to line 70 in module.py :

<< uuid xcfvdes34d 
def myFunctionA() : 
    print ("helloworld")
    ... 
>>

i only need to sync this part (L35-70) from source code to org corresponding uuid chunk.

#+NAME: myfunction
#+UUID: xcfvdes34d
#+BEGIN_SRC python :tangle module.py
def myFunctionA() : 
    print ("helloworld")
    ... 
#+END_SRC

Also Tangling from this org file only replace this part L35-70 into module.py.

My org file doesn't contain all my program, but only one part i want to describe. If everything goes well, and my colleague respect the uuid tag, for example with minor refactoring when the name of function evolve, my org file continue to work.

[reyman]

Hi @nobiot , i try the existing functionalities ::id-1234 with :end and i'm not successfull, is there something i don't understand ?

#+transclude: [[file:./main.jl::id-1234]]  :src julia :end "id-1234 finish"

with my julia code main.jl is :

# id-1234
function init_fig(model, observable)

    function ants_marker(b::Ants)
       φ = atan(b.vel[2], b.vel[1]) #+ π/2 + π
       scale(rotate2D(ants_polygon, φ), 2)
    end

    function ants_color(b::Ants)
       return b.color
    end

    fig, abmstepper, abmobs = abmplot(model; ants_agent_step!, ants_model_step!, am= ants_marker, ac=ants_color, figure=(;resolution=(800,800)))

    ax, hm = heatmap(fig[1,2], observable[1]; colormap=cgrad(:thermal))
    ax.aspect = AxisAspect(1)

    ax2, hm2 = heatmap(fig[2,1], observable[2]; colormap=cgrad(:thermal))
    ax2.aspect = AxisAspect(1)

    ax3, hm3 = heatmap(fig[2,2], observable[3]; colormap=cgrad(:thermal))
    ax3.aspect = AxisAspect(1)

    s = Observable(0) # counter of current step, also observable
    Colorbar(fig[1, 3], hm, width = 20,tellheight=false)
    rowsize!(fig.layout, 1 , ax.scene.px_area[].widths[2])

    return (fig, abmstepper, abmobs)

end

# id-1234 finish

[nobiot]

When you say “unsuccessful”, how does it behave?

[reyman]

@nobiot all file is included

[nobiot]

@reyman , I don’t seem to be able to reproduce the problem. Are you using the latest version?

For the code, does changing the major mode for julia’s bufffer change anything? For jl files, I only have fundamental mode — I don’t know if that changes the behavior though…

[reyman]

@nobiot I use the 2022-06-13 commit into doom, commit 72e0e

I will try with other mode than julia? to see if that change something.

[nobiot]

I use the 2022-06-13 commit into doom, commit 72e0e

The latest for Org-transcclusion on this GitHub repo is on 2022-06-12, commit 5cb9454. I don't know what this "commit into doom" is (?)

I tried again with your julia code as below; It worked as designed.

[reyman]

The latest for Org-transcclusion on this GitHub repo is on 2022-06-12, commit 5cb9454. I don't know what this "commit into doom" is (?)

Sorry for the lack of details, i'm using the latest doom-emacs version (sync today) with emacs 28.1 native on guix. The commit i give in my previous message is the comit of .doom.d/.local/straight/repos/folder/org-transclusion (melpa commit i suppose), but the README.org say this is the 1.3.0 version of org-transclusion.

To be sure, i up doom version and recompile all files from doom + emacs native.

Starting from empty org, i add in order :

#+transclude: [[file:/home/reyman/Volumes/openmole/julia/ants/main.jl::id-1234]]  :src julia :end "id-1234 finish"

then Run M-x org-transclusion-activate then Run M-x org-transclusion.add (=> F12)

But the main.jl continue to appear in full version. When i use starting and ending lines that works.

This could be a linux vs windows problem ? I'm using Guix (linux) actually. Is there a way to better debug what happen ?

[nobiot]

The screen shot was taken in my Ubuntu.

For further debugging, you can use edebug and see how this line onwards behave. L109 is to get the value of :end prop

[reyman]

  org-transclusion-content-range-of-lines((link (:type "file" :path "/home/reyman/Volumes/openmole/julia/ants/main.jl" :format bracket :raw-link "file:/home/reyman/Volumes/openmole/julia/ants/main..." :application nil :search-option "id-1234" :begin 1 :end 67 :contents-begin nil :contents-end nil :post-blank 0 :parent (paragraph (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity element :parent (section (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :robust-begin 1 :robust-end 65 :post-blank 0 :post-affiliated 1 :mode first-section :granularity element :parent (org-data (:begin 1 :contents-begin 1 :contents-end 67 :end 67 :robust-begin 3 :robust-end 65 :post-blank 0 :post-affiliated 1 :path nil :mode org-data :CATEGORY nil)))))))) (:end "id-1234 finish" :src "julia" :link "[[file:/home/reyman/Volumes/openmole/julia/ants/ma..." :current-indentation 0))
  org-transclusion-content-src-lines((link (:type "file" :path "/home/reyman/Volumes/openmole/julia/ants/main.jl" :format bracket :raw-link "file:/home/reyman/Volumes/openmole/julia/ants/main..." :application nil :search-option "id-1234" :begin 1 :end 67 :contents-begin nil :contents-end nil :post-blank 0 :parent (paragraph (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity element :parent (section (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :robust-begin 1 :robust-end 65 :post-blank 0 :post-affiliated 1 :mode first-section :granularity element :parent (org-data ...))))))) (:end "id-1234 finish" :src "julia" :link "[[file:/home/reyman/Volumes/openmole/julia/ants/ma..." :current-indentation 0))
  org-transclusion-add-src-lines((link (:type "file" :path "/home/reyman/Volumes/openmole/julia/ants/main.jl" :format bracket :raw-link "file:/home/reyman/Volumes/openmole/julia/ants/main..." :application nil :search-option "id-1234" :begin 1 :end 67 :contents-begin nil :contents-end nil :post-blank 0 :parent (paragraph (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity element :parent (section (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :robust-begin 1 :robust-end 65 :post-blank 0 :post-affiliated 1 :mode first-section :granularity element :parent (org-data ...))))))) (:end "id-1234 finish" :src "julia" :link "[[file:/home/reyman/Volumes/openmole/julia/ants/ma..." :current-indentation 0))

With tracing

Tracing...
Result: (link (:type "file" :path "/home/reyman/Volumes/openmole/julia/ants/main.jl" :format bracket :raw-link "file:/home/reyman/Volumes/openmole/julia/ants/main..." :application nil :search-option "id-1234" :begin 1 :end 67 :contents-begin nil :contents-end nil :post-blank 0 :parent (paragraph (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity element :parent (section (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :robust-begin 1 :robust-end 65 :post-blank 0 :post-affiliated 1 :mode first-section :granularity element :parent (org-data (:begin 1 :contents-begin 1 :contents-end 67 :end 67 :robust-begin 3 :robust-end 65 :post-blank 0 :post-affiliated 1 :path nil :mode org-data :CATEGORY nil))))))))
Result: "/home/reyman/Volumes/openmole/julia/ants/main.jl"
Result: (link (:type "file" :path "/home/reyman/Volumes/openmole/julia/ants/main.jl" :format bracket :raw-link "file:/home/reyman/Volumes/openmole/julia/ants/main..." :application nil :search-option "id-1234" :begin 1 :end 67 :contents-begin nil :contents-end nil :post-blank 0 :parent (paragraph (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity element :parent (section (:begin 1 :end 67 :contents-begin 1 :contents-end 67 :robust-begin 1 :robust-end 65 :post-blank 0 :post-affiliated 1 :mode first-section :granularity element :parent (org-data (:begin 1 :contents-begin 1 :contents-end 67 :end 67 :robust-begin 3 :robust-end 65 :post-blank 0 :post-affiliated 1 :path nil :mode org-data :CATEGORY nil))))))))
Result: "id-1234"
Result: "/home/reyman/Volumes/openmole/julia/ants/main.jl"
Result: #<buffer main.jl<2>>
Result: (:end "id-1234 finish" :src "julia" :link "[[file:/home/reyman/Volumes/openmole/julia/ants/ma..." :current-indentation 0)
Result: nil
Result: (:end "id-1234 finish" :src "julia" :link "[[file:/home/reyman/Volumes/openmole/julia/ants/ma..." :current-indentation 0)
Result: "id-1234 finish"

and after ...

Result: "## MAIN\n\ninclude(\"ants.jl\")\ninclude(\"sugar.jl\")\n\nu..." [2 times]
Result: #<buffer main.jl<2>>
Result: 1 (#o1, #x1, ?\C-a)
Result: 3426 (#o6542, #xd62)
Result: (:src-content "## MAIN\n\ninclude(\"ants.jl\")\ninclude(\"sugar.jl\")\n\nu..." :src-buf #<buffer main.jl<2>> :src-beg 1 :src-end 3426) [6 times]

Like this, i don't see where could be the problem ... The contents-begin 1 to 67 is strange because lines into main.jl file are located between 20 and 50.

Edit :

I made the same test using python file and it's work. So this is much a julia problem it seems.

[nobiot]

One thing I could try to change is to visit the buffer as raw file. Not sure if this is possible.

To be sure, what’s the mode you use for Julia files?

[nobiot]

The content-end 67 is point, I think, not the line number. It should be from org-element. Org-transclusion convers this to line number (based on ox code).

[nobiot]

Would you be able to try this new branch, named lines-org-id by any chance? I just pushed a commit to change find-file-no-select to open file as a raw file.

I also tried julia-mode; I still cannot reproduce your problem on my end... I can see that main.jl is visited with Julia mode on...

[reyman]

@nobiot I update emacs-doom, org-transclusion was rebuilded when i check log (running doom up command.) And now that works ...

Sorry, i didn't understand what happened ...

If you want i could also test the branch you created, the time to see how declare package branch in doom.

[nobiot]

Great that it worked for you. Sure I’d appreciate your testing if you have some time but please don’t force yourself :)

[reyman]

Great that it worked for you. Sure I’d appreciate your testing if you have some time but please don’t force yourself :)

Using this in my package.el, i get the branch and org-transclusion is reinstalled.

(package! org-transclusion :recipe
  (:host github
   :repo "nobiot/org-transclusion"
   :branch "lines-org-id"))

I check on my julia and python test and it seems thats :end continue to works normally !

I'm really happy to use this functionnality to write doc and program with my team, if you add new functionnality like unique uuid (like described on this original issue) or git commit associated to code block, etc i will be very happy to test them ;) !