python changes have no impact

[crackcomm]

Hey, I love the project. I might be using it wrong but I can't make any changes in my small test have any impact. I included the index.scip in the repo.

I made and committed a change:

diff --git a/helloer/pkg1/myfunc.py b/helloer/pkg1/myfunc.py
index 64f393e..2c6f279 100644
--- a/helloer/pkg1/myfunc.py
+++ b/helloer/pkg1/myfunc.py
@@ -1,2 +1,2 @@
 def my_func(name: str) -> str:
-    return f"Hello {name}!"
+    return f"Hello there {name}"
diff --git a/index.scip b/index.scip
index 2dd2556..3252213 100644
Binary files a/index.scip and b/index.scip differ

I'm trying it like this:

scip-python index .
srctx diff --lang PYTHON --scip index.scip --outputHtml out.html --nodeLevel func

The full output:

INFO[0000] srctx version v0.11.0 (https://github.com/williamfzc/srctx)
INFO[0000] no config: config file does not exist at /home/pah/ocxmr-repos/opt/examplepy/srctx_cfg.json
INFO[0000] start diffing: /home/pah/ocxmr-repos/opt/examplepy
INFO[0000] func level main entry
INFO[0000] using SCIP as index
INFO[0000] scip -> lsif converted
INFO[0000] read lsif file: /home/pah/ocxmr-repos/opt/examplepy/dump.lsif
INFO[0000] index parser ready
INFO[0000] cache type: file
INFO[0000] creating fact graph
INFO[0000] creating rel graph, reference result map size: 8
INFO[0000] base graph ready. fact: 14, rel 12
INFO[0000] file count: 5
INFO[0000] createing fact with sibyl2
{"level":"info","ts":1701063763.176803,"caller":"core/runner.go:22","msg":"valid file count: 5"}
{"level":"info","ts":1701063763.1772585,"caller":"[email protected]/extract_fs.go:107","msg":"extract cost: 1 ms"}
{"level":"info","ts":1701063763.178545,"caller":"core/runner.go:22","msg":"valid file count: 5"}
{"level":"info","ts":1701063763.1786468,"caller":"[email protected]/extract_fs.go:107","msg":"extract cost: 1 ms"}
INFO[0000] fact ready. creating func graph ...
INFO[0000] edges building
INFO[0000] detect entries: 0
INFO[0000] func graph ready. nodes: 2, edges: 2
INFO[0000] file helloer/pkg1/myfunc.py line [2] hit no func
INFO[0000] file index.scip line [] hit no func
INFO[0000] diff finished.
INFO[0000] creating output html: out.html
INFO[0000] creating stat json: stat.json
INFO[0000] everything done.

Change in this function impacts this reference.

The rendered graph has no edges:

stat.json:

{
  "unitLevel": "file",
  "unitMapping": {
    "helloer/pkg1/myfunc.py:#1-#2:||my_func||": 0,
    "helloer/pkg2/fnuser.py:#4-#5:||fn_user||": 1
  }
}

Is how I'm using it wrong?

[williamfzc]

Thanks for your interest :) What if change the --nodeLevel func to --nodeLevel file ?

[crackcomm]

Thank you for a quick response!

I tried file before and there was no graph and func showed the nodes that's why I didn't try it again. It shows a graph with direct impacts but no indirect impacts:

{
  "unitLevel": "file",
  "unitMapping": {
    "helloer/__init__.py": 3,
    "helloer/pkg1/__init__.py": 4,
    "helloer/pkg1/myfunc.py": 0,
    "helloer/pkg2/__init__.py": 1,
    "helloer/pkg2/fnuser.py": 2
  },
  "impactUnits": [
    2,
    0
  ],
  "transImpactUnits": [
    2,
    0
  ],
  "entries": [
    4,
    1,
    2,
    3
  ],
  "impactEntries": [
    2
  ]
}

[williamfzc]

It looks like I have forgotten the implementation of TagYellow in file level graph ..

https://github.com/williamfzc/srctx/blob/main/graph/file/api_draw.go https://github.com/williamfzc/srctx/blob/main/graph/function/api_draw.go

I will fix it soon and thanks for pointing it out :)

[crackcomm]

Thank you. I also noticed unitLevel": "file" on --nodeLevel func which is very minor.

[williamfzc]

Hey, I love the project. I might be using it wrong but I can't make any changes in my small test have any impact. I included the index.scip in the repo.

I made and committed a change:

...

Is how I'm using it wrong?

The func level graph depends on opensibyl/sibyl2. https://github.com/opensibyl/sibyl2/blob/master/pkg/extractor/python/extractor_python_func.go It did not handle the root level function, which also needs some fix.

To be honest I prefer using the file graph in production in my own cases, which only depending on the LSIF/SCIP, much more stable.

[williamfzc]

Thank you. I also noticed unitLevel": "file" on --nodeLevel func which is very minor.

At the current moment, I would suggest that you directly utilize the generated dot file (--outputDot a.dot) for consumption, if you are going to consume the diff data programmatically. Standard dot format should be easily parsed by other famous graph libs (such as networkx).

Sorry for the bad design in the past. stat.json looks bad actually.

[crackcomm]

I am wondering why did you choose to use a separate tree-sitter extractor for the functions? I think it should be possible using just LSIF textDocument/references if I understand it correctly, I'm only just starting to research LSIF/SCIP and similar code indexing methods.

[williamfzc]

I am wondering why did you choose to use a separate tree-sitter extractor for the functions? I think it should be possible using just LSIF textDocument/references if I understand it correctly, I'm only just starting to research LSIF/SCIP and similar code indexing methods.

Because the range concept is different. For example,

# a normal function
def function_name():
    # whatever
    pass

Range

• In LSIF: line1 col 5 - line1 col 15
• What I want: line1 col1 - line4 col 1

There is another conception named textDocument/foldingRange which may matches my need. But unfortunatelly by default it did not be emitted to index file.

[crackcomm]

I see that makes sense now. I wonder if SCIP does have this information directly and also what was the motivation behind using LSIF instead of SCIP? My intuition is that LSIF indexers are available for more languages but maybe your reason was different?

Looks like SCIP does ineed have this information under enclosing_range.

[williamfzc]

You are right. I also noticed that it's really new which was added in one week ago. (https://github.com/sourcegraph/scip-python/commit/1572f2f8c52928a5975bfa27d7120fc7782ea3af)

There is no any other special reasons. Only because when I started this project, SCIP is also young. So I choose the stable one.

[williamfzc]

I have created another issue for the native SCIP support. https://github.com/williamfzc/srctx/issues/72

It should not be hard but it really takes some times. Thanks for the information :)