Opening large file

[alxhoff]

I have a large dox file with ~24,000 nodes. Xdot hangs when I try to open it. Just wondering if there is a known limitation to Xdot or if something else is the problem. Cheers

[jrfonseca]

xdot.py is written in Python, which is byte-interpreted language. It's certainly possible it can't sustain rendering so many nodes.

It's also possible the issue is not Python, but Cairo underneath taking too much time to render.

Off hand I don't know how many nodes Xdot can sustain. I don't recall ever testing as many as 24K nodes.

I don't have time to look into this myself, but perhaps if you attach it to this bug, somebody eventually will.

[alxhoff]

Thanks for the reply. I'll generate a big graph again tomorrow over lunch. I think my last big one was cleaned up. I seem to be capping out with graphs with 2-3k nodes.

[yarbs]

@alxhoff If the file does not contain sensitive data, would you be willing to share the file for reproducing the error?

[iamahuman]

@alxhoff #63 would have fixed the cases where the graph is sparse and you're viewing only part of it zoomed. Still interested in the example, though.

[jtojnar]

Here is an example of a graph of package dependencies (3.4 MB uncompressed, 3000 nodes, >18000 edges):

gnome-pomodoro-closure.zip

Edit: In my case, it is already frozen in the filtering phase dot -Txdot gnome-pomodoro-closure.dot

Edit 2: After almost 3 hours the command finished producing a 23.3 MB uncompressed dot file: gnome-pomodoro-closure2.zip

Opening it directly with -n is still has a noticeable freeze but the ~5 seconds are laughable, compared to waiting on the filtering. After that the image renders and xdot is pretty responsive for such a mess of lines.

I recommend making all the potentially long-running operations asynchronous (using Python’s asyncio or GLib’s spawn_async function) and telling the user about the currently executed phase.

Edit 3: I deleted the tarball/patch nodes using gvpr -c 'N[name=="*.(patch|(tar.(gz|bz2|xz)|tgz|zip).drv)"]{delete(NULL, $)}' gnome-pomodoro-closure.dot, reducing the file to 1789 nodes, 12605 edges, 2.2 MB. The filtered file was now generated in only slightly under 2 hours.

[iamahuman]

@jtojnar Note that xdot.py accepts files with layout information, so you can just manually run the command for that matter.

To make exploring a large graph feasible we should also implement progressive quantizing rendering.

[jrfonseca]

If dot itself takes long then there's not much xdot.py can do, other than provide some feedback to diagnose the problem as suggested.

If the graph purpose is visualization by an human, a very large graph probably defeats the purpose. In my experience it's better to use domain knowledge to prune the graph to better highlight the interesting features.

Regarding better feedback, I don't know if asyncio is necessary. A status bar in the bottom stating Running dot ... would probably suffice. Either way, I'm afraid I don't have time to look into it myself. I'd accept patches though.