negative number of edges with g.ecount()

[slowkow]

I get a negative number of edge weights when my graph is very large.

Do you know of a way to work around this?

Let's start with a large adjacency matrix:

W
#> <2064298x2064298 sparse matrix of type '<class 'numpy.float64'>'
#>        with 4947583500 stored elements in Compressed Sparse Row format>

W.shape
#> (2064298, 2064298)

Convert it to a graph:

s, t = W.nonzero()
idx = s < t
s = s[idx]
t = t[idx]

len(s)
# 2473791750
len(t)
# 2473791750

G = igraph.Graph(directed=False)
G.add_vertices(W.shape[0])
G.add_edges(zip(s, t))

So, what did we get?

G.vcount()
#> 2064298

G.ecount()
#> -1821175546

Uh oh. A negative number of edges?

This might be the source of the problem when I try to set edge weights:

G.es["weight"] = 1

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-73-4587fc396ef8> in <module>
----> 1 G.es["weight"] = 1

SystemError: /tmp/build/80754af9/python_1565725737370/work/Objects/listobject.c:150: bad argument to internal function

igraph.__version__
#> '0.8.2'

[slowkow]

Uh oh, I guess I went over the 2 billion edge limit mentioned here:

https://lists.nongnu.org/archive/html/igraph-help/2016-04/msg00032.html

From:	Tamas Nepusz
Subject:	Re: [igraph] Maximum graph size that Python igraph can handle
Date:	Mon, 25 Apr 2016 11:12:18 +0200

Also, re the original question: there is no built-in constraint for the sizes of the graphs apart from the fact that vertex and edge IDs are typically stored as long integers, so you won't be able to have more than 2 billion vertices or edges. But, if you have that many vertices or edges then you probably have bigger problems anyway ;) T.

[szhorvat]

There is no way around this limit, but in future versions this will be addressed.

[slowkow]

Thanks! Looking forward to it. For now, I will find a way to reduce my edges.

For those of us who hit this limit, it would save us time to see a warning message like:

WARNING: You have more than 2 billion edges. This is not supported by python-igraph

I was scratching my head for a few days before I arrived at the negative edge number.

[szhorvat]

Yes, better error checking will also be included in the fix.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[iosonofabio]

@szhorvat @ntamas Since we are now moving to 64 bit integers, this might just wait until then.

Otherwise I'd be happy to push a quick commit that adds a warning when the graph reaches the max number of edges/nodes, what do you think?

[ntamas]

I'm wondering what the best place for that warning would be. In the end, all vertex / edges additions should trickle down to igraph_add_vertices() or igraph_add_edges() in the C core, so ideally we should add it there and not to the Python layer, especially considering the impeding transition from 32-bit to 64-bit integers in the C core. What do you think?

(Of course if we do it in the C layer, then it would be nice not to hard-code the limit but rely on sizeof(something) and powers of two instead).

[iosonofabio]

yes, those functions are where the action is. With the whole discussion about int64 taking place, I'll open an issue on the igraph repo and make it dependent on the int64 PR.

[ntamas]

This should be solved now that the C core has transitioned to 64-bit representations internally. I'm closing the issue, but feel free to reopen if you experience something similar with versions 0.10 or above.