Moran Local returns ValueError

[iki77]

I'm unable to run MoranLocal

weights_q =  lps.weights.Queen.from_dataframe(gdf)
weights_q.transform = "r"
np.random.seed(42)
li = esda.moran.Moran_Local(y, weights_q)

as it will return

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-24-6402547a292e> in <module>
      1 np.random.seed(42)
----> 2 li = esda.moran.Moran_Local(y, weights_q)

~\miniconda3\envs\pysal-workshop\lib\site-packages\esda\moran.py in __init__(self, y, w, transformation, permutations, geoda_quads, n_jobs, keep_simulations, seed)
   1030         self.__moments()
   1031         if permutations:
-> 1032             self.p_sim, self.rlisas = _crand_plus(
   1033                 z,
   1034                 w,

~\miniconda3\envs\pysal-workshop\lib\site-packages\esda\crand.py in crand(z, w, observed, permutations, keep, n_jobs, stat_func, scaling, seed)
    157 
    158     if n_jobs == 1:
--> 159         larger, rlocals = compute_chunk(
    160             0,  # chunk start
    161             z,  # chunked z, for serial this is the entire data

ValueError: cannot assign slice from input of different size

However there is no problem when I run Moran using the same data

weights_q =  lps.weights.Queen.from_dataframe(gdf)
weights_q.transform = "r"
np.random.seed(42)
mi = esda.moran.Moran(y, weights_q)

and no problem when I run Join Counts using the binary of the same data

yb = y >= y.median()
labels = ["0.Less than median", "1.Greater than median"]
yb = [labels[i] for i in 1*yb] 

weights_q =  lps.weights.Queen.from_dataframe(gdf)
weights_q.transform = "b"
np.random.seed(42)
jc = esda.join_counts.Join_Counts(yb, weights_q)

[jeffcsauer]

Hi @iki77 - thanks for raising this issue. Could you by chance provide gdf? I'm having trouble reproducing the issue. For example, running the following code causes no issue:

# Load  example geopandas data - https://geopandas.org/gallery/choro_legends.html
_ = libpysal.examples.load_example('South')
pth = libpysal.examples.get_path('south.shp')
gdf = read_file(pth)
print(gdf.__class__)
# Replace y
y = gdf['HR60']
# Create weights
weights_q =  libpysal.weights.Queen.from_dataframe(gdf)
weights_q.transform = "r"
# Insert iki77 code
np.random.seed(42)
li = esda.moran.Moran_Local(y, weights_q)
li.p_sim

results in...

<class 'geopandas.geodataframe.GeoDataFrame'>

array([0.379, 0.031, 0.101, ..., 0.319, 0.378, 0.209])

Could you share the various versions of libpysal and esda?

[iki77]

@jeffcsauer The gdf can be downloaded from https://biogeo.ucdavis.edu/data/gadm3.6/gpkg/gadm36_IDN_gpkg.zip

gdf = geopandas.read_file("gadm36_IDN.gpkg", layer="gadm36_IDN_2")
# Initialize y
y = gdf["CC_2"].astype(float)
# Create weights
weights_q = libpysal.weights.Queen.from_dataframe(gdf)
weights_q.transform = "r"
# Moran --> No Problem
np.random.seed(42)
mi = esda.moran.Moran(y, weights_q)
mi.p_sim
# Moran Local --> Error
np.random.seed(42)
li = esda.moran.Moran_Local(y, weights_q)
li.p_sim

[ljwolf]

~~I think this may be related to issues we plan to fix this weekend with the parallelisation code. can you try using esda.moran.Moran_Local(y, weights_q, n_jobs=1)?~~

[ljwolf]

Yes, this is indeed an issue in compute_chunk() for data with islands and happens right here at crand.py#267... I can replicate with the provided data. For now @iki77, use esda.moran.Moran_Local(y, weights_q, keep_simulations=False). This will still let you analyse the significance of the simulation, but will not retain the li.rlisas, which are the raw randomly-simulated values.

[iki77]

Ok thanks for the temporary solution @ljwolf

[rsbivand]

I'm afraid no-neighbour observations are still an issue with released libpysal 4.5.1 and esda 2.4.1:

no_neigh.zip

Is this resolved in the development version? I don't want to update unless it is fixed.

> library(spdep)
Loading required package: sp
Loading required package: spData
Loading required package: sf
Linking to GEOS 3.10.0dev, GDAL 3.3.1, PROJ 8.1.0
> data(afcon, package="spData")
> p1_nb <- edit(paper.nb, cbind(afcon$x, afcon$y)) # see attached GAL file
> write.nb.gal(p1_nb, "no_neigh.gal")
> library(reticulate)
> use_python("/usr/bin/python", required = TRUE)
> ps <- import("libpysal")
> con <- ps$io$open("no_neigh.gal", "r")
> nb_q <- con$read(sparse=TRUE)
> esda <- import("esda")
> res <- esda$Moran_Local(afcon$totcon, nb_q)
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: cannot reshape array of size 0 into shape (0)

Detailed traceback:
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1032, in __init__
    self.p_sim, self.rlisas = _crand_plus(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 179, in crand
    larger, rlocals = compute_chunk(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 297, in compute_chunk
    rstats = stat_func(chunk_start + i, z, permuted_ids, weights_i, scaling)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1796, in _moran_local_crand
    zi, zrand = _prepare_univariate(i, z, permuted_ids, other_weights)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 532, in _prepare_univariate
    zrand = z_no_i[flat_permutation_ids].reshape(-1, cardinality)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=1L)
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: cannot reshape array of size 0 into shape (0)

Detailed traceback:
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1032, in __init__
    self.p_sim, self.rlisas = _crand_plus(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 179, in crand
    larger, rlocals = compute_chunk(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 297, in compute_chunk
    rstats = stat_func(chunk_start + i, z, permuted_ids, weights_i, scaling)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1796, in _moran_local_crand
    zi, zrand = _prepare_univariate(i, z, permuted_ids, other_weights)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 532, in _prepare_univariate
    zrand = z_no_i[flat_permutation_ids].reshape(-1, cardinality)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=1L, keep_simulations=FALSE)Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: cannot reshape array of size 0 into shape (0)

Detailed traceback:
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1032, in __init__
    self.p_sim, self.rlisas = _crand_plus(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 179, in crand
    larger, rlocals = compute_chunk(
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 297, in compute_chunk
    rstats = stat_func(chunk_start + i, z, permuted_ids, weights_i, scaling)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/moran.py", line 1796, in _moran_local_crand
    zi, zrand = _prepare_univariate(i, z, permuted_ids, other_weights)
  File "/home/rsb/.local/lib/python3.9/site-packages/esda/crand.py", line 532, in _prepare_univariate
    zrand = z_no_i[flat_permutation_ids].reshape(-1, cardinality)
> tf1 <- tempfile(fileext=".gal")
> write.nb.gal(paper.nb, tf1)
> con <- ps$io$open(tf1, "r")
> nb_q <- con$read(sparse=TRUE)
> res <- esda$Moran_Local(afcon$totcon, nb_q)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=1L)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=5L)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=5L, keep_simulations=TRUE)
> res <- esda$Moran_Local(afcon$totcon, nb_q, n_jobs=5L, keep_simulations=FALSE)

[ljwolf]

OK! Well, we'll take a look. This is odd, as it should've been resolved ☹️

[rsbivand]

That was how I understood the issue, but both with a larger data set and with this reprex, it still seems to be extant. Thanks for looking!

[Ethan-Harris0n]

Not sure if I should open a separate issue for this so posting here! I am also having an issue with islands in contiguity weights when attempting utilize G_Local function from the getis.ord module of esda. I am guessing we are having the same issue but any suggestions on a fix? Should i just revert to an older package version for now? cheers!

[ljwolf]

Yes, we haven't yet made a release with the fix, but I believe it should be propagated through to all local functions. I've just gone on sabbatical, so I should have the time to release by Feb.

For now, you may want to use the libpysal.weights.attach_islands function, or omit them from the local calculations.

[jGaboardi]

This is resolved in the current release.