ndstore icon indicating copy to clipboard operation
ndstore copied to clipboard

cubes that are all 0 appear to be stored in the db

Open alexbaden opened this issue 8 years ago • 1 comments

I was running the npz -> blosc conversion script and kept getting failures when trying to loop over the entire dataset.

After a bit of debugging, it appears that there are some cubes in the database that are all zeros...

I added a print statement for any cuboid that has isNotZeros() evaluate to false. It prints the x,y,z cube coordinates and the zindex.

Zero cube: 6, 0, 0 - 72 

I then checked the database for this cube, and found it existed in the DB (even though the statement isNotZeros() returned false).

mysql> select zindex from NR2B_9thA_res2 where zindex = 72;
+--------+
| zindex |
+--------+
|     72 |
+--------+
1 row in set (0.00 sec)

And, sure enough, if we multiply out the cuboid coordinates by the cube dimension (e.g. 384,0,0 and check the cutout, I get an error...

nd/sd/kristina15/NR2B_9thA/xy/2/384,440/0,100/1/

But, a cutout inside a cuboid that didn't return zeros works fine...

nd/sd/kristina15/NR2B_9thA/xy/2/100,200/0,100/1/

(note: this project on synaptomes has been fully converted already regardless of cube state, see below)

Here's a SQL dump of this table: https://www.dropbox.com/s/s5b8ookx6bq8sk2/NR2B_9thA_res2.sql?dl=0

The dataset is kristina15, channel is 16-bits.

I don't want to delete this zero cuboids until we've verified that there actually is no data in them. In the meantime, I changed the convert code to also convert 0 cuboids. That will at least let us look at the data and move forward with TRA related goals, and seems to be working. Was curious if there were any thoughts as to what is going on...

@randalburns @perlman

alexbaden avatar Sep 18 '16 17:09 alexbaden