Read Rockstar members

[cphyc]

PR Summary

This allow to read member data from Rockstar catalogues.

PR Checklist

• [x] Adds a test for any bugs fixed. Adds tests for new features.

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

@cphyc I think adjustments are needed to pass the new test

[brittonsmith]

I'm not sure where the conflicts came from, but the rest looked good to me. If they can be cleared up, I think this is ready to be merged.

[neutrinoceros]

@cphyc conflicts probably came from https://github.com/yt-project/yt/pull/4275 Happy to merge this once they're resolved

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

@yt-fido test this please

[neutrinoceros]

We have one failing test to deal with

[brittonsmith]

I cannot reproduce the failing test locally with a Python 3.10 install. Is there a way to get more information?

[cphyc]

~I think this might be real. I need to investigate!~

Weird; I have tried locally and cannot reproduce the error on 3.9, 3.10 or 3.11.

Here's the code I tried

import yt

ds = yt.load_sample("rockstar_halos/halos_0.0.bin")

for halo_id, Npart in zip(
    ds.r["halos", "particle_identifier"],
    ds.r["halos", "num_p"],
):
    print(halo_id)
    halo = ds.halo("halos", halo_id)
    assert halo is not None

    # Try accessing properties
    halo.position
    halo.velocity
    halo.mass

    # Make sure we can access the member particles
    assert len(halo.member_ids) == Npart

This should be the code of the failing test, unless I am mistaken.

[cphyc]

@yt-fido test this please.

[neutrinoceros]

@cphyc the test server is dead, see discussion on Slack.

[cphyc]

OK - the test still doesn't pass on CI … but it does locally. Any suggestion how to proceed here?

[matthewturk]

So for posterity, the backtrace is:

Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
  File "/home/fido/workspace/yt_py310_git/.venv/lib/python3.10/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/fido/workspace/yt_py310_git/yt/frontends/rockstar/tests/test_outputs.py", line 60, in test_halo_loading
    assert len(halo.member_ids) == Npart
AssertionError

Would it be possible to replace Npart with an explicit number? Or use one of the pytest assertions, so that we get a better sense of the difference? For instance, if we're expecting 10854390 and we get 4137657600, it'd be an endian thing. Y'know?

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[cphyc]

Well, I am officially puzzled.

[matthewturk]

256 is suspiciously reminiscent of a field delimiter. I would guess that there is some location where a read is happening of 8 bytes on one and 4 bytes on the other, possibly due to some oddity in the way that dtypes are determined. I would not have thought this possible given the architectures but I'm also puzzled and grasping at things.

Perhaps it would be good to catch the error, and then print out the results of tell or something, to make sure we're at the same place.

[neutrinoceros]

@yt-fido test this please

[matthewturk]

Here's the final stdout:

Reading 25 particles for halo 295.0 dimensionless. position=404536
Reading 25 particles for halo 296.0 dimensionless. position=404736
Reading 2427 particles for halo 0.0 dimensionless. position=78664

And here's the relevant part of the assertion:

assert_equal(len(halo.member_ids), Npart)

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 2171., units='dimensionless'
Max relative difference: 8.4805, units='dimensionless'
 x: array(2427)
 y: unyt_quantity(256., '(dimensionless)')

And then it fails with 2171 != 256. So it is getting a length of 2171 for the halo member ids, but it has read that NPart is 256.

NPart is read here:

            halos = np.fromfile(f, dtype=io._halo_dt, count=pcount)
            self._member_offset = f.tell()
            self._ids_halos = list(halos["particle_identifier"])
            self._Npart = halos["num_p"]

The location in the file for both is the same -- 78664 -- which leads me to believe that the problem isn't in the particle reading, but rather in the setting of NPart.

A few other comments about this -- 2427 is exactly 256 higher than 2171, and 2171 doesn't show up anywhere in the halo sizes. This suggests to me that there's some subtraction going on somewhere ... but maybe not.

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[neutrinoceros]

@yt-fido Test this please

[cphyc]

I made a bit of progress - it seems that the checksum of the data is not the same on fido as it is on my computer:

# locally
/home/…/rockstar_halos/halos_0.0.bin has sha256sum b'09accfb7ab6f162f29fcfb7c6393e207ba1be9d61cbdc230f17d39f0e90914d7'
/home/…/rockstar_halos/halos_0.1.bin has sha256sum b'42e64f974f7d72f201443d42463fd642f8e37c7c485eac54810e3be8adfd0a22'
# on yt-fido
/mnt/yt/rockstar_halos/halos_0.0.bin has sha256sum b'd4c395708e153ae9ec32873c1b1b13f52ee5e829b6cee34e2ab91c67fe8bdf20'
/mnt/yt/rockstar_halos/halos_0.1.bin has sha256sum b'42e64f974f7d72f201443d42463fd642f8e37c7c485eac54810e3be8adfd0a22'

I should note that the local data was downloaded through yt.load_sample, so it should reflect whatever is on https://yt-project.org/data/. @Xarthisius do you have an idea what's happening?