Add C/Fortran Ordering Option to Raw Data Importer

[psavery]

Currently, the raw data importer always assumes that the data is Fortran ordering, and the user has to manually transpose the data to Fortran ordering if it is actually C ordering instead.

We should add an option to the raw data importer so that the user can specify the ordering of the raw data file. If it is Fortran, we will do nothing, but if it is C ordering, we will convert it to Fortran ordering before putting it into VTK. We can make the most common option the default.

Numpy, for instance, always writes out raw data (with the tofile() function) in C ordering. Perhaps C ordering should be the default.

[ercius]

I vote for C ordering

On Tue, Nov 12, 2019, 8:07 AM Patrick Avery [email protected] wrote:

Currently, the raw data importer always assumes that the data is Fortran ordering, and the user has to manually transpose the data to Fortran ordering if it is actually C ordering instead.

We should add an option to the raw data importer so that the user can specify the ordering of the raw data file. If it is Fortran, we will do nothing, but if it is C ordering, we will convert it to Fortran ordering before putting it into VTK. We can make the most common option the default.

Numpy, for instance, always writes out raw data (with the tofile() function) in C ordering. Perhaps C ordering should be the default.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenChemistry/tomviz/issues/2001?email_source=notifications&email_token=ADXORFHI6WA7B5USNOFN5N3QTLIC3A5CNFSM4JMFUGJ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HYXVLAQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXORFAV4OGJME7RRQWF4QTQTLIC3ANCNFSM4JMFUGJQ .

[cryos]

Has anyone actually hit an issue loading a raw data file? I think the ones I have been given have just worked, and I would assume they read from C order already. We could do with some real files that have issues before writing any code to add reordering.

[cryos]

In fact, why not just let people use the operator in the rare cases where this might be an issue? I think we need some examples of real world issues before we consider this a bug that we want to invest resources in fixing.

[psavery]

So, the big case we know of here that requires a conversion to Fortran ordering after loading are raw files written with numpy. Numpy always writes out raw data files using C ordering, and after loading them into tomviz, we have to convert the data to Fortran ordering before using it.

If the raw file is written in Fortran ordering, it won't require a conversion. I think matlab writes out raw files in Fortran ordering.

[cryos]

Now we support reading in the numpy binary format I would expect most people use that, and we should probably advise them to do that. We also have the transpose operator, and people dealing with raw data can use that. I guess my main point is that we haven't seen any bug reports.

[ercius]

I was the one who brought this up originally. It was connected to issue 1990 and Patrick and I corresponded over email. That lead to issue 2001.

Isn't it true that the raw data importer will be basically useless if you don't support conversion from C to Fortran? Almost all files written in raw format will be C ordering. Maybe I missed something?

On Tue, Dec 10, 2019, 11:20 AM Marcus D. Hanwell [email protected] wrote:

Now we support reading in the numpy binary format I would expect most people use that, and we should probably advise them to do that. We also have the transpose operator, and people dealing with raw data can use that. I guess my main point is that we haven't seen any bug reports.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenChemistry/tomviz/issues/2001?email_source=notifications&email_token=ADXORFDC4MIZFYNYY7Z54V3QX7TWVA5CNFSM4JMFUGJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQP7LI#issuecomment-564199341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXORFGKMSHT5ETZNJBYL7TQX7TWVANCNFSM4JMFUGJQ .

[cryos]

This is why it is better to keep some of these discussions on issues so we don't miss out on the context :-) We should take a look at this, I know Rob and I did quite a bit of testing and never hit issues, I will try to take a look at the raw reader code to see what it is doing.

[ercius]

True! Its best to keep these issues in one place. The issue was discovered during a separate discussion which then branched out from there.

I think we need to start from scratch here. I was not able to test this issue when it was created since tomviz 7.0 crashes when loading raw data. I was able to test with the nightly though.

Raw data written from numpy is not loaded properly. You need a file which can show an image to test this when you write out from numpy. If I save a data set using this code:

tt = np.load('testOBJprojTomviz.npy') # shape: (256,256,91); dtype: float32
with open('./Desktop/temp.raw','wb') as f1:
    tt.tofile(f1)

Then I load the data with the raw viewer using the shape exactly as above. The data comes out messed up. So, loading C-ordering data with the raw viewer does not work as expected.

Data transforms -- Transpose -- Fortran ordering fixes the issue.