Acquisition (live and real) time is apparently not loaded from bcf file

[CorentinLG]

If I do : s=hs.load('zone 2.bcf', signal_type = "EDS_TEM") I get the following warning: "WARNING:hyperspy.io_plugins.bcf:spectrum have no dead time records..."

Then a number of parameters are imported from the .bcf file, but apparently not the realtime and live time of the acquisition. Ex below :

adf, eds = s eds.metadata

├── Acquisition_instrument │ └── TEM │ ├── Detector │ │ └── EDS │ │ ├── azimuth_angle = 45.0 │ │ ├── detector_type = SuperX │ │ ├── elevation_angle = 18.0 │ │ └── energy_resolution_MnKa = 130 │ ├── beam_energy = 300.0 │ ├── stage_x = None │ ├── stage_y = None │ └── tilt_stage = 25 ├── General │ ├── datetime = datetime.datetime(2016, 10, 21, 12, 44, 24) │ ├── original_filename = zone 2.bcf │ └── title = EDX ├── Sample │ ├── elements = ['O', 'Fe', 'Mg', 'Mn', 'C', 'Cu', 'Si'] │ ├── name = Map data │ └── xray_lines = ['O_Ka', 'Fe_Ka', 'Mg_Ka', 'Mn_Ka', 'C_Ka', 'Cu_Ka', 'Si_Ka'] └── Signal ├── binned = True └── signal_type = EDS_TEM

[francisco-dlp]

Are you sure that the information is stored in the file?

[CorentinLG]

I will double check asap, but 90% chance that it is stored.

2016-12-14 12:36 GMT+01:00 Francisco de la Peña [email protected]:

Are you sure that the information is stored in the file?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hyperspy/rosettasciio/issues/239, or mute the thread https://github.com/notifications/unsubscribe-auth/AWSJzkZQu9ZHqJe6bq2lXoyYumojBO3rks5rH9S9gaJpZM4LMzF2 .

[sem-geologist]

that information is not stored in bcf. I am sure 100%, actually that warning is annoying, maybe we should take it out completely. And put a line in the documentation instead informing that generally bcf does not save those so we cant get those parameters? Alternatively the warning could be broadcasted during import of bcf? There are also some other parameters which somebody could find handy and are not saved too. ... eghm... i.e. energy resolution, or FWHM... the last ones I guess bruker does not save as it have maybe some internal table with it for all it's detector models... I don't know. Dead time (and live and real too) however is some kind of mystery, why it is not saved in bcf only bruker can tell.. It is normally present in stand alone bruker spectras (which have very similar xml file structure)

[sem-geologist]

hyperspy/hyperspy#1357

[thomasaarholt]

On our NSS system, we have to export an accompanying .msa/.emsa file with our data in order to calibrate it in Hyperspy. Perhaps this is possible on Bruker systems as well?

s = hs.load(data.bcf)
ref = hs.load(spec.msa)
s.get_calibration_from(ref)

[CorentinLG]

I checked on the esprit version (1.9) that we have that the time is "somehow" present in the bcf file. First, when loaded, a spectra Y-axis can be read either as pulse/eV (total number of counts) or as cps/eV (counts per second).So the time info must be there somewhere.

Second, after digging a bit more, in the "hypermap" mode, clicking on the little "i" that most likely stands for information, opens a window containing among other things some interesting infos (the number of frame, the dwell time per pixel, and the "overall time"), which are not presently imported in HS. I agree that "overall time is different from live/real time, but at least it is something.

So I don't know how they are obtained - if total pulse and cps are saved independantly, "overall time" might be recalculated rather than simply measured and saved... How easily those infos can be accessed from the raw .bcf file is beyond my knowledge, but there might be something to do to get them...

2016-12-15 11:15 GMT+01:00 Thomas Aarholt [email protected]:

On our NSS system, we have to export an accompanying .msa/.emsa file with our data in order to calibrate it in Hyperspy. Perhaps this is possible on Bruker systems as well?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hyperspy/rosettasciio/issues/239, or mute the thread https://github.com/notifications/unsubscribe-auth/AWSJzl_Ma7FW8L32fftMj1wbQL7iNje9ks5rIRM5gaJpZM4LMzF2 .

[sem-geologist]

@thomasaarholt , you dont need to get spectra separately in bcf, it is included in, and hypermap spectra is scaled to that spectra at loading step (I mean offset and channel energy length). @CorentinLG , bcf is single file system firstly. it basically have two files (some helper pointer tables from version >=2) which is: (1) xml - typical bruker header file (typical, as it looks similar to theyr EDS, particle, project, jobs files) and (2) binary delphi packed array of eds hypermap. The images and all calibration information is saved in xml. It is a bit readable (as xml can be readable hardly by human and machine), These information you are telling is present in the xml part (pixel time, overall time). Frame count actually is saved as array with frame count for every line (yes you can start and stop mappings with granularity of line not pixel in esprit). I didn't knew where to put those things,as they are not exposed to hyperspy in other formats. @francisco-dlp what do you think? should I expose those information somehow to metadata tree?

[sem-geologist]

Iam sorry it is not pixel timepresent, but dwell time, the time spent on pixel during one frame.

[thomasaarholt]

@sem-geologist dumping anything interesting into s.original_metadata is what happens with the digital_micrograph format. I'd encourage this behaviour, as it means it's possible to get it out without delving into the source code for hyperspy.

The nice thing about the dm files is that all that metadata is in one big dictionary-type object, so I think it was relatively straightforward to copy it over. Not sure if that is the case with bcf.

If bcf contains the term "dwell time per pixel" somewhere, then that would be a very useful addition to have in the metadata.

[francisco-dlp]

I think that it'll definitely be a good idea to add that information to the imported metadata. If it is relevant for the analysis (i.e. if a HyperSpy function would use it for analysis) then we should find a place for it in metadata. Otherwise it could be placed in original_metadata.

[sem-geologist]

@francisco-dlp I think this is also quite relevant to this issue: bruker EDS xml saves also such information as window type, window layer composition (the element and the thiknes). I lately found that it also saves the detector layer information (silly hidden with zlib and base64 LOL). It also contains the pulse response functions (what is that? I have no idea), channel shifting factors. Would such kind of information be ever useful in hyperspy?

[CorentinLG]

In some cases, you may want to check the count per second obtained on a sample, and you would have to divide the total number by the overall time, so the info is precious and should probably be placed in metadata. Same goes for the dwell time per pixel and the total number of frame.

Btw, another great thing would be to be able to load each frame separately in order to be able to look at them and to decide if one should sum only the first half of them or the full pack for instance. In case a sample gets damaged under the beam, that would allow for instance to monitor its composition evolution as a function of time. Ultimately, the signal could be kept only for valid frames, eg before damage becomes too important. Would that be feasible?

2016-12-15 16:47 GMT+01:00 Francisco de la Peña [email protected]:

I think that it'll definitely be a good idea to add that information to the imported metadata. If it is relevant for the analysis (i.e. if a HyperSpy function would use it for analysis) then we should find a place for it in metadata. Otherwise it could be placed in original_metadata.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hyperspy/rosettasciio/issues/239, or mute the thread https://github.com/notifications/unsubscribe-auth/AWSJzrxNQMHJtpBVZ0mQcNcAwBKRC6Wwks5rIWETgaJpZM4LMzF2 .

[sem-geologist]

loading the frames separately is not possible, all the data are added after every frame to the same array (the binary packed delphi one I was talking about). Xml contains only counter of frames per line, it have no x-raycounts it only counts how many times the line were scanned. Sorry.

[francisco-dlp]

@sem-geologist, as a rule-of-thumb, every information that can be extracted from the file should go to original_metadata. The subset of this information that could be consumed by a function in HyperSpy should also be placed in metadata.

[sem-geologist]

ok, so I will get those attributes exposed, but I think I should not do any dead time live time estimation (the division of total time, counting count rates per pixel and other stuff), as that would extend loading time, and it is not needed every time. (I am concerned because we do thousands of bcf loading and processing in batch, and speed counts for me). BTW today during practical lectures with students I just noticed how slow maximum-pixel-spectrum calculation is on Brukers Esprit software, and how blazingly fast it goes in hyperspy. It looks like unbcf_fast loads data faster than Esprit natively which I think is quite a nice side effect :) (exactly non compressed same file (500Mb) Esprit on Intel Xenon with SSD: 15s, hyperspy on i7 on laptop from hdd 4s; compressed same file Esprit: 1:50 min!!!, laptop with hyperspy and hdd disk 5s. Looks Esprit uses very bad compression implementation, but even without it pascal/delphi lays flat dead...

[francisco-dlp]

That sounds awesome indeed.

[sem-geologist]

This is actually where hyperspy shines, we use it a lot for searching for minerals in some very huge piece of rocks (10x10cm i.e.) and we can iterate through thousands of bcfs in reasonable time and get the max_pixel_spectrum from whole rock identifying if there is i.e. Au, Ag, Pt 1um sized particles (ore geological applications), or search for some minerals for more theoretical geological applications. (REE, Zr, P, -> monazites, zircons for dating rocks, Xenotime-zircon for geothermobarometry and so on... those previously were taking ages to manually search for and Esprit stitching of image implementation had very severe limitations (memory and precision)).

[francisco-dlp]

I am glad to know that it has been this useful to you. We should have a section of the website devoted to usage cases. Feedback like this one is a motivation booster for the developers. We know that we're doing the right thing, but it is always good to have experimental evidence ;)

[CorentinLG]

Pursuing in the same ideas, what about the "tilt stage" item? It is present in the metadata, but apparently always at 0. Is it that Esprit does not import it from the microscope or that the load function could be implemented?

Having this is important to calculate xray path within the sample, and therefore the absorption correction for EDS quantification...

2016-12-15 19:18 GMT+01:00 Francisco de la Peña [email protected]:

I am glad to know that it has been this useful to you. We should have a section of the website devoted to usage cases. Feedback like this one is a motivation booster for the developers. We know that we're doing the right thing, but it is always good to have experimental evidence ;)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hyperspy/rosettasciio/issues/239, or mute the thread https://github.com/notifications/unsubscribe-auth/AWSJzh_h8Npyw6yc4KRFVdYt9JsrgJVqks5rIYRwgaJpZM4LMzF2 .

[thomasaarholt]

@CorentinLG, that's likely to be the microscope not speaking to the software on that label, although I could be wrong.

On our ARM, the equivalent is that the specified aperture does not update in the metadata.

[sem-geologist]

Bruker have different interfaces for different instruments, I dont know about other instruments, but on Zeiss microscopes it does not gather tilt information from microscope. Which is very logical: why? Because stage tilt can be completely different than sample tilt (i.e. stage can be tilted to compensate samples uneven shape), thats why Bruker does not read this value from microscope even if it could: it is up to user to set that before doing the eds. It is exposed to hyperspy so as is saved in the file.

[sem-geologist]

To add up to the above: The bruker produces also other kind of stuff to microscopes which works with they esprit softwarem one of such things is EBSD. To do EBSD normaly you need to have sample tilted 70 degree. An there is the problem: Esprit can't know if you mount the sample flat, and tilt the stage of 70 degrees toward EBSD detector, or you use special holder which mounts horizontaly on the stage but keeps the sample in 70 degree, so the microscope stage would output 0 tilt and if Esprit would use that - that would be wrong.

[sem-geologist]

It is told that 'give the man a fish and feed him a day, teach him how to fish....'

So in case that I would not overlook something there is very 'simple' way to get header xml from the bcf:

from hyperspy.io_plugins import bcf
from lxml import etree

sfs = bcf.SFS_reader('your_bcffile_path.bcf')  # here we get the single file system handler
# all file items in this singlefilesystem class instance is held inside
# dictionary hierarchy, we fetch the header:
header = sfs.vfs['EDSDatabase']['HeaderData']  
#the items in the tree have special method which allows to get selection as BytesIO object:
bstring = header.get_as_BytesIO_string()
#rewind:
bstring.seek(0)
#the final steps to print the xml to file for examination in text editor:
xml = etree.fromstring(bstring.read())
xml_root = xml.getroottree()
xml_root.write('header_output.xml', pretty_print=True)
# without pretty_print everything would be in the single line

thats it, in that way you can check if your microscope+ brukerEsprit configurationsaves the stuff which you expect it to be saved :)

[sem-geologist]

Ok, I was spending some time mapping the xml to python objects, then I thought that there should be easy and more elegant solution. I got into this: https://gist.github.com/walkermatt/f747c666b42e787d9371

It didn't worked very well, so I used my knowledge and did this:

import json
from ast import literal_eval

def interpret(string):
    try:
        return literal_eval(string)
    except (ValueError, SyntaxError):
        # SyntaxError due to:
        # literal_eval have problems with strings like this '8842_80' 
        return string

class ObjectifyJSONEncoder(json.JSONEncoder):
    """ JSON encoder that can handle simple lxml objectify types,
        Handles xml attributes, also returns all data types"""
    def default(self, o):
        dictionary = {}
        if hasattr(o, '__dict__') and len(o.__dict__) > 0:
            d1 = o.__dict__.copy()
            for k in d1.keys():
                if len(d1[k]) > 1:
                    d1[k] = [interpret(i.text) for i in d1[k]]
            dictionary.update(d1)
        if len(o.attrib) > 0:
            d2 = dict(o.attrib)
            for j in d2.keys():
                d2[j] = interpret(d2[j])
            dictionary.update(d2)
        if o.text is not None:
            if len(dictionary) > 0:
                dictionary.update({'value': o.pyval})
            else:
                return interpret(o.text)
        if len(dictionary) > 0:
            return dictionary

which basically makes it breeze to get the dict from the xml which will go to original_metadata

[sem-geologist]

It is done, now we get most of original metadata under original_metadata (In the pull request); The only glitch is the ResponseFunctions which is list of lists and they look weird in the printed tree... some more thingies left to add...

[sem-geologist]

@CorentinLG, My really great apologies, it looks I mistake Tilt angle with tilt, so that stage_tilt in metadata show the manually written tilt angle in the hardware window of Esprit. Stage tilt is normally recorded to bcf if the Esprit have appropriate license. I am going to fix that.

[sem-geologist]

@CoretinLG, You could be interested in my latest implementation in hyperspy/hyperspy#1357.

There is just few things to keep in mind, the Bruker's inconsistency between how things are called in info window of hypermap in Esprit:

Esprit DSP setting:

and xml (or mapped values and tags to original_metadata):

<ClassInstance Type="TRTDSPConfiguration">
      <ImageWidth>1024</ImageWidth>
      <ImageHeight>768</ImageHeight>
      <PixelAverage>16</PixelAverage>
      <LineAverage>4</LineAverage>
      <SEBitCount>12</SEBitCount>
      <ChannelCount>2</ChannelCount>
      <Channel0>1</Channel0>
      <ChannelName0>AsB</ChannelName0>
      <ChannelName1>SEI</ChannelName1>
      <CounterIndex>1</CounterIndex>
      <CounterChannelUsed>0</CounterChannelUsed>
      <TiltAngle>0</TiltAngle>
      <CounterMode>0</CounterMode>
      <PixelTime>1</PixelTime>
    </ClassInstance>

So from those above we can understand that PixelTime in the xml (and inthe original_metadata) is always 1 and it is smallest unit, which is unreachable on the most of machines as the minimum of dwell time in GUI or 'PixelAverage' in xml can be 8. if we multiply this with line average (which fortunately is consistent in GUI and xml) we get time spend [μs]/pixel during one scan. However mappings can have few scan frames. It is saved in BCF's xml part as list for line and is called 'LineCounter', so can be reached in the original_metadata with the same name. update: it occurs that 'PixelTime is used on some machines too.

So finally to calculate the total time per pixel in hypermap we need to:

pix_time = EDS.original_metadata['DSP Configuration']['PixelAverage'] *
 EDS.original_metadata['DSP Configuration']['LineAverage'] *
EDS.original_metadata['DSP Configuration']['PixelTime'] * 
 EDS.original_metadata['LineCounter'][number_of_line_where_the_pixel_is]

the total time of mapping can be calculated by: pix_time * heigh * width

[francisco-dlp]

Can we close this?

[askorikov]

@sem-geologist Although it might not be very useful to HyperSpy, could you still share your ideas on how live/dead time can be calculated from the information contained in .bcf file? I've been trying to figure it out for a while to implement in a custom script, but no success.

[sem-geologist]

@5ht2, it can't as neither live, neither dead times are saved anywhere. The information is lost as it is not saved into bcf. Unless Bruker will implement it to be saved (If you will see anytime in the future Esprit gui which for hypermap info would show dead time or live time, please share the file). If it is not shown in bruker's Esprit GUI, and is not saved in xml part of the file, we have no way to get such information. Pixel time is same as Dwell time, or Real Time, and that is this much what at the moment we can get.