QE65000 integration_time_micros does not block until complete in cseabreeze?

[alterscape]

spectrometer and system information

• model: QE65000
• operating system: Windows 10 64-bit
• python version: Python 3.8.6 64-bit
• python-seabreeze version: 1.3.0
• installed-via: pip setup.py install

current problem

It appears that, in this configuration with the QE65000 spectrometer, calls to spec.integration_time_micros() do not block until the instrument is ready to measure with the new integration time.

steps to reproduce

Setting integration_time_micros with our USB2000 (on a different machine -- Win7 with Python 3.8.6 32-bit) blocks until the spectrometer is in the new state, so the following code works correctly:

spec.integration_time_micros(1e5)
first_intensities = spec.intensities()
spec.integration_time_micros(1e6)
second_intensities = spec.intensities()

As expected, first_intensities and second_intensities differ (the peaks in second_intensities are higher because of longer exposure time, of course!)

However, on our QE65000, it appears that the spec.integration_time_micros() call does not block. The same code results in two identical spectra (less noise, of course).

However, if we add a delay like

spec.integration_time_micros(1e5)
sleep(1.0)
first_intensities = spec.intensities()
spec.integration_time_micros(1e6)
sleep(1.0)
second_intensities = spec.intensities()

the resulting intensities are as expected!

minimal code example and error (very helpful if available)

See above.

[ap--]

Hi @alterscape

interesting. Thanks for the detailed report! Which backend does this happen with? cseabreeze or pyseabreeze?

In the meantime waiting at least one integration time is probably a good workaround.

Cheers, Andreas 😃

[alterscape]

We're using cseabreeze in both cases. I had trouble getting pyseabreeze installed in general (probably if I slowed down and stepped through things carefully I could get it up and running, I just haven't yet, because cseabreeze worked out of the box).

It might be worth sorting out pyseabreeze and testing if the behaviour is any different.

[ap--]

The easiest way to get pyseabreeze running is via conda.

$> conda install -c conda-forge seabreeze
$> seabreeze_os_setup

On Linux and OSX it'll then work ootb. On Windows maybe too... (Alltough it could be that you have to change the driver... but maybe not)

[alterscape]

File this under "user error" and close, please. I'm a newbie to spectrometers and didn't understand all the subtleties of timing in different capture modes (free-running/default vs. hardware trigger, etc). I believe seabreeze is operating as expected/intended in the case I reported.

Is there an example available of configuring either a USB2000+ or a QEPro/QE65000 for edge triggering? The Ocean docs are less than illuminating. I have hardware triggering working in OceanView but can't seem to accomplish anything but locking up my Python code on spectrometer.integrate().

[ap--]

Hi @alterscape

no worries ☺️ And I really don't consider this a user error. If you can't get something to run that should be easy to do, then it's either broken or the documentation is bad.

Sadly, the trigger mode settings are not only very confusing, but also probably the least tested code paths. So here are the things we can do:

1. If you could tell me where you got stuck, and what would have helped you to get to the point where you are now, we could improve the documentation and make python-seabreeze better for other user.

2. I own a USB2000+ and can sketch up a minimal hardware trigger and test it myself. But I'm super busy these days, and can't guarantee that I will find time to do this in the next 3 weeks. But I'll see what I can do.

3. We could start collecting triggermode information for every model and add it to the docs.

Let me know if you'd like to help with 1, 3 or even 2 😃

Regarding the example

This should do the trick... I haven't tested this with a spectrometer, so there might be errors in there. Let me know if you find any.

from seabreeze.spectrometers import Spectrometer

spec = Spectrometer.from_first_available()
spec.integration_time_micros(10000)  # just for this example: 10 milliseconds

# now the following is a bit annoying...
# in the USB2000+ OEM data sheet (or at least the one i have)
# "External Hardware Edge Trigger Mode" is defined as 3 on the USB protocol and
# as 4 on the RS232 protocol
# But I do believe that this is incorrect and it's likely that 4 is correct.
spec.trigger_mode(4)


idx = 0
while True:
    # the call to spec.intensities() should now block and only return once the
    # spectrometer detects a rising edge at TTL levels on the ExtTrigIn pin 7
    spectrum = spec.intensities()
    print("recorded", idx, spectrum)
    idx += 1

If that doesn't work, you're either doing something wrong, or something is broken in python-seabreeze 😃

Cheers, Andreas

[alterscape]

Our setup is very similar to your code. Our test procedure was to take four rapidly-decreasing exposures [ie: 1s, 0.1s, 0.01s] while holding the signal constant, and check that the mean intensity diminished approximately proportionally.

We observed the following, referencing https://www.oceaninsight.com/globalassets/catalog-blocks-and-images/manuals--instruction-old-logo/electronic-accessories/external-triggering-options_firmware3.0andabove.pdf for modes:

Trigger Mode 0 seems to be free-running/software (as expected). If you change the exposure and immediately integrate, you have very good odds of getting the values from a previous (already-running) integration with the previous integration time. Calling intensities() twice seems to solve this, at the cost of an extra integration time. Trigger Mode 2 seems to correlate with the synchronous hardware trigger (one pulse starts integration, one pulse stops). We didn't read the docs on this in detail until after a bunch of testing, so, need to revisit and make sure we can consistently recreate the behavior we think we're seeing. Don't plan to use this, but it's useful to validate the inconsistent documentation. Trigger Mode 3 always block forever, in our testing. It's possible this is the actual edge trigger, and the flaw is in our multi-threaded trigger generation code. The intent is to call intensities, wait, and then fire the hardware trigger -- this is what I plan to test next, by applying a trigger externally (and then debugging our software's timing if that works). Trigger Mode 4 seems to be free-running/software, indistinguishable from 0. This was surprising, since I expected almost anything else.

More testing required, of course!

[ap--]

So, I quickly tried reproducing the trigger issue on my USB2000+ and could not reproduce the error... Identical to yours, my USB2000+ also seems to be running in free-running mode in trigger mode 4. But if I use trigger mode 3, calls to spec.intensity() are blocking, and a spectrum is only returned on the rising edge signal when I pull the signal on pin 7 high. (I bridged pin 1 VUSB with pin 7 ExtTrigIn with a piece of wire)

So I believe there are two possibilities why it does not work for you:

1. Your external trigger hardware is operating incorrectly
2. There is an issue with some specific firmwares of the USB2000+

To check (2) either lookup your FW version in OceanView, or run the code below:

>>> import seabreeze
>>> seabreeze.use("pyseabreeze")
>>> from seabreeze.spectrometers import Spectrometer
>>> from seabreeze.pyseabreeze.features.fpga import _FPGARegisterFeatureOOI
>>> spec = Spectrometer.from_first_available()
>>> fpga = _FPGARegisterFeatureOOI(spec._dev._transport.protocol)
>>> fpga.get_firmware_version()
(1, 0, 0)

To check (1) just do as I do and use a piece of wire to apply a rising edge signal manually to the trigger pin. If it works, it's your trigger hardware that needs investigating.

I hope that helps ❤️ Have a great weekend, Andreas

[alterscape]

Andreas, Thank you for your help! I was finally able to get back to this and can report that the problem was due to a misunderstanding of the implications of Python's GIL (I'm mostly a C++/C# developer, so, my intuition was not fully developed for Python threads)!

I had a single Python process manage the spectrometer and the device that provides the hardware trigger. My hope was to have one thread blocking for spec.intensities() and another thread sending the command that eventually leads to the hardware trigger being fired. However, since spec.intensities() is a blocking call and that thread then holds the GIL forever, no amount of working with Threads could save me from blocking forever once that call was made.

I was able to resolve the issue by starting a separate process for the seabreeze-server library, and writing a modified client that splits intensities() into two calls -- one that sends the request to the server and returns immediately, and a second that checks the connection for returned data. I had to add a sleep() call at various points in my code to give the seabreeze server time to do things. Longer-term, I'm going to look at cseabreeze/pyseabreeze and see if I can modify it to split the intensities() call into two steps:

1. Ask the spectrometer to start waiting for intensities, return something indicating that the call to the spectrometer has been made and it's now safe to trigger the integration.
2. Check for returned data, yield if not, return data if returned.

I might still need a separate process, but at least it could render the timing more deterministic, ie:

client -> server: Please start integrating / waiting for trigger. client <- server: I have started, the spectrometer is now in a state where it will collect data / accept the hw trigger! client: do other work, including hw trigger client -> server: Do you have data? client <- server: No data yet. client -> server: Do you have data? server <- client: Here is the data!

Does that seem reasonable to you?

[ap--]

Hi @alterscape

That's interesting... which backend have you tested this with? pyseabreeze or cseabreeze? It's unexpected that the GIL would cause a problem here, since pyseabreeze uses pyusb which internally wraps libusb using ctypes (and should release the GIL) and cseabreeze is a cython wrapper in which I explicitly release the GIL around the blocking call... But of course there's always the possibility that it's broken.

Can you provide a minimal code example that reproduces the threading issue for your USB2000+? Then I should be able to debug why it's not working.

Otherwise running it in a multiprocessing environment and sending data via a message queue as you describe seems like the best solution to your problem 👍

Cheers, Andreas

[islandmonkey]

However, on our QE65000, it appears that the spec.integration_time_micros() call does not block. The same code results in two identical spectra (less noise, of course).

We have the same issue with our Flame (USB4000). The spec.integration_time_micros() call does not block and unless we add a 1 second sleep in between changing integration times we don't get the correct spectra.

Using the pyseabreeze backend.