artiq
artiq copied to clipboard
m-labs
DAC synchronization across Sayma cards
The code is there and somewhat works intermittently, but I cannot do anything until https://github.com/sinara-hw/sinara/issues/567 is resolved.
@gkasprow Can you expedite the rework, testing, and shipment of the replacement Sayma?
I'm solving #475 . It was caused by at least 3 independent factors. 2 were found and fixed, the last one I'm trying to identify but I think I'm very close.
The microtca mess is annoying but it's not blocking other people's developments and experiments, unlike this.
In email today with @marmeladapk and @hartytp, @sbourdeauducq said
Sayma v2 DAC synchronization doesn't quite work and I don't understand why.
By this do you mean synchronization between DAC chip on a single Sayma AMC v2? What debugging steps did you do?
DAC to FPGA. I cannot test between DACs since only one DAC is working on the board I have.
All Metlino tests (and tests involving several Sayma) were done when I was in Warsaw.
@sbourdeauducq We sent you 2 additional pairs a while ago. O.o
You confirmed reception ( https://github.com/sinara-hw/Sayma_AMC/issues/121#issuecomment-591333534 )
I received 2 Sayma pairs in total indeed, one without front panels and with only 1 DAC working (which is the one I am talking about here), and another one which isn't for M-Labs.
Glad you received the boards. Please look for DAC-DAC synchronization on the other two Sayma. When can you do this? It would help coordinate with Tom.
I'm not so worried about synchronisation between separate Sayma cards; we know DRTIO works well. I'd focus on DAC to FPGA synchronisation on a single Sayma for now.
@sbourdeauducq what's the plan/time-frame for looking at DAC->FPGA sync on a single board?
@marmeladapk and @sbourdeauducq Already showed basic SAWG + TTL sync.
https://github.com/sinara-hw/Sayma_AMC/issues/141#issuecomment-576285699
Based on this comment I had the impression that DAC -> FPGA doesn't work reliably yet.
@sbourdeauducq could you give a general summary of where we are with Sayma (or point me to one if it already exists). Does board bring up now work reliably for the DRTIO slave? Unsynchronised RF? BaseMod features (switch, attenuator)? Is that all decently well tested from kernels now? DAC -> FPGA synch?
Please look for DAC-DAC synchronization on the other two Sayma.
Again, this pair is not for M-Labs, and I don't have it anymore. If you will recall, we had independently ordered an additional set of Sayma boards. I've kept reminding Creotech to ship more hardware.
Does board bring up now work reliably for the DRTIO slave? Unsynchronised RF? BaseMod features (switch, attenuator)? Is that all decently well tested from kernels now?
All of this generally works. Sometimes (and rarely AFAICT) the DRTIO link appear to become corrupted with some bitstream builds (visible symptom is a storm of broken aux packets); making a minor/unrelated change to the gateware code appears to "fix" it. I have never seen this on Kasli so this is probably GTH/Ultrascale specific.
DAC -> FPGA synch?
There are unexplained/unpredictable/obscure bugs and I don't have a timeframe.
Thanks for the summary. Is there a write up of any of the symptoms you see? Anyway, it sounds like synchronisation between a single DAC and TTL is not reliable yet.
I've kept reminding Creotech to ship more hardware.
If you're not getting what you need out of Creotech or me or Tom or Xilinx to progress on your work make more noise. Let's get the hardware sorted so the synchronization testing can progress.
DAC -> FPGA synch?
There are unexplained/unpredictable/obscure bugs and I don't have a timeframe.
Do please get on with trying to reproduce these bugs and create Issues. Understood that you may not have a timeline for fixing bugs which you've not seen reproduce. What's your timeline for running tests on the hardware to try to reproduce the bugs?
Do please get on with trying to reproduce these bugs and create Issues. Understood that you may not have a timeline for fixing bugs which you've not seen reproduce. What's your timeline for running tests on the hardware to try to reproduce the bugs?
Do you mean me or @sbourdeauducq? Right now I don't have a good description of the bugs from @sbourdeauducq. Also, AFIACT @sbourdeauducq isn't having any trouble reproducing these issues, so I don't see that me reproducing them as well would help. I can do it but it will take a non-negligible amount of my time without contributing much clear value to the project.
AFAICT no one has done any real work/testing on Sayma for months now. It would be useful to understand what the issues are in more detail, who is going to work on them and when.
Right now I don't have a good description of the bugs from @sbourdeauducq.
Install the beta firmware (with synchronization) and reboot the board a few times while looking at the log. You'll see the sync errors, unless this is a problem with my board in particular.
Can you post a log?
Anyway, currently I do not know how the synchronisation process works in any details. It's been heavily rewritten/modified since I last used Sayma. As I don't think there are any docs, I don't expect that I would understand the log messages without investing a significant amount of time reverse engineering the process from the source code. I can do that, but it would be time consuming and I'm not clear that it would add any value to the project.
I'm not clear on the responsibilities or expected time commitments for Sayma. I naively assume that the plan here is for @sbourdeauducq to investigate the synchronisation issues on hw he already has. But please correct me if that's not the plan/not possible.
There are unexplained/unpredictable/obscure bugs and I don't have a timeframe. Do please get on with trying to reproduce these bugs and create Issues.
This was in reply to @sbourdeauducq. I'd like to know more about the "unexplained/unpredictable/obscure bugs."
we know DRTIO works well.
We know that it works well:
- on Kasli.
- with the old siphaser alignment algorithm, which has since been replaced (cc58318500ecfa537abf24127f2c22e8fe66e0f8) but not tested as thoroughly as the first version.
@hartytp If you want to help, could you validate, with the latest code, that you get reproducible RTIO clock phases (and to what tolerance?) between AMC and Kasli, and between AMC and RTM? You can check the outputs of the three Si5324s after the message "INFO(board_artiq::si5324::siphaser): calibration successful" on the satellites. If that doesn't work, this should be fixed before further synchronization attempts.