libiio
libiio copied to clipboard
analogdevicesinc
Add new iio_block based API and IIOD async. protocol
This PR contains my latest work on the new Libiio 1.x API.
The API changed quite a bit, to be more versatile, to map better to the latest IIO kernel APIs, and to enable more performance. Along with the API change, the communication protocol between libiio and IIOD changed; it will now optionally use an asynchronous protocol (if supported by both parties) to boost the throughput.
For instance, running iio_readdev (iio_rwdev on the "dev" branch) with blocks of 16 KiB each, over a direct network cable between my PC and a ZCU102 equipped with a FMCOMMS-3: new libiio + new iiod: 81 MiB/s sustained Any other configuration: 44 MiB/s sustained
The diff is huge so take your time to review.
@catkira It depends what modifications you need. If your timestamps appear as a separate streamable channel, then it should work with latest master without changes.
@catkira probably not a good idea to base your branch off this one, because I'm not done working on it and I will force-push some changes from time to time. Maybe a better idea would be to base your branch off the "dev" branch instead, which is where all new features are merged, and will never be force-pushed to. Just keep in mind that the API in the "dev" branch won't be stable until it's eventually merged into master.
@catkira The "dev" branch is basically "libiio2" (which will actually be libiio 1.x, since libiio1 was 0.x). It will be merged into master right after we release libiio v0.24, which will most likely be the last one of the 0.x series. Then the development will continue in the master branch until we are happy enough with the code to make a libiio 1.x release.
I don't think we will change anything in libiio 1.x regarding timestamps. They will simply appear as an extra channel, that you can enable (or not).
@catkira Right now what's implemented and works, is timestamp reporting at the sample level, if your HDL and kernel driver support that. Then the timestamp values appear as a separate channel.
What @tfcollins talks about in the link you pasted is supporting timestamps at the buffer level, where there is one timestamp value for a whole block of samples, which represents (for instance) the time at which the very first sample was read.
This kind of meta-data is not supported yet, and it is unclear if we want to add ways to support it. If we have meta-data embedded into the samples stream, then we can still transport it all the way to the application, which then must know that the first few bytes of each buffer is the timestamp, maybe with an extra kernel attribute that specifies that there is a timestamp for this buffer. This shifts the responsability of handling the timestamp to the application, which isn't ideal.
If we don't want to taint the samples stream with meta-data, then maybe the kernel could also somehow update a "timestamp" buffer attribute each time a block of samples is dequeued. This could be handled by libiio as a special attribute, and you'd get something like a iio_buffer_get_timestamp() call to retrieve it. But then, that means your HDL must write the timestamp value outside the samples buffer, and the kernel driver must retrieve the value to update the "timestamp" attribute.
We tend to see this problem as part of a bigger one: how should meta-data (in general) generated by the HDL or hardware be handled by the kernel, libiio, and the application? We don't really have an answer for that at the moment.
@catkira so I didn't profile it yet to see exactly what is the new bottleneck, but I'm pretty sure it's the CPU. The data is copied by the CPU from the IIO buffer to the network socket yes, and it's the only time it's copied. At 81 MiB/s the CPU usage was very high.
@catkira well I'm not testing on a Pluto ;) But on a ZCU706 with a FMCOMMS-3 board.
@catkira well I'm not testing on a Pluto ;) But on a ZCU706 with a FMCOMMS-3 board.
Ah that makes sense. My AntSDR should arrive soon, then I can test over its GBit ethnernet interface. Can You share Your benchmark script? Or do You just stream to file for a while and then divide size/duration?
The iio_buffer now represents a hardware IIO buffer. Generally you can create only one per device, but some IIO devices can have more than one. The iio_buffer has enable/disable functions to start/stop the DMA operation. Your application is now responsible for creating a few iio_blocks, which are dequeued by default, and can be enqueued to upload samples, or to request samples from the hardware. A iio_block basically just represents a block of samples.
The iio_stream object is something else, it is implemented on top of the iio_buffer and iio_block objects. It is totally optional, and is only there to simplify the process for simple programs that don't want/need to manually allocate the blocks. You just create the iio_stream object, and call iio_stream_get_next_block() in a loop - the old block will be enqueued, and it will return a pointer to a newly dequeued block.
I think we should discuss the timestamp feature somewhere else, feel free to open an issue about it.
I think we should discuss the timestamp feature somewhere else, feel free to open an issue about it.
yes, that makes sense. Its a bit too much spam here. I delete my posts here and open another issue when I have a working prototype.
@catkira you can post them to another issue now, then we can discuss it there. I can give some feedback but didn't want to pollute this PR further :)
I did a little benchmarking with ANTSDR (A pluto clone with gigabit ethernet and slightly larger FPGA). I can get about 12-14 MSPS single channel (tested with SigDigger and SoapyPlutoSDR). While I was doing that, 'top' on ANTSDR showed 50% CPU usage. Since the 50% is an average value, I guess there a phases with 100% too, so I guess CPU power is the limiting factor here. The reason why this libiio is faster, is probably because it is doing stuff more efficiently. I would like to try out this branch to see how many MSPS I can get. @pcercuei Is there a way You recommend how to try out this branch? Like patch buildroot and create a new image, or just scp the new libiio onto the SDR after bootup? Does this libiio branch also need a new kernel or updated drivers? For the PC its easy, I can just compile this branch I guess.
@catkira I believe for testing you can just SCP the updated libiio + IIOD, yes; then kill the running instance of iiod and run your own. But since it's a bit cumbersome to run it manually every time you restart the device, updating the buildroot package is a good idea as well.
The updated libiio does not need new kernel drivers.
What @tfcollins talks about in the link you pasted is supporting timestamps at the buffer level, where there is one timestamp value for a whole block of samples, which represents (for instance) the time at which the very first sample was read.
This kind of meta-data is not supported yet, and it is unclear if we want to add ways to support it. I
I think we absolutely need to support this type of support, There are too many other systems that build on top of IIO (or could) that require things like this. How are TDD systems managed otherwise? (Send this buffer at sample count=X is just as important as the Rx sample counting).
@rgetz yes, what I meant in my message was that it's unclear if we want to add support for this in libiio + kernel IIO, instead of just treating the metadata as sample data and let the IP blocks decypher. The major fear I have with supporting it in IIO is to come up with a system that's too "rigid" to support anything else that might come up in the future...
should we continue to discuss this in the timestamp issue?
Treating meta data as samples in libiio and kernel iio+dmac would mean that the user code would have to know how the hdl core expects meta data and that meta data would then be mandatory even if the user code just wants to do streaming. Therefore I think from a user perspective, it would be nicer to have a dedicated timestamp API. But that could also be done as another layer on top of (or inside) libiio so that all the timestamp stuff is transparent for libiio and kernel iio+dmac.
Yeah, I understand @pcercuei 'S concern - we don't know what we don't know yet. But I think there are lots of open implementations that would use/need/do similar things. We should be able to learn from others systems.
Thinking:
- Vita 49
- UHD
- open LTE stacks
- others
Most of those are on the high speed / SDR; but I think there are similar sample stamping needs on IMUs, and precision side too.
Some (high speed interface) may need to have some sort of hardware assist (HDL), where the standard SPI like interfaces might be able to get by with a pure SW implementation.
- Robin
When compiling, zlibstd is not found. I have written the details in this issue https://github.com/analogdevicesinc/libiio/issues/867#issuecomment-1204220359
Edit: its fixed now, I had to add zstd as dependency to the libiio package in buildroot -> https://github.com/catkira/buildroot/commit/d1c0130beef58096ff00343cfce5c8d4a6b228ec
@pcercuei which linux (repo and branch) should I use to try out the new dmabuf und mmap stuff of this PR?
I have tried to scp the cross compiled libiio to pluto. But when I start iiod I get an error message:
# /etc/init.d/S23udc start
Starting UDC Gadgets: sh: write error: No such device
FAIL
I have copied the files using this script:
#~/bin/sh
# This script copies libiio and iiod to pluto and restarts iiod
#
#default IP address
ipaddr=192.168.2.1
if [ "$#" -eq 1 ]; then
ipaddr="$1"
fi
echo "using ip address $ipaddr"
if [ ! -f ./build/libiio.so ]; then
echo no file to upload
exit
fi
ssh_cmd()
{
sshpass -v -panalog ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oCheckHostIP=no root@${ipaddr} "$1" 2>/dev/null &
wait
if [ "$?" -ne "0" ]; then
echo ssh command \"$1\" failed
exit
fi
}
scp_cmd()
{
sshpass -v -panalog scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oCheckHostIP=no "$1" root@${ipaddr}:"$2" 2>/dev/null &
wait
if [ "$?" -ne "0" ]; then
echo scp command copy $1 tp $2 failed
exit
fi
}
ssh_cmd "/etc/init.d/S23udc stop"
ssh_cmd "mkdir /root/libiio_orig"
echo "copy libiio.so"
ssh_cmd "mv /usr/lib/libiio* /root/libiio_orig"
scp_cmd "./build/libiio.so" "/usr/lib/"
echo "copy iiod"
ssh_cmd "mv /usr/sbin/iiod /root/libiio_orig"
scp_cmd "./build/iiod/iiod" "/usr/sbin/"
ssh_cmd "/etc/init.d/S23udc start"
on the build system libiio.so is a symlink to libiio.so.1 which is a symlink to libiio.so.1.0. But when I scp libiio.so it automatically follows the symlink and copies libiio.so.1.0. Thats why I only copy one lib. It should be ok right? @pcercuei Do I need to update more files? maybe some config files?
Ah i see the problem, libiio.so.1 is imported by iiod
benja@DESKTOP:/mnt/d/git/libiio_cross/build/iiod$ readelf -d iiod
Dynamic section at offset 0xfc04 contains 35 entries:
Tag Type Name/Value
0x00000001 (NEEDED) Shared library: [libiio.so.1]
I changed the script to look like this
#~/bin/sh
# This script copies libiio and iiod to pluto and restarts iiod
#
#default IP address
ipaddr=192.168.2.1
if [ "$#" -eq 1 ]; then
ipaddr="$1"
fi
echo "using ip address $ipaddr"
if [ ! -f ./build/libiio.so ]; then
echo no file to upload
exit
fi
ssh_cmd()
{
sshpass -v -panalog ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oCheckHostIP=no root@${ipaddr} "$1" 2>/dev/null &
wait
if [ "$?" -ne "0" ]; then
echo ssh command \"$1\" failed
exit
fi
}
scp_cmd()
{
sshpass -v -panalog scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oCheckHostIP=no "$1" root@${ipaddr}:"$2" 2>/dev/null &
wait
if [ "$?" -ne "0" ]; then
echo scp command copy $1 tp $2 failed
exit
fi
}
ssh_cmd "/etc/init.d/S23udc stop"
ssh_cmd "mkdir /root/libiio_orig"
cd build && tar -cvf libiio.tar libiio.so* && cd ..
echo "copy libiio.tar"
ssh_cmd "mv /usr/lib/libiio* /root/libiio_orig"
scp_cmd "build/libiio.tar" "/usr/lib/"
ssh_cmd "tar -xvf /usr/lib/libiio.tar -C /usr/lib"
echo "copy iiod"
ssh_cmd "mv /usr/sbin/iiod /root/libiio_orig"
scp_cmd "./build/iiod/iiod" "/usr/sbin/"
ssh_cmd "/etc/init.d/S23udc start"
but the problem is also that libc is not compatible, because I cross compiled libiio with vitis 2022.1 :/ Seems like I have to create a complete new image.
@pcercuei When I try to compile SoapyPlutoSDR with this new libiio I get some errors (I already included iio-compat.h in SoapyPlutoSDR and I also added iio-compat.h to the list of installed headers in CMakefiles.txt, see this commit https://github.com/catkira/libiio/commit/83a5f5d3c491f67b1a76bbcd49ce4e882fdcedaa)
benja@DESKTOP:/mnt/d/git/SoapyPlutoSDR/build$ make
Consolidate compiler generated dependencies of target PlutoSDRSupport
make[2]: Warning: File 'CMakeFiles/PlutoSDRSupport.dir/compiler_depend.make' has modification time 0.23 s in the future
[ 20%] Building CXX object CMakeFiles/PlutoSDRSupport.dir/PlutoSDR_Registration.cpp.o
[ 40%] Building CXX object CMakeFiles/PlutoSDRSupport.dir/PlutoSDR_Settings.cpp.o
/mnt/d/git/SoapyPlutoSDR/PlutoSDR_Settings.cpp:157:59: error: macro "iio_channel_attr_read" passed 4 arguments, but takes just 3
157 | if (iio_channel_attr_read(chn, "input", buf, sizeof(buf)) > 0) {
| ^
In file included from /usr/include/iio/iio-compat.h:16,
from /mnt/d/git/SoapyPlutoSDR/SoapyPlutoSDR.hpp:2,
from /mnt/d/git/SoapyPlutoSDR/PlutoSDR_Settings.cpp:1:
/usr/include/iio/iio.h:1029: note: macro "iio_channel_attr_read" defined here
I think the problem is this part
__api __check_ret ssize_t
iio_channel_attr_read_raw(const struct iio_channel *chn,
const char *attr, char *dst, size_t len);
/** @brief Read the content of the given channel-specific attribute
* @param chn A pointer to an iio_channel structure
* @param attr A NULL-terminated string corresponding to the name of the
* attribute
* @param ptr A pointer to the variable where the value should be stored
* @return On success, 0 is returned
* @return On error, a negative errno code is returned */
#define iio_channel_attr_read(dev, attr, ptr) \
_Generic((ptr), \
bool *: iio_channel_attr_read_bool, \
long long *: iio_channel_attr_read_longlong, \
double *: iio_channel_attr_read_double)(dev, attr, ptr)
with this define it is not possible to add a iio_channel_attr_read function with 4 arguments for compatibility. It is probably better not to use define like this in a header file. I think the comment for the function is also not correct, because 1st argument is not a channel.
When I try ad9361-iiostream, I get this error:
# ./ad9361-iiostream
* Acquiring IIO context
* Acquiring AD9361 streaming devices
* Configuring AD9361 for streaming
* Acquiring AD9361 phy channel 0
* Acquiring AD9361 RX lo channel
* Acquiring AD9361 phy channel 0
* Acquiring AD9361 TX lo channel
* Initializing AD9361 IIO streaming channels
* Enabling IIO streaming channels
* Creating non-cyclic IIO buffers with 1 MiS
* Starting IO streaming (press CTRL+C to cancel)
Segmentation fault
Its on a pluto clone (ant sdr). I had to outcomment a line in ad9361-iiostream., because of this error:
# ./ad9361-iiostream
* Acquiring IIO context
* Acquiring AD9361 streaming devices
* Configuring AD9361 for streaming
* Acquiring AD9361 phy channel 0
Error -22 writing to channel "rf_port_select"
value may not be supported.
* Destroying streams
* Destroying buffers
* Destroying channel masks
* Destroying context
# unloading
bool cfg_ad9361_streaming_ch(struct stream_cfg *cfg, enum iodev type, int chid)
{
struct iio_channel *chn = NULL;
// Configure phy and lo channels
printf("* Acquiring AD9361 phy channel %d\n", chid);
if (!get_phy_chan(type, chid, &chn)) { return false; }
//wr_ch_str(chn, "rf_port_select", cfg->rfport);
wr_ch_lli(chn, "rf_bandwidth", cfg->bw_hz);
wr_ch_lli(chn, "sampling_frequency", cfg->fs_hz);
// Configure LO channel
printf("* Acquiring AD9361 %s lo channel\n", type == TX ? "TX" : "RX");
if (!get_lo_chan(type, &chn)) { return false; }
wr_ch_lli(chn, "frequency", cfg->lo_hz);
return true;
}
@pcercuei there were many more things missing in iio-compat.h, so I gave up on that ported SoapyPlutoSDR to libiio 1.0.
Is it planned that iio-compat.h provides full 0.x compatibility?
@catkira the DMABUF kernel code can be found here: https://lore.kernel.org/linux-doc/[email protected]/T/
Your SoapyPlutoSDR needs to be ported to Libiio 1.0, you can't expect it to compile out of the box. The compat library is lagging behind and needs a good overhaul. Some functions changed (e.g. iio_device_create_buffer) so it's not really possible to just link the compat lib with libiio 1.x, instead it will need to dynamically load the symbols (dlopen / dlsym), and that's TODO.
About your ad9361-iiostream... Well, the Pluto does not have a AD9361 ;)
@pcercuei I already ported SoapyPlutoSDR to libiio 1.0. (At least the rx stuff). It seems to kind of work, but I have to further evaluate it.
https://github.com/catkira/SoapyPlutoSDR/tree/libiio_1.0
One thing I was wondering about was, if there is a way to get the lib and backend version in libiio 1.0?
In 0.x there was iio_context_get_version and iio_library_get_version, in 1.0 I only found iio_context_get_version_minor.
I have modified my pluto to have a AD9361 by chaning some variables with fw_setenv. My pluto clone, the antsdr, has a real ad9361. So both should appear to libiio as a AD9361. (all AD936x have the same silicon AFAIK and are just possibly selected by performance)
@catkira lib and backend versions can be obtained with iio_context_get_version_*, to get the library version, just pass ctx=NULL.