Copyright (c) 2013-2024, Nokia Solutions and Networks All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

=============================================================================== Event Machine on Open Data Plane - em-odp

em-odp is an implementation of the Event Machine (EM) framework on top of Open Data Plane (ODP). Many ODP implementations exist, both generic and HW or SoC optimized, and all should be suitable for em-odp to utilize and run upon. For development, e.g. odp for linux-generic could be most suitable, while a SoC optimized version would provide the best performance on a specific target. em-odp provides the EM API for the applications while ODP acts as a portability layer as well as providing APIs and services outside the scope of EM, e.g. odp pkt-io or crypto APIs.

=============================================================================== Compilation

First clone the ODP code (here using the ODP linux-generic version):

git clone https://github.com/OpenDataPlane/odp.git Go to the odp folder, create a build folder (optional) and configure, compile and install odp (optionally with a user specified installation folder): ./bootstrap mkdir build && cd build ../configure --prefix= If testing for performance, also give the --disable-abi-compat option to enable more inlining of code. Try also --disable-shared if a static lib is OK for you. (> ../configure --prefix= --disable-abi-compat
--disable-shared ...) Enabling Link Time Optimization (LTO) for EM, ODP and the application might give further performance benefits (--enable-lto). See further configure options: ../configure --help make && make install

Clone EM-ODP code:

git clone Here can be e.g. https://github.com/openeventmachine/em-odp.git. Go to the em-odp folder, optionally create a build folder, then configure, compile and install em-odp. Separate build folders are useful if you want to run em-odp with different odp installations or configurations. ./bootstrap mkdir build && cd build ../configure --prefix=
--with-odp-path= [--with-odp-lib=libodp-linux]
[--enable-esv --enable-check-level=3] ... Use the options --enable-check-level=3 --enable-esv to help catch bugs during development. Try --disable-shared also for EM if a static lib is OK for you. Enabling Link Time Optimization for EM, ODP and the application might give further performance benefits (--enable-lto). See further configure options: ../configure --help make && make install

Run some example application to test the functionality (path from build/), e.g.

./programs/example/fractal/fractal -c 0xe -t Optionally change ODP and/or EM run-time options by specifying your own runtime config files: ODP_CONFIG_FILE=my-odp.conf EM_CONFIG_FILE=my-em.conf
./programs/example/hello/hello -c 0xe -t

Stop by pressing Ctrl-C.

=============================================================================== ODP configuration changes for EM (em-odp)

The ODP configuration file lets the user tweak ODP settings. For odp-linux, some settings are better modified when using EM (em-odp), read about config file usage here: /config/README

Either change the default values in the config files and recompile or override the defaults by providing your own config files at startup:

ODP_CONFIG_FILE=my-odp.conf EM_CONFIG_FILE=my-em.conf
./programs/example/hello/hello -c 0xe -t

odp-linux default configuration file: /config/odp-linux-generic.conf Usually EM work better with the following config changes:

• Timer: Use inline timer implementation with EM. timer.inline = 0 -> 1

• Scheduler:

  • Priority level spread, the optimal value is typically the number of threads using the scheduler: sched_basic.prio_spread = 4 -> 'number of EM cores used'

  • Disable ODP automatically updated schedule groups, EM does not need them. (These options are actually DEPRECATED so prefer using the ODP API function odp_schedule_config() instead) sched_basic.group_enable: {all = 1 -> 0 worker = 1 -> 0 control = 1 -> 0}

  • Ordered queue reorder stash size. ODP might drop events if the stash/queue becomes full, causing EM ESV failure since event(s) dropped outside of EM can't be tracked, thus disable this. sched_basic.order_stash_size = 512 -> 0

• Stash: Disable strict stash size. Rely on stash 'put' return value to determine operation status. stash.strict_size = 1 -> 0

=============================================================================== CPU Architecture specific options

Arch specific compilation options should be passed to EM and ODP via configure. Example for a made up ARMv8 SoC called "soc-x": odp-soc-x/build> ../configure … --with-platform=soc-x [default=linux-generic] … em-odp/build> ../configure … CFLAGS=”-O3 -march=armv8.2-a+… -mcpu=soc-x”
--with-odp-path=
--with-odp-lib=libodp-soc-x … The whole chain of used SW libs/dependencies should preferably be compiled with the same -march/-mcpu options for best results: other-libs/build> ../configure … CFLAGS=”-O3 -march=armv8.2-a+… -mcpu=soc-x”

=============================================================================== Debug compilation

For debugging activities it may be useful to compile separate debug libs and executables of ODP and EM-ODP. The configuration script has to be run with different options, other commands as above.

ODP debug configuration:

../configure --prefix=
--enable-debug --enable-helper-debug CFLAGS='-O0 -g3' ... EM-ODP debug configuration: ../configure --prefix=
--with-odp-path=
--enable-check-level=3 --enable-esv --enable-debug-print CFLAGS='-O0 -g3' ...

=============================================================================== Packet-I/O (ODP-linux-generic + socket or dpdk based io)

The default ODP for linux-generic has pkt-io support based on linux networking and sockets that is quite slow but works nicely for development purposes. The linux-generic version of ODP also supports dpdk based packet io. Use dpdk based packet io on x86 targets for better performance. Alternatively, try XDP sockets (AF_XDP) that has been added to odp-linux. See odp-linux documentation for setup.

dpdk-io: DPDK can be used to accelerate packet io on x86 targets with the linux-generic version of ODP. This is different from the separate odp-dpdk implementation described below, but requires the same DPDK installation. DPDK can be installed via e.g. apt-get or yum or compiled from source code from https://www.dpdk.org/. See the supported dpdk versions from the odp documentation.

DPDK compilation from source code (example for dpdk v19.11):

cd dpdk make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc ... edit x86_64-native-linuxapp-gcc/.config to your liking ... make install T=x86_64-native-linuxapp-gcc DESTDIR=
EXTRA_CFLAGS="-fPIC" (fPIC optional) See ODP and DPDK instructions for more detailed info - e.g. uio vs vfio usage. Note that 'DESTDIR' is required when compiling to generate a similar installation as installing via apt-get would generate, use e.g. DESTDIR=./install After DPDK installation, configure and compile odp linux-generic: cd odp/build/ #assume odp/bootstrap already run ../configure --prefix= --with-dpdk-path= make && make install Recompile em-odp and make sure the --with-odp-path points to the odp with dpdk-pktio installed above, then start an example packet io application. Note that DPDK interfaces are accessed using only indexes and not "ethX" names. cd em-odp/build sudo ./programs/packet_io/loopback -c 0xe -t -i 0,1

=============================================================================== Using odp-dpdk (note: different from odp linux-generic + dpdk pkt-io)

There is also an ODP version that is optimized to be run on top of Intel DPDK

• this version is called odp-dpdk. Note that also odp for linux-generic can use dpdk for packet-io, but the optimized odp-dpdk utilizes dpdk internally for other functionality also.

DPDK instructions can be found from http://dpdk.org/doc/guides/

Get the odp-dpdk code:

git clone https://github.com/OpenDataPlane/odp-dpdk.git The odp-dpdk readme file (/platform/linux-dpdk/README) describes how to configure and compile dpdk and odp-dpdk on top of it. ODP compilations as above but different installation directories i.e. ../configure --prefix= ...

EM-ODP compilation with odp-dpdk:

mkdir build--odp-dpdk && cd build--odp-dpdk ../configure --with-odp-path=
--with-odp-lib=libodp-dpdk --prefix=...