SITL not working on alpine for Copter 4.5 (still)

[Zavitit]

Bug report

As was mentioned in #27227 for some reason building SITL on alpine failed with the error in that issue

@peterbarker fixed the issue that was stated there but SITL still won't build properly I've added a comment there (that ill quote here at the end) with the details of the docker I'm using and the error I've encountered I'd really appreciate help with this problem

Version Copter 4.5

Platform [ ] All [ ] AntennaTracker [X ] Copter [ ] Plane [ ] Rover [ ] Submarine

Airframe type cotacopter

Hardware type Cube

Logs adding the comment from the previous issue:

@peterbarker Thank you very much for the help but unfortunately I'm still having this issue

I'm adding my Dockerfile maybe I'm just doing something wrong?

FROM alpine:3.19 AS builder

WORKDIR /ardupilot

RUN apk update && apk add --no-cache \
        linux-headers \
        g++ \
        python3 \
        py-future \
        py-pip \
        libxml2-dev \
        libxslt-dev \
        git \
    && ln -sf python3 /usr/bin/python \
    &&  rm -rf /var/cache/apk/*

RUN python3 -m pip install --user --no-deps --no-cache-dir empy==3.3.4 pexpect ptyprocess --break-system-packages

RUN git clone https://github.com/ardupilot/ardupilot.git --depth 1 --single-branch src \
    && cd src \
    && git submodule update --init --recursive --depth 1

RUN cd /ardupilot/src && ./waf configure --board sitl --no-submodule-update \
    && ./waf copter
# ------ fails during ./waf copter------

ENTRYPOINT ["sh"]

i did some manual experimentation and just rerunning the ./waf copter command resulted in around 5 targets that won't compile output for the last run:

Waf: Entering directory `/ardupilot/src/build/sitl'
Embedding file locations.txt:Tools/autotest/locations.txt
Embedding file models/Callisto.json:Tools/autotest/models/Callisto.json
Embedding file models/freestyle.json:Tools/autotest/models/freestyle.json
Embedding file models/plane-3d.parm:Tools/autotest/models/plane-3d.parm
Embedding file models/plane.parm:Tools/autotest/models/plane.parm
Embedding file models/xplane_heli.json:Tools/autotest/models/xplane_heli.json
Embedding file models/xplane_plane.json:Tools/autotest/models/xplane_plane.json
Generating compile_commands.json
Build commands will be stored in build/sitl/compile_commands.json
Generating compile_commands.json
Build commands will be stored in build/sitl/compile_commands.json
[ 802/1331] Compiling libraries/SITL/SIM_VectorNav.cpp
[ 843/1331] Compiling libraries/SITL/SIM_MicroStrain7.cpp
[ 844/1331] Compiling libraries/SITL/SIM_MicroStrain.cpp
[ 873/1331] Compiling libraries/SITL/SIM_GPS_NMEA.cpp
[ 884/1331] Compiling libraries/SITL/SIM_GPS_MSP.cpp
[ 889/1331] Compiling libraries/SITL/SIM_GPS.cpp
[ 893/1331] Compiling libraries/SITL/SIM_FlightAxis.cpp
[1073/1331] Compiling libraries/AP_HAL_SITL/SITL_cmdline.cpp
../../libraries/SITL/SIM_GPS_NMEA.cpp: In member function 'virtual void SITL::GPS_NMEA::publish(const SITL::GPS_Data*)':
../../libraries/SITL/SIM_GPS_NMEA.cpp:39:20: error: aggregate 'SITL::timeval tv' has incomplete type and cannot be defined
   39 |     struct timeval tv;
      |                    ^~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_MicroStrain.cpp: In static member function 'static void SITL::MicroStrain::simulation_timeval(SITL::timeval*)':
../../libraries/SITL/SIM_MicroStrain.cpp:44:27: error: aggregate 'SITL::timeval first_tv' has incomplete type and cannot be defined
   44 |     static struct timeval first_tv;
      |                           ^~~~~~~~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_GPS_MSP.cpp: In member function 'virtual void SITL::GPS_MSP::publish(const SITL::GPS_Data*)':
../../libraries/SITL/SIM_GPS_MSP.cpp:57:20: error: aggregate 'SITL::timeval tv' has incomplete type and cannot be defined
   57 |     struct timeval tv;
      |                    ^~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_GPS.cpp: In static member function 'static void SITL::GPS_Backend::simulation_timeval(SITL::timeval*)':
../../libraries/SITL/SIM_GPS.cpp:122:27: error: aggregate 'SITL::timeval first_tv' has incomplete type and cannot be defined
  122 |     static struct timeval first_tv;
      |                           ^~~~~~~~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_FlightAxis.cpp: In function 'double timestamp_sec()':
../../libraries/SITL/SIM_FlightAxis.cpp:103:12: error: reference to 'timeval' is ambiguous
  103 |     struct timeval tval;
      |            ^~~~~~~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_MicroStrain7.cpp: In member function 'virtual void SITL::MicroStrain7::send_gnss_packet()':
../../libraries/SITL/SIM_MicroStrain7.cpp:39:24: error: aggregate 'SITL::timeval tv' has incomplete type and cannot be defined
   39 |         struct timeval tv;
      |                        ^~
compilation terminated due to -Wfatal-errors.

../../libraries/SITL/SIM_VectorNav.cpp: In function 'void simulation_timeval(SITL::timeval*)':
../../libraries/SITL/SIM_VectorNav.cpp:71:27: error: aggregate 'SITL::timeval first_tv' has incomplete type and cannot be defined
   71 |     static struct timeval first_tv;
      |                           ^~~~~~~~
compilation terminated due to -Wfatal-errors.

../../libraries/AP_HAL_SITL/SITL_cmdline.cpp: In member function 'void HALSITL::SITL_State::_parse_command_line(int, char* const*)':
../../libraries/AP_HAL_SITL/SITL_cmdline.cpp:240:19: error: reference to 'timeval' is ambiguous
  240 |     static struct timeval first_tv;
      |                   ^~~~~~~
compilation terminated due to -Wfatal-errors.

Waf: Leaving directory `/ardupilot/src/build/sitl'
Build failed
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632954304: cxx SIM_GPS_NMEA.cpp -> SIM_GPS_NMEA.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632958112: cxx SIM_MicroStrain.cpp -> SIM_MicroStrain.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632954192: cxx SIM_GPS_MSP.cpp -> SIM_GPS_MSP.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632953968: cxx SIM_GPS.cpp -> SIM_GPS.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632953072: cxx SIM_FlightAxis.cpp -> SIM_FlightAxis.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632958224: cxx SIM_MicroStrain7.cpp -> SIM_MicroStrain7.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/SITL' failed (exit status 1): 
        {task 137329632964496: cxx SIM_VectorNav.cpp -> SIM_VectorNav.cpp.0.o}
 (run with -v to display more information)
 -> task in 'objs/AP_HAL_SITL/ArduCopter' failed (exit status 1): 
        {task 137329632802448: cxx SITL_cmdline.cpp -> SITL_cmdline.cpp.3.o}
 (run with -v to display more information)

[khancyr]

hello,

Looks like a well know issue on alpine, it needs to include #include <sys/time.h>. Would you like to make a PR to correct this ? or I can do it later today

[Zavitit]

I started doing some changes with the <sys.time.h> and <time.h> but it became messy pretty fast and I don't want to make changes I don't fully understand I'd really appreciate it if you could give it a go later :pray:

[khancyr]

ok np, I will look at it tonight

[Zavitit]

@khancyr have you had a chance to look at this issue?

[Zavitit]

Sorry about the status change is was a miss-click 😅

[khancyr]

Hello,

I forget... sorry. I will do it soon and add some test so alpine don't break again

[khancyr]

solved in https://github.com/ArduPilot/ardupilot/pull/27818

[peterbarker]

@Zavitit could you test that @khancyr 's patches fix the problem, please?

[khancyr]

Should be good for SITL. Linux target still have a lot of issues

[Zavitit]

unfortunately this fails for me but it looks like it might be a compatibility issue with the python version installed on my docker (3.11) what version where you using to test?

the crash i get when running ./waf compter

 ...
expanding uavcan.equipment.gnss.RTCMStream
expanding uavcan.equipment.gnss.Fix
expanding uavcan.equipment.gnss.ECEFPositionVelocity

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/ardupilot/src/modules/DroneCAN/dronecan_dsdlc/dronecan_dsdlc.py", line 118, in expand_message
    output = em.expand(msg_header_template, get_empy_env_broadcast(msg))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 5961, in expand
    result = interp.expand(data, locals, name, dispatcher=None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 4681, in expand
    self.string(data, locals, dispatcher)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 4830, in string
    while not self.safe(scanner, True, locals, dispatcher):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 4862, in safe
    if dispatcher():
       ^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 4843, in safe
    return self.parse(scanner, locals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 4870, in parse
    token = scanner.one()
            ^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 3999, in one
    token.scan(self)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2648, in scan
    self.subscan(scanner, self.type)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2664, in subscan
    token = scanner.one()
            ^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 3999, in one
    token.scan(self)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2648, in scan
    self.subscan(scanner, self.type)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2664, in subscan
    token = scanner.one()
            ^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 3999, in one
    token.scan(self)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2648, in scan
    self.subscan(scanner, self.type)
  File "/root/.local/lib/python3.11/site-packages/em.py", line 2664, in subscan
    token = scanner.one()
            ^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/em.py", line 3995, in one
    raise ParseError("unknown markup sequence: `%s%s`%s" % (self.config.prefix, first, self.factory.addendum(first)))
em.ParseError: unknown markup sequence: `@)`; the `@)` markup has been removed, just use `)` instead
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ardupilot/src/modules/DroneCAN/dronecan_dsdlc/dronecan_dsdlc.py", line 194, in <module>
    x = result.get()
        ^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
em.ParseError: unknown markup sequence: `@)`; the `@)` markup has been removed, just use `)` instead

dronecangen returned 1 error code
Waf: Leaving directory `/ardupilot/src/build/sitl'
Build failed
 -> task in 'dronecan' failed (exit status 1): 
        {task 130332061951024: dronecangen ardupilot,com,cuav,dronecan,mppt,tests,uavcan -> }
 (run with -v to display more information)

[peterbarker]

@Zavitit the empy version should be pinned to known-working version. Check the install script to see which one.

[MichaelThamm]

Edit: I just realized this issue was specific to the Alpine image. If it's better suited for its own issue, I am happy to create a new one.

I'm getting a similar issue on Ubuntu (24.04) when running:

git clone --recurse-submodules https://github.com/Ardupilot/ardupilot.git
Tools/environment_install/install-prereqs-ubuntu.sh
. ~/.profile
./waf configure --board sitl
source ~/venv-ardupilot/bin/activate
./waf copter

Waf: Entering directory `/home/mt/work/repos/drone/ardupilot/build/sitl'
Embedding file locations.txt:Tools/autotest/locations.txt
Embedding file models/Callisto.json:Tools/autotest/models/Callisto.json
Embedding file models/freestyle.json:Tools/autotest/models/freestyle.json
Embedding file models/plane-3d.parm:Tools/autotest/models/plane-3d.parm
Embedding file models/plane.parm:Tools/autotest/models/plane.parm
Embedding file models/xplane_heli.json:Tools/autotest/models/xplane_heli.json
Embedding file models/xplane_plane.json:Tools/autotest/models/xplane_plane.json
[10/12] Processing dronecangen: modules/DroneCAN/DSDL/ardupilot modules/DroneCAN/DSDL/com modules/DroneCAN/DSDL/cuav modules/DroneCAN/DSDL/dronecan modules/DroneCAN/DSDL/mppt modules/DroneCAN/DSDL/tests modules/DroneCAN/DSDL/uavcan
Traceback (most recent call last):
  File "/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/dronecan_dsdlc/dronecan_dsdlc.py", line 12, in <module>
    import em
ModuleNotFoundError: No module named 'em'

dronecangen: cmd=['/usr/bin/python3', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/dronecan_dsdlc/dronecan_dsdlc.py', '-O/home/mt/work/repos/drone/ardupilot/build/sitl/modules/DroneCAN/libcanard/dsdlc_generated', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/ardupilot', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/com', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/cuav', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/dronecan', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/mppt', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/tests', '/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/DSDL/uavcan'] 
Traceback (most recent call last):
  File "/home/mt/work/repos/drone/ardupilot/modules/DroneCAN/dronecan_dsdlc/dronecan_dsdlc.py", line 12, in <module>
    import em
ModuleNotFoundError: No module named 'em'
dronecangen returned 1 error code
Generating compile_commands.json
Build commands will be stored in build/sitl/compile_commands.json
Generating compile_commands.json
Build commands will be stored in build/sitl/compile_commands.json
Waf: Leaving directory `/home/mt/work/repos/drone/ardupilot/build/sitl'
Build failed
 -> task in 'dronecan' failed (exit status 1): 
	{task 128738346600944: dronecangen ardupilot,com,cuav,dronecan,mppt,tests,uavcan -> }
 (run with -v to display more information)

With the notable error: No module named 'em' but when I check the venv-ardupilot site-packages I see that empy-3.3.4 is installed. When I check the ardupilot/Tools/environment_install/install-prereqs-ubuntu.sh script as suggested, I see that this matches the installed version.

PYTHON_PKGS="future lxml pymavlink pyserial MAVProxy geocoder empy==3.3.4 ptyprocess dronecan"

I also tried to run ./waf copter --disable-DroneCAN hoping it would ignore these files but it didn't work.

Is there anything I am doing wrong?