Requirements not satisfying when trying to build img for esp32s3

[simplyrohan]

Describe the bug

While running build-img with rg_tool I am getting an error saying I am not satisfying the packages. In specific, I am running

./rg_tool.py build-img launcher prboom-go --target qtpy-gamer

To Reproduce

1. Clone retro-go
2. Run ./rg_tool.py build-img launcher prboom-go --target qtpy-gamer Here is the error I am getting:

=== Step: Building ===

Building app 'launcher'
Running command: idf.py app -DRG_BUILD_VERSION=1.41-2-g9b53f -DRG_BUILD_TARGET=QTPY_GAMER -DRG_BUILD_TYPE=0 -DRG_ENABLE_PROFILING=0 -DRG_ENABLE_NETWORKING=1
The following Python requirements are not satisfied:
Requirement 'cryptography<35,>=2.1.4' was not met. Installed version: 42.0.7
Requirement 'pyparsing<2.4.0,>=2.0.3' was not met. Installed version: 3.1.2
pyelftools>=0.22
idf-component-manager~=1.1
pygdbmi<=0.9.0.2; python_version > "3.10"
kconfiglib==13.7.1
Requirement 'reedsolo<=1.5.4,>=1.5.3' was not met. Installed version: 1.7.0
Requirement 'bitstring<4,>=3.1.6' was not met. Installed version: 4.2.2
construct==2.10.54
Please follow the instructions found in the "Set up the tools" section of ESP-IDF Getting Started Guide
Diagnostic information:
    IDF_PYTHON_ENV_PATH: (not set)
    Python interpreter used: /usr/local/bin/python
    Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
    PATH: ...

ESP-IDF v4.3.7-1-g3a71941503

Task failed: Command '['idf.py', 'app', '-DRG_BUILD_VERSION=1.41-2-g9b53f', '-DRG_BUILD_TARGET=QTPY_GAMER', '-DRG_BUILD_TYPE=0', '-DRG_ENABLE_PROFILING=0', '-DRG_ENABLE_NETWORKING=1']' returned non-zero exit status 1.

Expected behavior

According to documentation, BUILDING.md, it should make an image for the device.

Additional context

I see that I can resolve these errors by downgrading some packages, but I would really like to avoid doing that because some of them, like cryptography, are core package.

[ducalex]

First I think we should confirm that your esp-idf installation is working. If you go to the launcher directory and run idf.py app, does it succeed?

If it fails then it would indicate a problem with your esp-idf installation.

But if it succeeds then I guess we have a compatibility issue with rg_tool.py and your particular setup. I suspect IDF_PYTHON_ENV_PATH: (not set) is a clue, somehow rg_tool.py doesn't pass down all the environment when calling idf.py?

[ducalex]

By the way the qtpy-gamer target is likely broken :(. I've made too many changes since it was introduced and I had no way of ensuring that it kept working. At the very least I'm almost sure that I broke the extended GPIO support.

[simplyrohan]

Sorry for the late reply.

First I think we should confirm that your esp-idf installation is working. If you go to the launcher directory and run idf.py app, does it succeed?

It seems the same dependency issues are happening here.

Here is the error

The following Python requirements are not satisfied:
Requirement 'cryptography<35,>=2.1.4' was not met. Installed version: 42.0.7
Requirement 'pyparsing<2.4.0,>=2.0.3' was not met. Installed version: 3.1.2
pyelftools>=0.22
idf-component-manager~=1.1
pygdbmi<=0.9.0.2; python_version > "3.10"
kconfiglib==13.7.1
Requirement 'reedsolo<=1.5.4,>=1.5.3' was not met. Installed version: 1.7.0
Requirement 'bitstring<4,>=3.1.6' was not met. Installed version: 4.2.2
construct==2.10.54
Please follow the instructions found in the "Set up the tools" section of ESP-IDF Getting Started Guide
Diagnostic information:
    IDF_PYTHON_ENV_PATH: (not set)
    Python interpreter used: /usr/local/bin/python
    Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
    PATH: ...

ESP-IDF v4.3.7-1-g3a71941503

I have installed ESP-IDF according to the docs and got no errors.

Also, I don't really know what the qtpygamer build was for. I want to build this for my QT Py ESP32S3, so should I build it for an ESP32S3? If so, what device would that be? Sorry, but I'm really new to this library 😅

I know this is a separate issue, but I am using an ST7735 TFT display with my QT Py, and it has a custom pin layout. How/Where would I define these? Idealy, I was hoping I could pass the display object (with Adafruit_GFX?)

[ducalex]

For the esp32s3 you will need at least esp-idf 4.4. Also I see that the qtpy-s3 is sold in multiple versions. Retro-Go absolutely needs 2MB of PSRAM or more, if your module doesn't have it then it's futile to continue.

Now to the bad news...

Retro-Go isn't a library, it's an entire operating system. It cannot be embedded in something like an Arduino project unfortunately, nor can it uses Arduino libraries for hardware support. So depending on your hardware, you might have to write drivers or modify existing ones.

If you still want to go ahead I recommend you use esp32s3-devkit-c as your starting point target, likely the closest to what you have. Then you'll have to tweak components/retro-go/targets/esp32s3-devkit-c/config.h to suit your device. It's quite possible you'll also have to patch retro-go to add support for your specific display/input/sound.

Hope you're not too discouraged!

[simplyrohan]

Welp, I have the one with no PSRAM... Thanks for the help though!

[simplyrohan]

@ducalex I got my hands on two QT Py's with 2MB of PSRAM and I'm currently figuring how to use the ESP-IDF

[simplyrohan]

Alright @ducalex, now that I have ESP-IDF setup properly, and have a much better understanding about how building things for any ESP32 work, I can try setting this repository up. (I had far to many issues with ESP-IDF but now it works)

$ idf.py --version
ESP-IDF v5.4-dev-1873-g41dd1a351b

I'm copying a lot of GPIO definitions from the qtpygamer config while double checking them with the QT Py docs. It is using the ILI9341 display right now, and I was hoping to change that. You mentioned patching retro-go to fit my hardware. Do you have a place to start? I don't plan on having audio support, but my input format is a little weird. Where could I implement the custom display and input code?

My current understanding is components/retro-go/rg_display.c and components/retro-go/rg_input.c are the only files I need to modify to fit my hardware. Is this correct?

Just to test, I tried building esp32s3-devkit-c as is (with no modifications). It failed however, with these errors

retro-go/components/retro-go/rg_input.c:10:10: fatal error: driver/adc.h: No such file or directory
retro-go/components/retro-go/drivers/display/ili9341.h:5:10: fatal error: driver/ledc.h: No such file or directory

I'm pretty sure I can solve these by downgrading my ESP-IDF version. Is that correct? (Reference)

Thanks for all your help! I'm really excited to get retro-go working!

(@algebric what are you doing? Why are you just thumbs-downing all my comments?)

[ducalex]

Just to test, I tried building esp32s3-devkit-c as is (with no modifications). It failed however, with these errors

retro-go/components/retro-go/rg_input.c:10:10: fatal error: driver/adc.h: No such file or directory
retro-go/components/retro-go/drivers/display/ili9341.h:5:10: fatal error: driver/ledc.h: No such file or directory

I have not tried esp-idf 5.4 yet, but I know that the adc driver was deprecated since 5.0 so maybe they finally removed it in 5.4?

I'm not really sure why ledc.h wouldn't be found however, it doesn't seem to be deprecated. So maybe there is something wrong with one of retro-go's CMakeLists.txt that breaks 5.4 or maybe in your setup? I will try 5.4 soon and make sure it builds!

Where could I implement the custom display and input code?

Modifying rg_input.c for input is correct. rg_input_init and rg_input_read_gamepad_raw are the functions you might have to modify. (Note that AW9523 support is likely completely broken so I wouldn't rely on that code.)

For display I have recently separated "drivers" to distinct files, they are in drivers/display. Since your display seems to be SPI, to get started you can modify drivers/display/ili9341.h (or clone the file and give yourself a new RG_SCREEN_DRIVER include in rg_display.c).

[simplyrohan]

Downgraded to 5.2 and the unmodified build works perfectly! I'll get started porting it to my hardware

(@algebric, can you at least comment something)

[simplyrohan]

Got a successful build with my custom screen! Although after trying to flash it, I had some problems. First, following the instructions printed after building

Bootloader build complete. To flash, run:
 idf.py bootloader-flash
or
 idf.py -p PORT bootloader-flash
or
 python -m esptool --chip esp32s3 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 16MB --flash_freq 80m 0x0 build/bootloader/bootloader.bin
or from the "/Users/rohangupta/Documents/Code/retroWorld/retro-go/launcher/build" directory
 python -m esptool --chip esp32s3 -b 460800 --before default_reset --after hard_reset write_flash "@flash_args"

I could only flash the launcher, not the prboom build I also wanted. None of the above commands except for the last worked

After flashing, the device just crashes on boot due to a flash size issue

E (214) spi_flash: Detected size(4096k) smaller than the size in the binary image header(16384k). Probe failed.

assert failed: do_core_init startup.c:328 (flash_ret == ESP_OK)

Do you think this has to do with how I'm flashing it?

I don't think this is an issue with retro-go though, and also not pertaining to this issue thread. If this is getting too off topic, do you think I should close this one (since the original issue was solved) and maybe start a new one with this issue?

[ducalex]

Esp-idf has no support for multiple apps in one project unfortunately, so you have to ignore all its messages about flashing stuff and instead use rg_tool.py for flashing. (I wish we could get rid of rg_tool.py!)

First you generate an .img: python rg_tool.py build-img launcher prboom-go. The file is a raw flash image for the esp32 (contains the partition table, bootloader, launcher, prboom). To flash it use either esptool or rg_tool.py --port=... install which calls esptool for you.

Once that initial image is flashed, you can use rg_tool.py to flash individual apps as you work on them, eg rg_tool.py flash launcher or rg_tool.py run launcher to build, flash, and then open serial monitor.

It's necessary to use rg_tool.py even when flashing individual apps because it will locate the correct partition whereas idf flash will always overwrite the first partition (the launcher).

rg_tool.py also passes the correct arguments to esptool.py in regards to the flash (--flash_size detect), so that error should go away too.

[ducalex]

I will try 5.4 soon and make sure it builds!

I've now tried 5.3 and 5.4. The ledc issue is an easy fix, just add esp_driver_ledc to COMPONENTS. But unfortunately they did indeed remove the legacy ADC and I2S drivers so it's impossible to use 5.3 or 5.4 for now.

[simplyrohan]

Hmmm, after flashing with rg_tool, a similar error about flash size still caused the ESP32S3 to crash at boot I used this command

./rg_tool.py --port=/dev/tty.usbmodem101 install launcher prboom-go --target retrotoids

(retrotoids is the modified target that I made)

The command has a mostly succesful output

Flashing image file 'retro-go_1.43-dirty_retrotoids.img' to /dev/tty.usbmodem101
Running command: esptool.py write_flash --flash_size detect 0x0 retro-go_1.43-dirty_retrotoids.img
esptool.py v4.7.0
Serial port /dev/tty.usbmodem101
Connecting...
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded Flash 4MB (XMC), Embedded PSRAM 2MB (AP_3v3)
Crystal is 40MHz
MAC: f4:12:fa:5a:50:78
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 1152000
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00000000 to 0x001bffff...
Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=4MB in order to remove this warning, or use the --dont-append-digest option for the elf2image command in order to generate an image file without a hash checksum
Compressed 1835008 bytes to 1164826...
Wrote 1835008 bytes (1164826 compressed) at 0x00000000 in 23.2 seconds (effective 631.8 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
All done!

The part that concerns me is the warning

Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=4MB in order to remove this warning, or use the --dont-append-digest option for the elf2image command in order to generate an image file without a hash checksum

I think that is what is causing the error E (308) spi_flash: Detected size(4096k) smaller than the size in the binary image header(16384k). Probe failed.

I'm wondering is this is an issue with my target configuration, and if I passed incorrect information about the chip in sdkconfig (which was copied from esp32s3-devkit-c) Any idea what is causing this?

[ducalex]

Interesting, I guess I never tried it myself on esp-idf 5.x. Might be possible to work around the checksum but maybe it would be best to just set the correct flash size in your target's sdkconfig.

To do this:

1. rg_tool.py clean rg_tool.py build launcher (this is so that a fresh sdkconfig file is generated)
2. Go to the launcher's folder and run idf menuconfig
3. Go to serial flasher config and change stuff to match your board
4. Save and exit
5. Replace your target's sdkconfig with the launcher's (mv -f launcher/sdkconfig components/retro-go/targets/retrotoids/sdkconfig
6. Then rg_tool.py clean rg_tool.py build-img rg_tool.py install

(Ideally your target's sdkconfig should contain only non-default values, so you can just overwrite the "serial flasher config" section instead of the entire file. But there's also no harm in being a full sdkconfig either and it's easier to get right :))

[simplyrohan]

Awesome! The configurations work! Thank you so much, for all your help. This repository looks so capable, and I'm super excited to get it working completely.

I've got the display working showing a "external memory not found error." Now I'll work on getting the SD card and input working. Thanks for all your help!