tasmocompiler icon indicating copy to clipboard operation
tasmocompiler copied to clipboard

Published 20 hours ago • verified publisher icon benzino77

chore: update Dockerfile to reduce the image size

Open tdurieux opened this issue 2 years ago • 8 comments

Hi there,

I've made a small improvement to the Dockerfile that I think could help optimize the image size.

Summary of the changes:

  • I added the --no-cache-dir to the pip command to disable the package cache.
  • I added the --no-install-recommends to with apt-get in order to not install unnecessary packages and reduce the image size.

Impact on the image size:

  • Image size before repair: 1.18 GB
  • Image size after repair: 887.29 MB
  • Difference: 323.78 MB (26.73%)

I hope that you will find these changes useful to you. Let me know if you have any questions or concerns.

Thanks,

tdurieux avatar Jan 22 '23 19:01 tdurieux

It was deleted by purpose: https://github.com/benzino77/tasmocompiler/commit/ec1a0070bb1db2d8338d7955e43e19d8a3d0a22f

benzino77 avatar Jan 23 '23 12:01 benzino77

Sorry I did not see that.

tdurieux avatar Jan 23 '23 12:01 tdurieux

There was a problem with one package installation (can't remember which one - zeroconf?) as a dependency for platformio. That package was installed in wrong version when --no-cache-dir was in place.

benzino77 avatar Jan 23 '23 13:01 benzino77

That is an interesting behavior, I was not aware that --no-cache-dir could impact the version selection. I could only see it happening in a weird version conflict inside transitive dependencies. I am not sure how to fix this problem correctly because relying on the cache to solve the problem seems a bit unreliable :)

tdurieux avatar Jan 23 '23 13:01 tdurieux

I'm not sure where the problem is (maybe with defined dependencies for platformio?), but I've made an observation that starting TasmoCompiler on GitPod, where there is no --no-cache-dir flag, works as expected.

benzino77 avatar Jan 23 '23 13:01 benzino77

I reproduced the bug I dont have a better solution for the moment. I will check the behavior of the image with only --no-install-recommends.

Processing tasmota4M (board: esp8266_4M2M; platform: https://github.com/tasmota/platform-espressif8266/releases/download/v2.7.4/platform-espressif8266-2.7.4.zip; framework: arduino)
--------------------------------------------------------------------------------
Platform Manager: Installing https://github.com/tasmota/platform-espressif8266/releases/download/v2.7.4/platform-espressif8266-2.7.4.zip
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80%
Unpacking
Platform Manager: [email protected]+20221113 has been installed!
Tool Manager: Installing platformio/toolchain-xtensa @ ~2.40802.0
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking
Tool Manager: [email protected] has been installed!
Tool Manager: Installing tasmota/framework-arduinoespressif8266 @ ~2.7.4
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking
Tool Manager: [email protected]+9 has been installed!
Tool Manager: Installing https://github.com/tasmota/esptool/releases/download/v4.3.1/esptool-4.3.1.zip
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking
Tool Manager: [email protected] has been installed!
Tool Manager: Installing platformio/tool-scons @ ~4.40400.0
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking
Tool Manager: [email protected] has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp8266_4M2M.html
PLATFORM: Espressif 8266 (2.7.4+20221113) > Espressif Generic ESP8266 Tasmota 1M sketch 1M OTA 2M FS
HARDWARE: ESP8266 80MHz, 80KB RAM, 972.00KB Flash
PACKAGES: 
 - framework-arduinoespressif8266 @ 2.7.4+9 
 - tool-esptoolpy @ 1.40301.221017 (4.3.1) 
 - toolchain-xtensa @ 2.40802.200502 (4.8.2)
Converting tasmota.ino
Installing Python dependencies
Collecting zopfli>=0.2.1
  Downloading https://files.pythonhosted.org/packages/9a/ed/d004d5737f9546167eecf0ecd995ee1a796703e512deb2f2ea26cdbe4b3e/zopfli-0.2.2.zip (205kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Building wheels for collected packages: zopfli
  Running setup.py bdist_wheel for zopfli: started
  Running setup.py bdist_wheel for zopfli: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-flclb6c9/zopfli/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-ydt4aa7u --python-tag cp37:
  /tmp/pip-build-env-b8eo6qsm/lib/python3.7/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  /tmp/pip-build-env-b8eo6qsm/lib/python3.7/site-packages/setuptools/installer.py:30: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
    SetuptoolsDeprecationWarning,
  WARNING: The wheel package is not available.
  WARNING: The wheel package is not available.
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Running setup.py clean for zopfli
  Failed building wheel for zopfli
Failed to build zopfli
Installing collected packages: zopfli
  Running setup.py install for zopfli: started
    Running setup.py install for zopfli: finished with status 'error'
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-flclb6c9/zopfli/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-b4x6o0jj/install-record.txt --single-version-externally-managed --compile:
    /tmp/pip-build-env-b8eo6qsm/lib/python3.7/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
      warnings.warn(msg, warning_class)
    /tmp/pip-build-env-b8eo6qsm/lib/python3.7/site-packages/setuptools/installer.py:30: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
      SetuptoolsDeprecationWarning,
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    running install
    /tmp/pip-build-env-b8eo6qsm/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      setuptools.SetuptoolsDeprecationWarning,
    running build
    running build_py
    creating build
    creating build/lib.linux-aarch64-cpython-37
    creating build/lib.linux-aarch64-cpython-37/zopfli
    copying src/zopfli/_version.py -> build/lib.linux-aarch64-cpython-37/zopfli
    copying src/zopfli/gzip.py -> build/lib.linux-aarch64-cpython-37/zopfli
    copying src/zopfli/png.py -> build/lib.linux-aarch64-cpython-37/zopfli
    copying src/zopfli/__init__.py -> build/lib.linux-aarch64-cpython-37/zopfli
    copying src/zopfli/zlib.py -> build/lib.linux-aarch64-cpython-37/zopfli
    running egg_info
    writing src/zopfli.egg-info/PKG-INFO
    writing dependency_links to src/zopfli.egg-info/dependency_links.txt
    writing requirements to src/zopfli.egg-info/requires.txt
    writing top-level names to src/zopfli.egg-info/top_level.txt
    listing git files failed - pretending there aren't any
    reading manifest file 'src/zopfli.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    adding license file 'COPYING'
    writing manifest file 'src/zopfli.egg-info/SOURCES.txt'
    running build_ext
    building 'zopfli.zopfli' extension
    creating build/temp.linux-aarch64-cpython-37
    creating build/temp.linux-aarch64-cpython-37/src
    creating build/temp.linux-aarch64-cpython-37/zopfli
    creating build/temp.linux-aarch64-cpython-37/zopfli/src
    creating build/temp.linux-aarch64-cpython-37/zopfli/src/zopfli
    creating build/temp.linux-aarch64-cpython-37/zopfli/src/zopflipng
    creating build/temp.linux-aarch64-cpython-37/zopfli/src/zopflipng/lodepng
    aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python3.7m -c src/zopflimodule.c -o build/temp.linux-aarch64-cpython-37/src/zopflimodule.o
    error: command 'aarch64-linux-gnu-gcc' failed: No such file or directory: 'aarch64-linux-gnu-gcc'
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-flclb6c9/zopfli/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-b4x6o0jj/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-flclb6c9/zopfli/
*** Error 1
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict
Found 94 compatible libraries
Scanning dependencies...
Dependency Graph
|-- Adafruit CCS811 Library @ 1.0.0
|   |-- Wire @ 1.0
|-- Adafruit MAX31865 library @ 1.0.1
|   |-- SPI @ 1.0
|-- Adafruit MCP9808 Library @ 1.1.2
|   |-- Wire @ 1.0
|-- Adafruit SGP30 Sensor @ 1.2.0
|   |-- Wire @ 1.0
|-- Adafruit SGP40 Sensor @ 1.1.0
|   |-- Adafruit BusIO @ 1.11.0
|   |   |-- Wire @ 1.0
|   |   |-- SPI @ 1.0
|-- Adafruit VEML7700 Library @ 1.0.0
|   |-- Adafruit BusIO @ 1.11.0
|   |   |-- Wire @ 1.0
|   |   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|-- Adafruit seesaw Library @ 1.3.1
|   |-- Wire @ 1.0
|-- ESP8266Audio @ 1.9.5
|   |-- ESP8266HTTPClient @ 1.2
|   |   |-- ESP8266WiFi @ 1.0
|   |-- SD(esp8266) @ 2.0.0
|   |   |-- SDFS @ 0.1.0
|   |   |   |-- SPI @ 1.0
|   |   |   |-- ESP8266SdFat @ 1.1.0
|   |   |   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|-- BME8563
|   |-- Wire @ 1.0
|-- EEPROM 24C128_256_521
|   |-- Wire @ 1.0
|-- I2Cdevlib-Core
|   |-- Wire @ 1.0
|-- LibTeleinfo @ 1.1.3
|-- MPU6050 I2C
|   |-- I2Cdevlib-Core
|   |   |-- Wire @ 1.0
|-- Mutichannel_Gas_Sensor @ 0.0.1
|   |-- Wire @ 1.0
|-- NTP Library
|   |-- ESP8266WiFi @ 1.0
|-- OpenTherm Library @ 0.9.0
|-- Process_control @ 1.0.0
|-- SevenSegmentTM1637 @ 1.1.0
|-- UdpListener @ 1.0
|-- Unishox Compressor Decompressor highly customized and optimized for ESP8266 and Tasmota @ 1.0
|-- VL53L0X @ 1.0.2
|   |-- Wire @ 1.0
|-- VL53L1X @ 1.0.1
|   |-- Wire @ 1.0
|-- TLS mini client derived from Arduino @ 1.0
|   |-- ESP8266WiFi @ 1.0
|   |-- BearSSL @ 0.6
|-- C2Programmer @ 1.0.0
|-- headers
|-- cc1101 @ 1.0
|   |-- SPI @ 1.0
|-- arduino-mcp2515-1.0.1
|   |-- SPI @ 1.0
|-- A4988_Stepper @ 0.0.1
|-- Adafruit Fingerprint Sensor Library @ 2.0.4
|   |-- TasmotaSerial @ 3.6.0
|-- Adafruit GFX Library @ 1.5.6
|   |-- SPI @ 1.0
|-- Adafruit LED Backpack Library @ 1.1.6
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|-- Adafruit SH1106-gemu-1.0 @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|-- Adafruit SSD1306 @ 1.3.0
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- Adafruit SSD1331 OLED Driver Library for Arduino @ 1.2.0
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- Adafruit TSL2591
|   |-- Wire @ 1.0
|-- Arduino ST7789 Library @ 0.9.5
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|-- DNSServer @ 1.1.1
|   |-- ESP8266WiFi @ 1.0
|-- DnsClient @ 1.0
|   |-- ESP8266WiFi @ 1.0
|-- ESP8266HTTPClient @ 1.2
|   |-- ESP8266WiFi @ 1.0
|-- ESP8266SAM @ 1.0.1
|   |-- ESP8266Audio @ 1.9.5
|   |   |-- ESP8266HTTPClient @ 1.2
|   |   |   |-- ESP8266WiFi @ 1.0
|   |   |-- SD(esp8266) @ 2.0.0
|   |   |   |-- SDFS @ 0.1.0
|   |   |   |   |-- SPI @ 1.0
|   |   |   |   |-- ESP8266SdFat @ 1.1.0
|   |   |   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- ESP8266WebServer @ 1.0
|   |-- ESP8266WiFi @ 1.0
|-- ESP8266WiFi @ 1.0
|-- ESP8266httpUpdate @ 1.3
|   |-- ESP8266HTTPClient @ 1.2
|   |   |-- ESP8266WiFi @ 1.0
|   |-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|   |-- ESP8266WiFi @ 1.0
|-- FT5206_Library @ 1.0.0
|   |-- Wire @ 1.0
|-- FrogmoreScd30
|   |-- Wire @ 1.0
|-- FrogmoreScd40
|   |-- Wire @ 1.0
|-- ILI9341 @ 1.0.0
|   |-- SPI @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- IRremoteESP8266 @ 2.8.4
|-- JSMN JSON parser customized and optimized for ESP8266 and Tasmota @ 1.0
|-- KeeloqLib @ 1.1
|-- TasmotaLList @ 1.0
|-- LOLIN_HP303B @ 1.0.0
|   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|-- LedControl @ 1.0.6
|-- LinkedList
|-- LiquidCrystal_I2C
|   |-- Wire @ 1.0
|-- LittleFS(esp8266) @ 0.1.0
|-- MFRC522 @ 1.4.7
|   |-- SPI @ 1.0
|-- MLX90640
|   |-- Wire @ 1.0
|-- MPU_accel
|   |-- Wire @ 1.0
|-- NeoPixelBus @ 2.6.7
|   |-- SPI @ 1.0
|-- NewPing @ 1.9.1
|-- OneWire @ 2.3.2
|-- PubSubClient @ 2.8
|-- RA8876 @ 1.0.2
|   |-- SPI @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- rc-switch @ 1.0.0
|-- RF24 @ 1.3.3
|   |-- SPI @ 1.0
|-- Ext-printf @ 1.0
|-- SD(esp8266) @ 2.0.0
|   |-- SDFS @ 0.1.0
|   |   |-- SPI @ 1.0
|   |   |-- ESP8266SdFat @ 1.1.0
|   |   |   |-- SPI @ 1.0
|-- SPI @ 1.0
|-- SSD3115 @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|-- ESP8266SdFat @ 1.1.0
|   |-- SPI @ 1.0
|-- TM1638plus @ 1.7.0
|-- TasmotaModbus @ 3.6.0
|   |-- TasmotaSerial @ 3.6.0
|-- TasmotaSerial @ 3.6.0
|-- Ticker @ 1.0
|-- Joba_Tsl2561 @ 2.0.10
|   |-- Wire @ 1.0
|-- Wire @ 1.0
|-- XPT2046_Touchscreen
|   |-- SPI @ 1.0
|-- base64 @ 1.1.1
|-- BME68x
|-- Waveshare esp 2.9 inch e-paper display driver @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- Waveshare esp 4.2 inch e-paper display driver @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|-- Display renderer @ 1.0
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|-- ESP KNX IP Library @ 0.5.2
|   |-- EEPROM @ 1.0
|   |-- ESP8266WebServer @ 1.0
|   |   |-- ESP8266WiFi @ 1.0
|   |-- ESP8266WiFi @ 1.0
|-- HPMA115S0 Arduino Library @ 1.0.0
|-- stm32_flash @ 1.0.0
|-- BearSSL @ 0.6
|-- universal display Library @ 0.1
|   |-- Adafruit GFX Library @ 1.5.6
|   |   |-- SPI @ 1.0
|   |-- SPI @ 1.0
|   |-- Wire @ 1.0
|   |-- Display renderer @ 1.0
|   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
|   |-- SSD3115 @ 1.0
|   |   |-- Display renderer @ 1.0
|   |   |   |-- Adafruit GFX Library @ 1.5.6
|   |   |   |   |-- SPI @ 1.0
|   |   |   |-- SPI @ 1.0
|   |   |-- SPI @ 1.0
Building in release mode
ModuleNotFoundError: No module named 'zopfli':
  File "/usr/local/lib/python3.7/dist-packages/platformio/builder/main.py", line 195:
    env.SConscript(item, exports="env")
  File "/root/.platformio/packages/tool-scons/scons-local-4.4.0/SCons/Script/SConscript.py", line 597:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/root/.platformio/packages/tool-scons/scons-local-4.4.0/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/tmp/Tasmota/pio-tools/gzip-firmware.py", line 33:
    from zopfli.gzip import compress
========================= [FAILED] Took 34.60 seconds =========================

Environment    Status    Duration
-------------  --------  ------------
tasmota4M      FAILED    00:00:34.601
==================== 1 failed, 0 succeeded in 00:00:34.601 ====================
Finished. Exit code: 1.

tdurieux avatar Jan 23 '23 13:01 tdurieux

That error was due to --no-install-recommends not due to --no-cache-dir.

With only --no-cache-dir, I succeed to compile with the config provided in #320 but the size reduction is much smaller (29mb).

image image image

tdurieux avatar Jan 23 '23 15:01 tdurieux

My experience, trying to use cache with Platformio is a lottery game. Been there. After removing every cache ALL weird issues we had where solved permanently. So for Tasmota we will never try again "cache"

Jason2866 avatar Jan 08 '24 10:01 Jason2866