mirrexagon
"idf.py create-project" fails with "PermissionError: [Errno 13] Permission denied: ..."
Created dev environment with nix --experimental-features 'nix-command flakes' develop github:mirrexagon/nixpkgs-esp-dev#esp32-idf
And tried to create an empty project causes permission errors:
[capreme@thinkpad:~/Projects/tmp_fresh]$ idf.py create-project hello
WARNING: Python interpreter "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/bin/python3.12" used to start idf.py is not from installed venv "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/python-env"
Executing action: create-project
Traceback (most recent call last):
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 855, in <module>
main()
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 745, in main
cli(argv, prog_name=PROG, complete_var=SHELL_COMPLETE_VAR)
File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1720, in invoke
return _process_result(rv)
^^^^^^^^^^^^^^^^^^^
File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 1657, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 640, in execute_tasks
task(ctx, global_args, task.action_args)
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf.py", line 212, in __call__
self.callback(self.name, context, global_args, **action_args)
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf_py_actions/create_ext.py", line 76, in create_new
func_action_map[action](target_path, action_args['name'])
File "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/tools/idf_py_actions/create_ext.py", line 49, in create_project
os.rename(os.path.join(main_folder, 'main.c'), os.path.join(main_folder, '.'.join((name, 'c'))))
PermissionError: [Errno 13] Permission denied: '/home/capreme/Projects/tmp_fresh/hello/main/main.c' -> '/home/capreme/Projects/tmp_fresh/hello/main/hello.c'
idf --version works and reports a version
[capreme@thinkpad:~/Projects/tmp_fresh]$ idf.py --version
WARNING: Python interpreter "/nix/store/6ipv375swmlk0yf1m1fwn0fq6i7ph2jl-python3-3.12.6-env/bin/python3.12" used to start idf.py is not from installed venv "/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/python-env"
fatal: not a git repository: '/nix/store/b0d45nsqi08wyqvfqadgmxnw820fpfrz-esp-idf-v5.4/.git'
WARNING: Git version unavailable, reading from source
ESP-IDF v5.4.0
Running all on Nixos uname -a: Linux thinkpad 6.12.12 #1-NixOS SMP PREEMPT_DYNAMIC Sat Feb 1 17:39:40 UTC 2025 x86_64 GNU/Linux
Might be related issue 53, but in my case issue is with main.c file not CMakeLists.txt and I assume that it happends at a different step.
Okay I have the same Problem but I think I got it to work by just giving my user read and write permission for the whole folder:
chmod -R +rw hello
after that you can build the project.
This tool shouldn't need super user privileges, what does it do that requires it?
nope you are correct. I commented that when I had gotten rid of that error but not yet build the project succesfully.
These two commands ended up fixing it for me:
sudo chown -R <user>:users <path_to_project_file>
chmod -R u+rw <path to project folder>
The problem seems to be that idf.py create-project copies a sample project from $IDF_PATH/tools/templates/sample_project. Because $IDF_PATH is in the Nix store, all the files and directories are read-only.
idf.py tries to ensure that permissions aren't copied: https://github.com/espressif/esp-idf/blob/v5.4.1/tools/idf_py_actions/create_ext.py#L45-L46
But that only seems to affect files, not directories. When I try idf.py create-project, the main directory within the created project directory is still read-only (and has a last modified date in 1970, which is also normal for Nix store files).
https://github.com/mirrexagon/nixpkgs-esp-dev/pull/54 presumably fixed this previously. I will need to see if reapplying that fixes it again, or if something else needs to be done.
The functions used to do the copying have changed since #54, so that exact fix doesn't work.
https://github.com/espressif/esp-idf/commit/524f44dfc352e8046fd4c1d6d345433c3a230536 looks like they had this problem on Windows and fixed it there. The commit message mentions there is a test for copying from read-only ESP-IDF - I wonder if it is succeeding for them but still this happens for us, or the situation is a bit different (eg. permissions tested are a bit different from our case).
The function they use to do the copying (shutil.copytree()) has documentation that explicitly says it copies directory permissions and times: https://docs.python.org/3.12/library/shutil.html#shutil.copytree
I see that I'm not the only Nixer to find this out: https://github.com/python/cpython/issues/44602#issuecomment-1658653781
Based on the report of a known working commit of this repo and git bisect, it seems like the issue started after upgrading from ESP-IDF 5.3 to 5.4 (f3434d1fd5a51ec325a085d2638e6c1b7c132e5b).
When I get more time I will look at what changed in ESP-IDF and what can be done about it. If anyone wants to look, look for changes to https://github.com/espressif/esp-idf/blob/v5.4.1/tools/idf_py_actions/create_ext.py#L41-L52 between 5.3.1 and 5.4. I assume it is whatever changed between what worked in #54 and now.
Have done some investigation myself, and it seems this issue is related to shutil.copytree. I tried running the corresponding test with pytest, and it failed on my work computer at least. However, when I switched to the implementation of v5.3 (which use distutils), it passed.
However, distutils was removed in python 3.12, and this is the reason they changed the implementation.
Edited
Right now I don't have a particularly good solution. The closest approach I can think of is the following monkey patch, but it's still quite tricky.
diff --git a/tools/idf_py_actions/create_ext.py b/tools/idf_py_actions/create_ext.py
index 00d9fa2881..ec41d46e24 100644
--- a/tools/idf_py_actions/create_ext.py
+++ b/tools/idf_py_actions/create_ext.py
@@ -2,14 +2,57 @@
# SPDX-License-Identifier: Apache-2.0
import os
import re
+import shutil
import sys
-from shutil import copyfile
-from shutil import copytree
from typing import Dict
import click
from idf_py_actions.tools import PropertyDict
+def custom_copytree(src: str, dst: str, dirs_exist_ok: bool = False) -> str:
+ if not os.path.exists(src):
+ raise FileNotFoundError(f"Source directory '{src}' does not exist.")
+ if not os.path.isdir(src):
+ raise NotADirectoryError(f"Source path '{src}' is not a directory.")
+ if (not dirs_exist_ok) and os.path.exists(dst) and os.listdir(dst):
+ raise FileExistsError(
+ f"Destination directory '{dst}' already exists and is not empty and dirs_exist_ok is False."
+ )
+
+ try:
+ os.makedirs(dst, exist_ok=True)
+ except FileExistsError:
+ raise NotADirectoryError(f"Destination path '{dst}' exists and is not a directory.")
+ except OSError as e:
+ raise shutil.Error(f"Could not create destination directory '{dst}': {e}")
+
+ names = os.listdir(src)
+ errors = []
+ for name in names:
+ srcname = os.path.join(src, name)
+ dstname = os.path.join(dst, name)
+ try:
+ if os.path.islink(srcname):
+ if os.path.isdir(srcname): # This follows the link
+ custom_copytree(srcname, dstname, dirs_exist_ok=True)
+ else:
+ shutil.copyfile(srcname, dstname)
+ elif os.path.isdir(srcname):
+ custom_copytree(srcname, dstname, dirs_exist_ok=True)
+ else:
+ shutil.copyfile(srcname, dstname)
+ except shutil.Error as e:
+ if hasattr(e, 'args') and e.args and isinstance(e.args[0], list):
+ errors.extend(e.args[0])
+ else:
+ errors.append((srcname, dstname, str(e)))
+ except OSError as why:
+ errors.append((srcname, dstname, str(why)))
+
+ if errors:
+ raise shutil.Error(errors)
+ return dst
+
def get_type(action: str) -> str:
return action.split('-')[1]
@@ -39,11 +82,9 @@ def is_empty_and_create(path: str, action: str) -> None:
def create_project(target_path: str, name: str) -> None:
- copytree(
+ custom_copytree(
os.path.join(os.environ['IDF_PATH'], 'tools', 'templates', 'sample_project'),
target_path,
- # 'copyfile' ensures only data are copied, without any metadata (file permissions)
- copy_function=copyfile,
dirs_exist_ok=True,
)
main_folder = os.path.join(target_path, 'main')
@@ -53,11 +94,9 @@ def create_project(target_path: str, name: str) -> None:
def create_component(target_path: str, name: str) -> None:
- copytree(
+ custom_copytree(
os.path.join(os.environ['IDF_PATH'], 'tools', 'templates', 'sample_component'),
target_path,
- # 'copyfile' ensures only data are copied, without any metadata (file permissions)
- copy_function=copyfile,
dirs_exist_ok=True,
)
os.rename(os.path.join(target_path, 'main.c'), os.path.join(target_path, '.'.join((name, 'c'))))
At least related test are passed.
/media/spedon/eden/esp-idf/tools/test_build_system on git master [!?] via py v3.11.12 took 2s
❯ pytest -k test_create_project
================================================================================ test session starts =================================================================================
platform linux -- Python 3.11.12, pytest-8.3.5, pluggy-1.5.0
rootdir: /media/spedon/eden/esp-idf/tools/test_build_system
configfile: pytest.ini
plugins: embedded-1.16.1, ignore-test-results-0.3.0, timeout-2.4.0, rerunfailures-15.1
collected 103 items / 101 deselected / 2 selected
test_common.py::test_create_project
----------------------------------------------------------------------------------- live log call ------------------------------------------------------------------------------------
2025-05-16 13:54:20 INFO Check that command for creating new project will fail if the target folder is not empty.
2025-05-16 13:54:20 INFO Check that command for creating new project will fail if the target path is file.
PASSED [ 50%]
test_common.py::test_create_project_with_idf_readonly
----------------------------------------------------------------------------------- live log call ------------------------------------------------------------------------------------
2025-05-16 13:54:22 INFO Check that command for creating new project will success if the IDF itself is readonly.
PASSED [100%]
================================================================================== warnings summary ==================================================================================
test_common.py::test_create_project
test_common.py:218: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
def test_create_project(idf_py: IdfPyFunc, idf_copy: Path) -> None:
test_common.py::test_create_project_with_idf_readonly
test_common.py:234: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
@pytest.mark.skipif(sys.platform == 'darwin', reason='macos runner is a shell executor, it would break the file system')
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== 2 passed, 101 deselected, 2 warnings in 3.98s ====================================================================
I will try to create a stable reproduction from the idf's docker image. If can, I will try to report this issue to upstream.
should be fixed by https://github.com/espressif/esp-idf/commit/b3f24a9533dadbbcb2a69ee18cc074f7fd779814
should be fixed by espressif/esp-idf@b3f24a9
I wish this worked, but I'm getting an error when I override the version.
idf.py --version lb
The following Python requirements are not satisfied:
Error while checking requirement 'esp-idf-diag'. Package was not found and is required by the application: No package metadata was found for esp_idf_diag
To install the missing packages, please run "install.sh"
Diagnostic information:
IDF_PYTHON_ENV_PATH: /nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/python-env
Python interpreter used: /nix/store/h0ak3p412pcrc1106ia9lf17k6abw9ph-python3-3.11.9-env/bin/python3.11
Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
PATH: /home/lb/.cache/antidote/https-COLON--SLASH--SLASH-github.com-SLASH-chisui-SLASH-zsh-nix-shell/scripts:/nix/store/6wf8zxqql7gmmi96k6swmm0pvqp13gi8-patchelf-0.15.0/bin:/nix/store/r6k305g2rn0qqkvdxfvzqg7mhh7rw679-gcc-wrapper-13.2.0/bin:/nix/store/6g5fhxv0bdm7236ixdwq1izzawbc7grm-gcc-13.2.0/bin:/nix/store/4y2kcviw0cczb56lfl84h7n7j7xm8hh0-glibc-2.39-52-bin/bin:/nix/store/mr63za5vkxj0yip6wj3j9lya2frdm3zc-coreutils-9.5/bin:/nix/store/si9c1ncjikjq2hl5m8i6dzrfznmsz7g5-binutils-wrapper-2.41/bin:/nix/store/q3sm4x963a996qc3d6baw54609ryifak-binutils-2.41/bin:/nix/store/h0ak3p412pcrc1106ia9lf17k6abw9ph-python3-3.11.9-env/bin:/nix/store/vdd8ccp9x69v7kp5g9khq7j43n6dqf16-git-2.44.1/bin:/nix/store/kq34ksagmi9cqcvdwyfarr7kmmmxdzkz-wget-1.21.4/bin:/nix/store/qgsv4qck6z0plwz2mmxw76473pipb0zn-gnumake-4.4.1/bin:/nix/store/c9kis6qynbfp15936nq76vnm41x65aml-flex-2.6.4/bin:/nix/store/fy9xx920hk3cyvxx0wjzq2dkfjf0s0fx-gnum4-1.4.19/bin:/nix/store/w2agjznxiyjlj87dwmclw3f78bz9jza3-bison-3.8.2/bin:/nix/store/ld5621ah2ybhi5z9b60w9idp95ghsjh4-gperf-3.1/bin:/nix/store/4lssw6pk59r654j0yrfydv0kngk8zrqp-pkg-config-wrapper-0.29.2/bin:/nix/store/b893ksdqbdpzrdcmms59aq4fgqmh083w-cmake-3.29.2/bin:/nix/store/j8ly6xc733w2wz98mz9avhy2bmibbcg9-ninja-1.11.1/bin:/nix/store/shl99d7ksiy7qdkv68fnga4fsasy7rgr-ncurses-abi5-compat-6.4-dev/bin:/nix/store/1nsscr4hbxpcrdjiprj3mcr99f7zfjdr-ncurses-abi5-compat-6.4/bin:/nix/store/h3vmvr4w38w1clcwi96w9j7ik61c4227-dfu-util-0.11/bin:/nix/store/am8b9ykildsd2zmgs6g0z2vz6fgm37il-esp32ulp-elf-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/bin:/nix/store/9kz0cz5qspi53p15ib381jxh3ha0fdvc-openocd-esp32-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/bin:/nix/store/j2pva28z3vqqyx2vaxvc3w0x5gssfi5n-xtensa-esp-elf-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/bin:/nix/store/4hkmaw7zhrlbfyl089l26bwz47v3sydc-xtensa-esp-elf-gdb-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/bin:/nix/store/myb4rzvdyahvvlyyx32i19nhmr78bc8k-findutils-4.9.0/bin:/nix/store/wk9vn3lcxqya6a5j17v87x18lrkg2ngy-diffutils-3.10/bin:/nix/store/nhr25jgz6bpxbrz6kndjyvgaxipall0j-gnused-4.9/bin:/nix/store/3jc4m5gwimj4mbm01ijgkm2di8hiy5l3-gnugrep-3.11/bin:/nix/store/g7rq3khf0bnm64amflrc4964b7q57dqv-gawk-5.2.2/bin:/nix/store/wqbq6prmrhgm19qqdqm3ijjbap9x74cn-gnutar-1.35/bin:/nix/store/jpqm5igl1gmahp7lxx8j1dy874zvirgm-gzip-1.13/bin:/nix/store/azwrrzpvar2qyn5s4cmrj4jfgfjgmkbv-bzip2-1.0.8-bin/bin:/nix/store/mywvjmd80bayzbi161nly2vfpikyc47h-gnumake-4.4.1/bin:/nix/store/516kai7nl5dxr792c0nzq0jp8m4zvxpi-bash-5.2p32/bin:/nix/store/cjjaglbcsg30bc7rf6xajpp73fkna2mk-patch-2.7.6/bin:/nix/store/kpabg6z89nfry6frc3p5m3zfmk94zyqn-xz-5.4.7-bin/bin:/nix/store/vqhj138xim73r80qcsc546crm4y80ca2-file-5.45/bin:/nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/tools:/nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/components/espcoredump:/nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/components/partition_table:/nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/components/app_update:/run/wrappers/bin:/home/lb/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin:/home/lb/.nix-profile/bin:/nix/profile/bin:/home/lb/.local/state/nix/profile/bin:/etc/profiles/per-user/lb/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
Requirement files:
- /nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/tools/requirements/requirements.core.txt
Python being checked: /nix/store/2fzqwj27vghbssj3wbpdirmn0nz0hgp3-esp-idf-5b11d5b26a8bf151fc6bac400158859eedd413bc/python-env/bin/python
The rev I'm pointing to isn't the exact one you linked, but the current master. I also tried the one you linked, same error.
let
nixpkgs-esp-dev = builtins.fetchGit {
url = "https://github.com/mirrexagon/nixpkgs-esp-dev.git";
};
pkgs = import <nixpkgs> {
overlays = [ (import "${nixpkgs-esp-dev}/overlay.nix") ];
};
in pkgs.mkShell {
name = "esp-project";
buildInputs = with pkgs;
[
(esp-idf-esp32.override {
rev = "5b11d5b26a8bf151fc6bac400158859eedd413bc";
sha256 = "sha256-S7SHi1Yiw/apwUai5sY9iB9e1CHoCkcGb9i35pZC+2c=";
})
];
}
should be fixed by espressif/esp-idf@b3f24a9
Actually, the error makes sense. There's a new python dependency in the newer idf versions called esp-idf-diag. It's only that we cannot add python packages through .override.
@Azeirah maybe you can try add the esp-idf-diag to https://github.com/mirrexagon/nixpkgs-esp-dev/blob/master/pkgs/esp-idf/python-packages.nix?
but still it is such a pain to setup esp-idf to work with nix, i already gave up and use nix-ld instead, if you are interested you can check out this gist
@Azeirah maybe you can try add the esp-idf-diag to https://github.com/mirrexagon/nixpkgs-esp-dev/blob/master/pkgs/esp-idf/python-packages.nix?
but still it is such a pain to setup esp-idf to work with nix, i already gave up and use nix-ld instead, if you are interested you can check out this gist
I got it working by pointing at the fork with the C5 update branch from @timblaktu
Still, the permission error remains so I just did the dumb chmod chown workaround after forcing it with sudo :/
{
description = "ESP-IDF development environment with VSCodium";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11";
flake-utils.url = "github:numtide/flake-utils";
nix-vscode-extensions.url = "github:nix-community/nix-vscode-extensions";
esp-dev.url = "github:timblaktu/nixpkgs-esp-dev";
};
outputs = { self, nixpkgs, flake-utils, nix-vscode-extensions, esp-dev, }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
overlays =
[ esp-dev.overlays.default nix-vscode-extensions.overlays.default ];
};
extensions = pkgs.vscode-marketplace;
vscodium-with-extensions = pkgs.vscode-with-extensions.override {
vscode = pkgs.vscodium;
vscodeExtensions = [
extensions.golang.go
extensions.rust-lang.rust-analyzer
extensions.espressif.esp-idf-extension
];
};
in {
devShells.default = pkgs.mkShell {
buildInputs = [ vscodium-with-extensions pkgs.esp-idf-full ];
};
});
}