gramineproject
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Hi, I'm trying End-To-End Confidential PyTorch Workflow.
When I run gramine-sgx ./pytorch pytorchexample.py, I got
Detected a huge manifest, preallocating 128MB of internal memory. Gramine detected the following insecure configurations:
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- loader.insecure__use_host_env = true (forwarding environment vars from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
Detected a huge manifest, preallocating 128MB of internal memory. Traceback (most recent call last): File "pytorchexample.py", line 8, in
alexnet = torch.load("alexnet-pretrained.pt") File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 699, in load with _open_file_like(f, 'rb') as opened_file: File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 231, in _open_file_like return _open_file(name_or_buffer, mode) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 212, in init super(_open_file, self).init(open(name, mode)) PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt' Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes File "/usr/lib/python3/dist-packages/apport/init.py", line 5, in from apport.report import Report File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in import apport.fileutils File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 27, in from apport.packaging_impl import impl as packaging File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in import apt File "/usr/lib/python3/dist-packages/apt/init.py", line 36, in apt_pkg.init_system() apt_pkg.Error: E:Error reading the CPU table Original exception was: Traceback (most recent call last): File "pytorchexample.py", line 8, in
alexnet = torch.load("alexnet-pretrained.pt") File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 699, in load with _open_file_like(f, 'rb') as opened_file: File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 231, in _open_file_like return _open_file(name_or_buffer, mode) File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 212, in init super(_open_file, self).init(open(name, mode)) PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Can someone help me?
@Camelliavin At which step exactly does it fail?
Did you do this step: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-input-files
Did you do this step: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-manifest-file
It's important to (a) download the file alexnet-pretrained.pt and to (b) mark it as Gramine-encrypted file ({ path = "/alexnet-pretrained.pt", uri = "file:alexnet-pretrained.pt", type = "encrypted" }).
Does the file alexnet-pretrained.pt exist on your machine? Maybe you accidentally deleted it or something. Can you show the output of ls -la alexnet-pretrained.pt? Can you show the output of head alexnet-pretrained.pt?
Facing the same issue. Could someone please help?
~/examples/pytorch-confidential$ ls -la alexnet-pretrained.pt
-rw-rw-r-- 1 jashan jashan 246964224 Apr 6 12:07 alexnet-pretrained.pt`
Error:
Traceback (most recent call last):
File "pytorchexample.py", line 8, in <module>
alexnet = torch.load("alexnet-pretrained.pt")
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 791, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 271, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 252, in __init__
super().__init__(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module>
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 27, in <module>
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 36, in <module>
apt_pkg.init_system()
apt_pkg.Error: E:Error reading the CPU table
Original exception was:
Traceback (most recent call last):
File "pytorchexample.py", line 8, in <module>
alexnet = torch.load("alexnet-pretrained.pt")
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 791, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 271, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 252, in __init__
super().__init__(open(name, mode))
PermissionError: [Errno 13] Permission denied: 'alexnet-pretrained.pt'
Did you encrypt alexnet-pretrained.pt after you downloaded it? In other words, did you run this exactly: https://gramine.readthedocs.io/en/latest/tutorials/pytorch/index.html#preparing-input-files
Yes
/examples/pytorch-confidential$ head alexnet-pretrained.pt
����2�PF82�雠�r�0��<���'�FZ�(r�dJ>�-U��9�ȋ�q>h}�]�T 1��>���əK�{�VeRSrD����.�i��DB���(��Hi� ﳇ����gM��
�����mb)�G���*G�s˱���_�D��Kke�|
�TՎ�������n�����%*�o{F6;�6m�m��F�n�b����P�8�?Av,븖�U��t�f�$&*��p�6�t�"$�H(��}�K*8�#���I�T TiY���)^�לz"O)Q�_y�k�9'�O�����L�~�<�
aP�4�<(*���(�ڿ�l��/R� x���s�!�'�0���y��~�0q-�d]����ҾK�{/I�5i^�t����m��{�z�p��B}Dw��ێ�s8���ȏOI-0y�Ox3[���H$�������
�Ti9TS�3��6���Nr=ĞX�;
�vs�wr̎H�N���X��I�cǶ��Q�jz�7�����U�j�!NBO���� 2�9_���9 �T@A���
�cY?'I�!��Fnk�9��e!�M���Ч���2;I�2�{h����.�iB�J��ϼڛ��~�>@ �鶱��b$�� ��%
?AjԽ��!�Ks%m��A���]��Τ9Z�G��̓J�]��A��!a"�S6�mƺ��Ԗ���*ҳ#�]-/��j⬉P��>��+�)&6��5�� ڄr9��P#i�(Abxn���g#������ \D�Z�w���$�*�e'\ �9y{��
N����Gq�6+
r��9
�~�� ���c�:S���ޓ�)��]��/4.�B�
This is my manifest template
# PyTorch manifest template
loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"
loader.log_level = "{{ log_level }}"
loader.env.LD_LIBRARY_PATH = "/lib:/usr/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"
# Restrict the maximum number of threads to prevent insufficient memory
# issue, observed on CentOS/RHEL.
loader.env.OMP_NUM_THREADS = "8"
loader.insecure__use_cmdline_argv = true
loader.insecure__use_host_env = true
loader.pal_internal_mem_size = "128M"
fs.mounts = [
{ uri = "file:{{ gramine.runtimedir() }}", path = "/lib" },
{ uri = "file:{{ arch_libdir }}", path = "{{ arch_libdir }}" },
{ uri = "file:/usr", path = "/usr" },
{ uri = "file:/etc", path = "/etc" },
{ uri = "file:{{ pillow_path }}", path = "{{ pillow_path }}" },
{ type = "tmpfs", path = "/tmp" },
{ path = "/classes.txt", uri = "file:classes.txt", type = "encrypted" },
{ path = "/input.jpg", uri = "file:input.jpg", type = "encrypted" },
{ path = "/alexnet-pretrained.pt", uri = "file:alexnet-pretrained.pt", type = "encrypted" },
{ path = "/result.txt", uri = "file:result.txt", type = "encrypted" },
]
# PyTorch loads its pre-trained models from here
# Add below uncommented line to fs.mounts array if you want to use torchvision.model.alexnet(pretrained=True)
# { type = "chroot", uri = "file:{{ env.HOME }}/.cache/torch", path = "{{ env.HOME }}/.cache/torch" }
sgx.enclave_size = "4G"
sgx.max_threads = 32
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.nonpie_binary = true
sgx.remote_attestation = "dcap"
loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "localhost:4433"
sgx.trusted_files = [
"file:{{ gramine.libos }}",
"file:{{ entrypoint }}",
"file:{{ gramine.runtimedir() }}/",
"file:{{ arch_libdir }}/",
"file:/usr/{{ arch_libdir }}/",
"file:{{ python.stdlib }}/",
"file:{{ python.distlib }}/",
"file:{{ pillow_path }}",
"file:{{ python.get_path('stdlib', vars={'installed_base': '/usr/local'}) }}/",
"file:pytorchexample.py",
"file:pytorchexample.py",
"file:ssl/ca.crt"
# Uncomment line below if you want to use torchvision.model.alexnet(pretrained=True)
# "file:{{ env.HOME }}/.cache/torch/checkpoints/alexnet-owt-4df8aa71.pth",
]
sgx.allowed_files = [
]
# Gramine optionally provides patched OpenMP runtime library that runs faster inside SGX enclaves
# (add `-Dlibgomp=enabled` when configuring the build). Uncomment the line below to use the patched
# library. PyTorch's SGX perf overhead decreases on some workloads from 25% to 8% with this patched
# library. Note that we need to preload the library because PyTorch's distribution renames
# libgomp.so to smth like libgomp-7c85b1e2.so.1, so it's not just a matter of searching in the
# Gramine's Runtime path first, but a matter of intercepting OpenMP functions.
# loader.env.LD_PRELOAD = "/lib/libgomp.so.1"
Yes
How did you encrypt it? The binary form you copy-pasted doesn't look like proper Gramine FS encryption... (It should start with GRAFS in the first 5 bytes.)
Ah, I just checked, it's because head shows binary files in a weird way. Could you instead check with vim alexnet-pretrained.pt or less alexnet-pretrained.pt or some editor like this?
Sorry my bad I was using different wrap key for encryption and starting the server.