gramine
gramine copied to clipboard
gramineproject
Update the manifest defaults for importing numpy package
Signed-off-by: TejaswineeL [email protected]
Description of the changes
-test_numpy.py executable imports a package called Numpy -While executing test_numpy.py with SGX, it demands an extra space to load the package to enclave. -Increase the enclave size to 1G and internal pal memory size to 128M (for manifest). This resolves the insufficient space issue. -OMP_num_threads limits the number of Numpy threads creation. -During runtime, SGX requests for python local installation path and in turn the Numpy package location. -Add the relevant path in the SGX trusted files.
How to test this PR?
-run test-numpy.py file from the python examples with "gramine-sgx ./python scripts/test-numpy.py" -the .py file should successfully execute by importing numpy package with SGX
This change is 
CI-Examples/python/python.manifest.template line 52 at r1 (raw file):
When numpy package is installed using "sudo apt-get install python3-numpy", the package gets stored at /usr/lib/python3/dist-packages/. Also, when "import numpy" executes , it flashes an error because of some incompatibility : "cannot import name 'multiarray' from 'numpy.core' (/usr/lib/python3/dist-packages/numpy/core/init.py)" [i made sure installed numpy package is not broken]
To solve the above error if we install numpy package using "pip install numpy" [which is a recommended way of installing a package], the package gets stored at "/usr/local/lib/python3.7/" That is where the newly added line to the trusted files comes to the picture. the path -"file:{{ python.get_path('stdlib', vars={'installed_base': '/usr/local'}) }}/" expands to "/usr/local/lib/python3.7/"
CI-Examples/python/python.manifest.template line 12 at r1 (raw file):
Numpy script is expected to work with the default configuration. without the line loader.env.OMP_NUM_THREADS = "4", we need to have bigger enclave size like 16G etc. as Threads spawned by Numpy go on consuming memory till a memory fault.
CI-Examples/python/python.manifest.template line 52 at r1 (raw file): Yes, after some steps I could import numpy package without the newly added line in the trusted files.
CI-Examples/python/python.manifest.template line 14 at r1 (raw file): I have more number of packages installed, thats why i had to use pal_internal_mem_size as 128M. As it works with 64M for you. I removed the line (as its default value is 64M)
After the suggestions, I tried the CI-examples on a fresh system and found that we don't need to make any changes to the manifest template in order to run the examples with SGX
However, adding OMP_NUM_THREADS was considered useful during the conversation @dimakuv kindly, let me know if I should go ahead and add OMP_NUM_THREADS line change or else we can cancel this PR
@TejaswineeL we need this changes for Ubuntu 22.04, CentOS and RHEL machines
https://github.com/gramineproject/gramine/pull/938#issuecomment-1315044132
@dimakuv any specific comment we need to add for loader.env.OMP_NUM_THREADS
how about a comment #To limit the number of threads created
