pygradle
pygradle copied to clipboard
linkedin
Running integTests in pygradle-plugin directory modifies python libraries and causes python startup to fail
Python is installed via brew (python2.7) and python binary is used from /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7
Python 2.7.14 (default, Jan 6 2018, 12:16:16)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from _struct import *
>>>
drwxr-xr-x 127 <user> admin 4.0K Jan 9 08:39 encodings
drwxr-xr-x 13 <user> admin 416B Jan 9 08:39 config
Now, running tests from inside pygradle-plugin directory:
pygradle-plugin$ ../gradlew integTest
leads to the python libraries being overwritten and the folders are empty.
drwxr-xr-x 2 <user> admin 64B Jan 9 08:43 lib-dynload
drwxr-xr-x 2 <user> admin 64B Jan 9 08:43 encodings
drwxr-xr-x 2 <user> admin 64B Jan 9 08:43 config
python2.7$ python
Python 2.7.14 (default, Jan 6 2018, 12:16:16)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from _struct import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named _struct
This is always reproducible by running the integTest from pygradle-plugin directory.
Infact, if you just run PythonPluginIntegrationTest the issue is reproducible, so it does not require the whole set of tests to run.
I was able to triage it further to get to the root cause:
So for the tests, a temporary folder is created using Junit via DefaultProjectLayoutRule.groovy. The virtualenv is created in the temporary folder and we see the below:
/private/var/folders/z_/k8__6n_d10s6zd775lq749vh0000gr/T/junit2069112546178614649/foo/build/venv/lib/python3.6 <---- the temporary folder created for the test, and looking at the build directory in there, we see
lrwxr-xr-x 1
A lot of python libraries are symlinked. When the tests are done, junit cleans up the directory and it ends up destroying the directories that are symlinked as well thereby cleaning up the python libraries that are installed in the system.
@prachetaa Can you check what version of Gradle you are running when this happens? There was at least one release of Gradle 2.x that had this issue (the cleanup would resolve the symlinks and wipe things under them erroneously). We were less affected by it internally, because it didn't have permission to remove Python files owned by a root. In your case, I suspect the files are owned by you since they are installed by brew. The resolution for us was to just start using the next version of Gradle 2.x+1 and it didn't happen any more.
Anyway, let us know the Gradle version you're running with. This shouldn't be happening and it's not really an error in PyGradle, but rather Gradle itself.
Yeah, this is definitely due to the fact that the python installed is owned by me rather than root as it was installed with brew.
I am using Gradle 4.2. I do not believe the issue is due to gradle, this happens only while running integTests of pygradle-plugin and not while using pygradle. So it looks like the issue is with JUnit wiping out the TemporaryFolder that was created and the symlinks to python libraries being wiped out as well as they were inside the TemporaryFolder.
I was able to work around the issue by deploying the following workaround in pygradle/pygradle-plugin/src/integTest/groovy/com/linkedin/gradle/python/plugin/testutils/DefaultProjectLayoutRule.groovy
< void after() {
< /*
< * TODO : Please note that this is a hack, we are intentionally not using
< * junit TemporaryFolder's after method as pygradle creates symbolic links
< * from the temporary directory to libraries in python, which then in turn
< * deletes the libraries and causes python to freak out. Hence, we are using
< * apache common's delete to work around this until there is a fix.
< *
< * https://github.com/linkedin/pygradle/issues/179
< */
< FileUtils.deleteDirectory(tempFolder.getRoot().getAbsoluteFile());
< //It's useful to comment this out if you need to look at the test env
< //tempFolder.after()
< }
> @Override
> @SuppressWarnings("UnnecessaryOverridingMethod")
> void after() {
> //It's useful to comment this out if you need to look at the test env
> //tempFolder.after()
> }
OK. If you didn't use Gradle 2.x then we need to take a look at this. Thanks for posting the workaround. We'll resolve this issue when we apply the fix or let you know if we find any other root cause for this.
Nice find!
@prachetaa would you be willing to make a PR for this and we'll pull it in?
