SOLR-16433: Security Manager prevents Solr SQL from working

[risdenk]

trafficstars

https://issues.apache.org/jira/browse/SOLR-16433

[dsmiley]

Is there a test? I'm not insisting; just wondering if you thought about it. I suspect maybe it's because the security manager in tests allows access to CWD (maybe?) but not in a real-world scenario. If so, this is the kind of thing we'd ideally test via Docker; a real integration test.

[madrob]

If so, this is the kind of thing we'd ideally test via Docker; a real integration test

Or BATS. https://github.com/apache/solr/blob/main/solr/packaging/test/test_modules.bats#L35

[risdenk]

I'm out for a few weeks due to personal reasons - I would be grateful if someone can help push this a little further. If not I'll look at it when I get back to things.

[risdenk]

Well @madrob thanks for the pointer - weirdly https://github.com/apache/solr/blob/main/solr/packaging/test/test_modules.bats#L35 doesn't fail. I would expect it to :( I'll keep digging.

I suspect maybe it's because the security manager in tests allows access to CWD (maybe?) but not in a real-world scenario.

even if CWD is allowed, I'd be surprised if "../calcite-test-dataset" or "../../calcite-test-dataset" is allowed in tests... I don't know why we don't see this in tests yet.

[risdenk]

I don't know how to test this w/ BATS - since we have mixed security policies in build dirs :/

➜  solr git:(main) ✗ tail -f solr/packaging/build/test-output/solr-home/logs/solr-8983-console.log | grep -F calcite-tes
t
access: access allowed ("java.io.FilePermission" "../calcite-test-dataset" "read")
access: access allowed ("java.io.FilePermission" "../../calcite-test-dataset" "read")

The canonical path for ../calcite-test-dataset is /Users/risdenk/repos/apache/solr/solr/packaging/build/solr-10.0.0-SNAPSHOT/calcite-test-dataset and for ../../calcite-test-dataset is /Users/risdenk/repos/apache/solr/solr/packaging/build/calcite-test-dataset

I'm running ./gradlew integrationTests --tests test_modules.bats in /Users/risdenk/repos/apache/solr/ and added -a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005" -Djava.security.debug=access to test_modules.bats

Based on the security.policy file:

• ../calcite-test-dataset is allowed by permission java.io.FilePermission "${solr.install.dir}", "read,write,delete,readlink"; permission java.io.FilePermission "${solr.install.dir}${/}-", "read,write,delete,readlink"; since solr.install.dir=/Users/risdenk/repos/apache/solr/solr/packaging/build/solr-10.0.0-SNAPSHOT
• TODO ../../calcite-test-dataset is allowed by `` since =

[risdenk]

wow so I tracked this down to a potentially really ugly security policy bug...

https://github.com/apache/solr/blob/main/solr/server/etc/security.policy#L199

permission java.io.FilePermission "${solr.data.home}", "read,write,delete,readlink";
permission java.io.FilePermission "${solr.data.home}${/}-", "read,write,delete,readlink";

solr.data.home is set to "" which as far as I can tell means allow read under /.

I found this by adding -a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005" -Djava.security.debug=access,policy to the test_modules.bats

when I commented out the solr.data.home line Solr fails to start since it can't read/write temp files. I added in

// tmpdir
permission java.io.FilePermission "${java.io.tmpdir}", "read,write";
permission java.io.FilePermission "${java.io.tmpdir}${/}-", "read,write,delete";

and this reproduces what SOLR-16433 said it would.

So solr.data.home needs to be set or we need to figure out how to unset it completely to ensure that the security policy doesn't think "" is ok.

[risdenk]

So solr.data.home needs to be set or we need to figure out how to unset it completely to ensure that the security policy doesn't think "" is ok.

I think I found the bug in bin/solr that is setting solr.data.home to "". I'll put up a PR soon since its separate from this effort.

[risdenk]

Created https://issues.apache.org/jira/browse/SOLR-16457 and https://github.com/apache/solr/pull/1066

[risdenk]

Thanks @dsmiley and @madrob for prompting for an integration test. My fix didn't work and was found by integration tests after fixing the security policy. I'm pushing up a change soon building on https://github.com/apache/solr/pull/1066

[sonatype-lift[bot]]

:warning: 314 God Classes were detected by Lift in this project. Visit the Lift web console for more details.

[risdenk]

The docker test failure gave no details :)

2022-10-11T17:27:59.2856817Z > Task :solr:docker:dockerBuild UP-TO-DATE
2022-10-11T17:28:01.2858986Z 
2022-10-11T17:28:01.2860970Z > Task :solr:docker:testDocker
2022-10-11T17:28:01.2862326Z Testing Solr Image ID: sha256:3f5229c5954c280c03eefbed73a4603ad49a03d9702a109939cdefa81d201570 (sequentially)
2022-10-11T17:28:01.2864708Z Starting Solr Docker test: prometheus-exporter
2022-10-11T17:33:01.4867925Z 
2022-10-11T17:33:01.4868721Z > Task :solr:docker:testDocker FAILED
2022-10-11T17:33:01.5852441Z 
2022-10-11T17:33:01.5857229Z FAILURE: Build failed with an exception.
2022-10-11T17:33:01.5858359Z 76 actionable tasks: 3 executed, 73 up-to-date
2022-10-11T17:33:01.5858856Z 
2022-10-11T17:33:01.5859111Z * Where:
2022-10-11T17:33:01.5859898Z Build file '/home/runner/work/solr/solr/solr/docker/build.gradle' line: 523
2022-10-11T17:33:01.5860288Z 
2022-10-11T17:33:01.5860510Z * What went wrong:
2022-10-11T17:33:01.5861260Z Execution failed for task ':solr:docker:testDocker'.
2022-10-11T17:33:01.5861815Z > There was a failure while executing work items
2022-10-11T17:33:01.5862503Z    > A failure occurred while executing TestDockerImageTask$SingleTestAction
2022-10-11T17:33:01.5863555Z       > Docker test failure=1: Test: prometheus-exporter Output: /home/runner/work/solr/solr/solr/docker/build/test-results/prometheus-exporter
2022-10-11T17:33:01.5864035Z 
2022-10-11T17:33:01.5864259Z * Try:
2022-10-11T17:33:01.5865134Z Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
2022-10-11T17:33:01.5865604Z 
2022-10-11T17:33:01.5865900Z * Get more help at https://help.gradle.org
2022-10-11T17:33:01.5866234Z 
2022-10-11T17:33:01.5866827Z BUILD FAILED in 5m 8s

need to figure out how to run locally.

[risdenk]

Error running locally ./gradlew -p solr/docker testDocker:

java.security.AccessControlException: access denied ("java.io.FilePermission" "/opt/solr-10.0.0-SNAPSHOT/server/start.ini" "read")
	at java.base/java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.base/java.security.AccessController.checkPermission(Unknown Source)
	at java.base/java.lang.SecurityManager.checkPermission(Unknown Source)
	at java.base/java.lang.SecurityManager.checkRead(Unknown Source)
	at java.base/sun.nio.fs.UnixPath.checkRead(Unknown Source)
	at java.base/sun.nio.fs.UnixPath.toRealPath(Unknown Source)
	at org.eclipse.jetty.start.config.DirConfigSource.<init>(DirConfigSource.java:106)
	at org.eclipse.jetty.start.config.JettyBaseConfigSource.<init>(JettyBaseConfigSource.java:34)
	at org.eclipse.jetty.start.BaseHome.<init>(BaseHome.java:142)
	at org.eclipse.jetty.start.Main.processCommandLine(Main.java:281)
	at org.eclipse.jetty.start.Main.main(Main.java:75)

So removing the wide open permissions is finding some interesting things.

[madrob]

I remember rubbing into this with @markrmiller i think we left a comment in the security policy about it?

[risdenk]

I remember rubbing into this with @markrmiller i think we left a comment in the security policy about it?

don't see a comment in there. but the issue is /opt/solr vs /opt/solr-10.0.0-SNAPSHOT - Jetty is trying to hit the canonical URL - the permissions are on the symlink /opt/solr for jetty.home. This is easily fixable in Docker - set SOLR_SERVER_DIR to /opt/solr-10.0.0-SNAPSHOT/server (replace 10.0.0-SNAPSHOT w/ version) instead of /opt/solr

I am testing a fix locally on this.

[risdenk]

SOLR_SERVER_DIR="/opt/solr-${SOLR_VERSION}/server" \ added to handle Jetty home looking at canonical path

[risdenk]

integration test is actually working - see failure in https://github.com/apache/solr/pull/1066 and this PR here builds on https://github.com/apache/solr/pull/1066 and fixes it