apache
RANGER-4640: Trino ranger plugin for 433 snapshot
What changes were proposed in this pull request?
Changes made to support Trino Ranger Plugin with 433 version. Also made ranger compatible by adding nashorn-core to agent-commons to support build it with java 11 and java 17.
How was this patch tested?
Yes this is been deployed with our prod env and is working fine for more than an month now.
@shreyas-dview - this PR includes binary file ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz. Can you please remove this file and update the PR?
Would 17 be the minimum JDK version after this patch? Given many existing plugins require JDK8 support, it is critical to be able to build with earlier version JDK as well.
@mneethiraj removed ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz, thanks for pointing that out.
Would 17 be the minimum JDK version after this patch? -> if you look at pom.xml changes I have removed trino-plugin from default and jdk-11 profile , so if anyone wants to build trino-plugin they have to use -Pranger-jdk17 profile
if you look at pom.xml changes I have removed trino-plugin from default and jdk-11 profile , so if anyone wants to build trino-plugin they have to use -Pranger-jdk17 profile
@shreyas-dview - build fails with the following error due to changes to agents-common module. Can you please review and update?
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ ranger-plugins-common --- [INFO] Compiling 250 source files to /home/build/Apache/git/ranger/agents-common/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/build/Apache/git/ranger/agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java:[26,40] error: cannot access ClassFilter [INFO] 1 error [INFO] -------------------------------------------------------------
Nashron is not available from jdk15+. You can exclude from the build and use graalvm for jdk-17.
The dependency is already added in the plugins-common module, you can enable that dependency and try to build.
ZooKeeper 3.5 has a transitive dependency on a zookeeper-jute artifact.
./distro/src/main/assembly/plugin-trino.xml should have
"
The latest version of Trino supports the update statement. vim ./ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java
246 //kimdy 247 @Override 248 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set<String> updatedColumnNames) { 249 try { 250 activatePluginClassLoader(); 251 systemAccessControlImpl.checkCanUpdateTableColumns(securityContext, table, updatedColumnNames); 252 } finally { 253 deactivatePluginClassLoader(); 254 } 255 }
vim ./plugin-trino/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java
631 // kimdy 632 @Override 633 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set<String> updatedColumnNames){ 634 for (RangerTrinoResource res : createResource(table, updatedColumnNames)) { 635 if (!hasPermission(res, securityContext, TrinoAccessType.UPDATE)) { 636 LOG.debug("RangerSystemAccessControl.checkCanUpdateTableColumns(" + table.getSchemaTableName().getTableName() + ") denied"); 637 AccessDeniedException.denyUpdateTableColumns(table.getSchemaTableName().getTableName(), updatedColumnNames); 638 } 639 } 640 }
// UDPATE 추가 923 enum TrinoAccessType { 924 CREATE, DROP, SELECT, INSERT, DELETE, USE, ALTER, ALL, GRANT, REVOKE, SHOW, IMPERSONATE, EXECUTE, UPDATE; 925 }
- 나의 빌드 환경 java -version openjdk version "17.0.7" 2023-04-18 LTS OpenJDK Runtime Environment Zulu17.42+19-CA (build 17.0.7+7-LTS) OpenJDK 64-Bit Server VM Zulu17.42+19-CA (build 17.0.7+7-LTS, mixed mode, sharing)
mvn -v Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39) Maven home: /opt/apps/maven Java version: 17.0.7, vendor: Azul Systems, Inc., runtime: /opt/apps/zulu17.42.19-ca-jdk17.0.7-linux_x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"
mvn clean package -DskipTests -P ranger-trino-plugin,-linux -am -pl distro,plugin-trino,ranger-trino-plugin-shim,agents-installer,credentialbuilder
-
Lastly, ranger's group policy doesn't seem to apply well to trino. The role policy applies well to trino.
-
I'm an old school poor IT guy who's not familiar with GIT, so I don't know what form and procedure I should use to submit a proposal. I hope this was of some help. Your efforts have been of great help to me. thank you!
@shreyas-dview - this PR includes binary file ranger-3.0.0-SNAPSHOT-trino-plugin.tar.gz. Can you please remove this file and update the PR?
Would 17 be the minimum JDK version after this patch? Given many existing plugins require JDK8 support, it is critical to be able to build with earlier version JDK as well.
I had resolved the problem by maven-tool-chains plugin (https://stackoverflow.com/questions/12414209/compile-maven-module-with-different-java-version), but my ranger version is 2.3.
Give me some time over the weekend, will try to resolve the issues (if exists) or will answer them with proper explanation.
ZooKeeper 3.5 has a transitive dependency on a zookeeper-jute artifact. ./distro/src/main/assembly/plugin-trino.xml should have "org.apache.zookeeper:zookeeper-jute:jar:${zookeeper.version}"
The latest version of Trino supports the update statement. vim ./ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java
246 //kimdy 247 @OverRide 248 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames) { 249 try { 250 activatePluginClassLoader(); 251 systemAccessControlImpl.checkCanUpdateTableColumns(securityContext, table, updatedColumnNames); 252 } finally { 253 deactivatePluginClassLoader(); 254 } 255 }
vim ./plugin-trino/src/main/java/org/apache/ranger/authorization/trino/authorizer/RangerSystemAccessControl.java
631 // kimdy 632 @OverRide 633 public void checkCanUpdateTableColumns(SystemSecurityContext securityContext, CatalogSchemaTableName table, Set updatedColumnNames){ 634 for (RangerTrinoResource res : createResource(table, updatedColumnNames)) { 635 if (!hasPermission(res, securityContext, TrinoAccessType.UPDATE)) { 636 LOG.debug("RangerSystemAccessControl.checkCanUpdateTableColumns(" + table.getSchemaTableName().getTableName() + ") denied"); 637 AccessDeniedException.denyUpdateTableColumns(table.getSchemaTableName().getTableName(), updatedColumnNames); 638 } 639 } 640 }
// UDPATE 추가 923 enum TrinoAccessType { 924 CREATE, DROP, SELECT, INSERT, DELETE, USE, ALTER, ALL, GRANT, REVOKE, SHOW, IMPERSONATE, EXECUTE, UPDATE; 925 }
- 나의 빌드 환경 java -version openjdk version "17.0.7" 2023-04-18 LTS OpenJDK Runtime Environment Zulu17.42+19-CA (build 17.0.7+7-LTS) OpenJDK 64-Bit Server VM Zulu17.42+19-CA (build 17.0.7+7-LTS, mixed mode, sharing)
mvn -v Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39) Maven home: /opt/apps/maven Java version: 17.0.7, vendor: Azul Systems, Inc., runtime: /opt/apps/zulu17.42.19-ca-jdk17.0.7-linux_x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"
mvn clean package -DskipTests -P ranger-trino-plugin,-linux -am -pl distro,plugin-trino,ranger-trino-plugin-shim,agents-installer,credentialbuilder
- Lastly, ranger's group policy doesn't seem to apply well to trino. The role policy applies well to trino.
- I'm an old school poor IT guy who's not familiar with GIT, so I don't know what form and procedure I should use to submit a proposal. I hope this was of some help. Your efforts have been of great help to me. thank you!
@origin0099 I have tried your way,but the "Permissions" had no "Update" option. Could you help me?
I omitted the part where I had to manually modify RANGER's BACK-END DB. I'm using POSTGRESQL and used DBeaver.
I hope it helps you as it helped me.
I omitted the part where I had to manually modify RANGER's BACK-END DB. I'm using POSTGRESQL and used DBeaver.
I hope it helps you as it helped me.
@origin0099 Thanks for your reply! I modify the file agents-common/src/main/resources/service-defs/ranger-servicedef-trino.json and re-compile ranger. Now I have solved the problem.
Hi,
Thanks so much for this contribution, It helped me a lot. I have tested it with Trino 433 and Trino server is up and running and there seems to be an issue in Trino ranger raudits initialization when using Elastic search, so audits are not sent to ranger.
I am going to look more into this issue and will report my findings.
Elastic search version used : 7.16.2 Trino version: 433 Ranger: 2.4.0
Below is the error I am getting:
024-01-30T13:32:13.562Z INFO main stdout INFO - Connecting to ElasticSearch: User:ranger, http://es-internal.infra:80/ranger_audits
2024-01-30T13:32:13.895Z INFO main stdout WARN - request [POST http://es-internal.infra:80/ranger_audits/_open?master_timeout=30s&ignore_throttled=false&ignore_unavailable=false&expand_wildcards=closed&allow_no_indices=true&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.17.16-2b23fa076334f8d4651aeebe458a955a2ae23218 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2024-01-30T13:32:13.899Z INFO main stdout ERROR - Can't connect to ElasticSearch server: User:ranger, http://es-internal.infra:80/ranger_auditsjava.lang.BootstrapMethodError: bootstrap method initialization exception
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:188)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:312)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
at org.elasticsearch.client.IndicesClient.open(IndicesClient.java:614)
at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.newClient(ElasticSearchAuditDestination.java:267)
at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.getClient(ElasticSearchAuditDestination.java:187)
at org.apache.ranger.audit.destination.ElasticSearchAuditDestination.init(ElasticSearchAuditDestination.java:101)
at org.apache.ranger.audit.provider.AuditProviderFactory.init(AuditProviderFactory.java:183)
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:234)
at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl.<init>(RangerSystemAccessControl.java:120)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl.<init>(RangerSystemAccessControl.java:69)
at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl$$FastClassByGuice$$1345336.GUICE$TRAMPOLINE(<generated>)
at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControl$$FastClassByGuice$$1345336.apply(<generated>)
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:33)
at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
at com.google.inject.Guice.createInjector(Guice.java:87)
at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:262)
at org.apache.ranger.authorization.trino.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53)
at io.trino.spi.security.SystemAccessControlFactory.create(SystemAccessControlFactory.java:30)
at io.trino.security.AccessControlManager.createSystemAccessControl(AccessControlManager.java:198)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Collections$2.tryAdvance(Collections.java:5073)
at java.base/java.util.Collections$2.forEachRemaining(Collections.java:5081)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.trino.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:167)
at io.trino.server.Server.doStart(Server.java:162)
at io.trino.server.Server.lambda$start$0(Server.java:91)
at io.trino.$gen.Trino_433____20240130_133147_1.run(Unknown Source)
at io.trino.server.Server.start(Server.java:91)
at io.trino.server.TrinoServer.main(TrinoServer.java:38)
Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type interface org.apache.http.Header; not a subtype of implementation type interface org.apache.http.NameValuePair
at java.base/java.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:273)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:340)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
... 57 common frames omitted
Hello @tarekabouzeid, How did you build the ranger plugin, and have you setup TLS on elasticsearch ? I was getting connection error initially with ranger when there is TLS enabled on elasticsearch. I have setup ranger with elasticsearch and trino 433, you can take a look once https://github.com/nil1729/trino-ranger-noob
Hi @nil1729 Thank you for sharing, I guess its related to different JDK versions between Trino and Ranger, which version have you used to build ranger and which one you are using to run Trino ?
For me, I build ranger using : maven:3.8.5-openjdk-17-slim, it was built fine but ES was not working in Trino. When I tried to build ranger using : maven:3.9.6-eclipse-temurin-17-focal, built failed
I got this error while building,
#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[53,36] error: cannot find symbol
#7 191.6 [ERROR] symbol: class TimeValue
#7 191.6 [ERROR] location: package org.elasticsearch.common.unit
#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[54,40] error: cannot find symbol
#7 191.6 [ERROR] symbol: class XContentType
#7 191.6 [ERROR] location: package org.elasticsearch.common.xcontent
#7 191.6 [ERROR] /opt/ranger/embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/ElasticSearchIndexBootStrapper.java:[253,49] error: cannot find symbol
Trino is using Temurin JDK in its official docker image.
Hi @tarekabouzeid, I built the ranger locally with the JDK version openjdk-17.0.8 and maven version 3.6.3. There was some error when building on MAC due to the shadow.h header file not being found. Then I used those jars to build docker image and tested trino with docker only