pxf icon indicating copy to clipboard operation
pxf copied to clipboard

Published 20 hours ago • verified publisher icon greenplum-db

pxf support read multi files

Open vicigel opened this issue 4 years ago • 6 comments

Does pxf support wildcard pattern to load data from s3 storage(or s3 like)? when I use methods below, all encountered error.

  1. LOCATION ('pxf://BUCKET_NAME/dirname/?PROFILE=s3:text&COMPRESSION_CODEC=gzip') FORMAT 'TEXT' (delimiter=E',');

  2. LOCATION ('pxf://BUCKET_NAME/dirname/*?PROFILE=s3:text&COMPRESSION_CODEC=gzip') FORMAT 'TEXT' (delimiter=E',');

  3. LOCATION ('pxf://BUCKET_NAME/dirname/*.gz?PROFILE=s3:text&COMPRESSION_CODEC=gzip') FORMAT 'TEXT' (delimiter=E',');

As far as I know, gphdfs support wildcard pattern to load multi files, is that my wrong method to use or really not support ?

vicigel avatar Jan 07 '21 02:01 vicigel

@vicigel yes, PXF supports wildcards. What is the error you are getting? Can you share your log files from your $PXF_CONF/logs/ directory? Also, what version of PXF are you using? Which S3 storage are you accessing?

frankgh avatar Jan 07 '21 13:01 frankgh

pxf version:

[gpadmin@HOSTNAME ~]$ /usr/local/greenplum-db/pxf/bin/pxf version PXF version 5.15.1

minio-site.xml

[gpadmin@HOSTNAME ~]$ cat pxf/servers/default/minio-site.xml

fs.s3a.endpoint MY_ENDPOINT fs.s3a.access.key MY_KEY fs.s3a.secret.key MY_SECRET fs.s3a.fast.upload true fs.s3a.path.style.access true

files in my company's s3 storage

aws --endpoint=MY_ENDPOINT s3 ls s3://BUCKET_NAME/DIR_NAME/ 2021-01-07 12:42:14 36 2.gz 2021-01-07 12:42:21 36 23.gz

external table with specific file

wzq=# select * from wzqtest1; id | b ----+---- 3 | 4 10 | 20 5 | 8 (3 rows)

wzq=# \d wzqtest1 External table "public.wzqtest1" Column | Type | Modifiers --------+---------+----------- id | integer | b | integer | Type: readable Encoding: UTF8 Format type: text Format options: delimiter ',' null '\N' escape '' External options: {} External location: pxf://BUCKET_NAME/DIR_NAME/2.gz?PROFILE=s3:text&COMPRESSION_CODEC=gzip

external table with wildcard pattern

wzq=# \d wzqtest3; External table "public.wzqtest3" Column | Type | Modifiers --------+---------+----------- id | integer | b | integer | Type: readable Encoding: UTF8 Format type: text Format options: delimiter ',' null '\N' escape '' External options: {} External location: pxf://BUCKET_NAME/DIR_NAME/*?PROFILE=s3:text&COMPRESSION_CODEC=gzip Execute on: all segments

wzq=# select * from wzqtest3; ERROR: remote component error (500) from '127.0.0.1:5888': Type Exception Report Message javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: d8f58e98-5158-11eb-943f-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b), S3 Extended Request ID: d31efe88f54299a02981434e097d507b: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: d8f58e98-5158-11eb-943f-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b) Description The server encountered an unexpected condition that prevented it from fulfilling the request. Exception javax.servlet.ServletException: javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: d8f58e98-5158-11eb-943f-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b), S3 Extended Request ID: d31efe88f54299a02981434e097d507b: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: d8f58e98-5158-11eb-943f-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b) (libchurl.c:985) (seg0 slice1 ip:port pid=210771) (libchurl.c:985) CONTEXT: External table wzqtest3, line 1 of file pxf://BUCKET_NAME/DIR_NAME/*?PROFILE=s3:text&COMPRESSION_CODEC=gzip wzq=#

wzq=# \d wzqtest4 External table "public.wzqtest4" Column | Type | Modifiers --------+---------+----------- id | integer | b | integer | Type: readable Encoding: UTF8 Format type: text Format options: delimiter ',' null '\N' escape '' External options: {} External location: pxf://BUCKET_NAME/DIR_NAME/?PROFILE=s3:text&COMPRESSION_CODEC=gzip Execute on: all segments

wzq=# wzq=# wzq=# select * from wzqtest4; ERROR: remote component error (500) from '127.0.0.1:5888': Type Exception Report Message javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541d5e36-515c-11eb-b887-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b), S3 Extended Request ID: d31efe88f54299a02981434e097d507b: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541d5e36-515c-11eb-b887-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b) Description The server encountered an unexpected condition that prevented it from fulfilling the request. Exception javax.servlet.ServletException: javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541d5e36-515c-11eb-b887-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b), S3 Extended Request ID: d31efe88f54299a02981434e097d507b: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541d5e36-515c-11eb-b887-a6fdda2a7003; S3 Extended Request ID: d31efe88f54299a02981434e097d507b) (libchurl.c:985) (seg4 slice1 ip:port pid=306423) (libchurl.c:985) CONTEXT: External table wzqtest4, line 1 of file pxf://BUCKET_NAME/DIR_NAME/?PROFILE=s3:text&COMPRESSION_CODEC=gzip wzq=#

wzq=# \d wzqtest5; External table "public.wzqtest5" Column | Type | Modifiers --------+---------+----------- id | integer | b | integer | Type: readable Encoding: UTF8 Format type: text Format options: delimiter ',' null '\N' escape '' External options: {} External location: pxf://BUCKET_NAME/DIR_NAME/*.gz?PROFILE=s3:text&COMPRESSION_CODEC=gzip Execute on: all segments

wzq=# select * from wzqtest5; ERROR: remote component error (500) from '127.0.0.1:5888': Type Exception Report Message javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb) Description The server encountered an unexpected condition that prevented it from fulfilling the request. Exception javax.servlet.ServletException: javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb) (libchurl.c:985) (seg0 slice1 ip:port pid=240301) (libchurl.c:985) CONTEXT: External table wzqtest5, line 1 of file pxf://BUCKET_NAME/DIR_NAME/*.gz?PROFILE=s3:text&COMPRESSION_CODEC=gzip wzq=#

log of one pxf node: localhost.log

Jan 08, 2021 10:50:55 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2668) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2628) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:274) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:50:55 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2668) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2628) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:274) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:50:55 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2668) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2628) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:274) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:50:55 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listLocatedStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 541ddbae-515c-11eb-bfa7-ba2878504539; S3 Extended Request ID: 39250c44001325926b9c370c81b34620), S3 Extended Request ID: 39250c44001325926b9c370c81b34620 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2668) at org.apache.hadoop.fs.s3a.S3AFileSystem.listLocatedStatus(S3AFileSystem.java:2628) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:274) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:52:16 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:96) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1600) at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561) at org.apache.hadoop.fs.Globber.listStatus(Globber.java:77) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:235) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1976) at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:2490) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:266) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:52:16 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:96) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1600) at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561) at org.apache.hadoop.fs.Globber.listStatus(Globber.java:77) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:235) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1976) at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:2490) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:266) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:52:16 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:96) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1600) at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561) at org.apache.hadoop.fs.Globber.listStatus(Globber.java:77) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:235) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1976) at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:2490) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:266) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

Jan 08, 2021 10:52:16 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [PXF REST Service] in context with path [/pxf] threw exception [javax.servlet.ServletException: org.apache.hadoop.fs.s3a.AWSS3IOException: listStatus on s3a://BUCKET_NAME/DIR_NAME: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb)] with root cause com.amazonaws.services.s3.model.AmazonS3Exception: Invalid Argument (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 848b5c8a-515c-11eb-a7c8-f2602b052b9e; S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb), S3 Extended Request ID: 698be2b7b404e5825bba458679a333cb at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:870) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1067) at org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:548) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:119) at org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:96) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1600) at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561) at org.apache.hadoop.fs.Globber.listStatus(Globber.java:77) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:235) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1976) at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:2490) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:266) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) at org.greenplum.pxf.plugins.hdfs.utilities.PxfInputFormat.listStatus(PxfInputFormat.java:54) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getSplits(HdfsDataFragmenter.java:119) at org.greenplum.pxf.plugins.hdfs.HdfsDataFragmenter.getFragments(HdfsDataFragmenter.java:73) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:186) at org.greenplum.pxf.service.rest.FragmenterResource.access$100(FragmenterResource.java:62) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:128) at org.greenplum.pxf.service.rest.FragmenterResource$1.call(FragmenterResource.java:122) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) at com.google.common.cache.LocalCache.get(LocalCache.java:3953) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) at org.greenplum.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:122) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.greenplum.pxf.service.servlet.SecurityServletFilter.lambda$doFilter$0(SecurityServletFilter.java:146) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.greenplum.pxf.service.servlet.SecurityServletFilter.doFilter(SecurityServletFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

are those enough to diagnose? if any other requirements needed, let me know, thanks very much!

vicigel avatar Jan 08 '21 03:01 vicigel

Hi @vicigel , I'm trying to reproduce the issue. However I am not able to reproduce the issue. Can you try the following:

  1. Look in the minio logs for any clues when running the query.
  2. Can you confirm the server name for your minio server? Was it configured under the default server?
  3. Are there any other files under your server configuration directory?
  4. Can you confirm the minio server version you are trying to access?

Thank you,

frankgh avatar Jan 08 '21 16:01 frankgh

@frankgh 2. the server is default server. ll /home/gpadmin/pxf/servers/default/ total 4 -rw-r--r-- 1 gpadmin gpadmin 641 Jan 6 15:22 minio-site.xml 3. no other files under the default server. 4. the s3 service is my company's self developed, compitable with s3 protocol, when using s3-site.xml, it does not provide the configuration of endpoint, but minio-site.xml does, so I use minio-site.xml, however, the storage is not minio service.

vicigel avatar Jan 09 '21 02:01 vicigel

@vicigel is there anything in the logs of your S3 service? Maybe try to correlate queries with log entries in your companies S3 service for some clues

frankgh avatar Jan 09 '21 16:01 frankgh

@frankgh ok, I will ask for help of related colleagues, thanks

vicigel avatar Jan 11 '21 02:01 vicigel