Shubham Tagra

Another instance with a LIMIT 10 query: ``` 2020-06-04T06:28:44.795Z ERROR 20200604_062840_00009_vupau.1.1-1-46 com.qubole.rubix.bookkeeper.BookKeeper Could not fetch FileStatus from remote file system for s3a://XXX/jane-presto-benchmark-test-data/XXX_orc/20190221_133439_00001_t4knf_d4294c0a-f465-44ed-8511-5a0f0c26ea71 java.lang.RuntimeException: java.lang.InterruptedException: sleep interrupted at io.prestosql.plugin.hive.util.RetryDriver.run(RetryDriver.java:154) at io.prestosql.plugin.hive.s3.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:631)...

`The fix for this is to let the Chains open and close stream themselves`: while this solves the issue, this can have impact on performance when streaming reads api is...

This will need thorough testing, with embedded mode and stand alone mode. And also some perf testing.

The exception show up as serailization exception: com.fasterxml.jackson.databind.JsonMappingException: No serializer found for class java.util.concurrent.TimeoutException and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: com.google.common.collect.Values[0]->com.facebook.presto.failureDetector.Stats["lastFailureException"])

It was found that the serialization exception was because of an issue in presto because of which all v1/node calls would fail. Nevertheless, we should make this change in PrestoClusterManger...

> We can keep discarded connections count too while returning a connection to the pool. yes, that should also be published.

There is LocalDataServer daemons as well whose metrics would be more useful in slaves Then there is service for async warmup c.c. @abhishekdas99

Isnt the problem more about finding why we wrote nothing but still set the cached status to true for these blocks? Do we have logs from RemoteReadRequestChain caching this data...

Sure @JamesRTaylor

There are two issues now around sharing the stream: this and https://github.com/qubole/rubix/issues/375. Even though the intent in sharing stream was that it would never be used in parallel but it...