Bulk writer error with java 24: UnsupportedOperationException: getSubject is not supported

[opiolartb]

On java 24 local bulk writer fails with

java.lang.UnsupportedOperationException: getSubject is not supported

in

[error] 	at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
[error] 	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
[error] 	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3884)
[error] 	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3874)
[error] 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3662)
[error] 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
[error] 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
[error] 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:541)
[error] 	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
[error] 	at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
[error] 	at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:273)
[error] 	at io.milvus.bulkwriter.Buffer.persistParquet(Buffer.java:156)
[error] 	at io.milvus.bulkwriter.Buffer.persist(Buffer.java:120)
[error] 	at io.milvus.bulkwriter.LocalBulkWriter.flush(LocalBulkWriter.java:132)
[error] 	at io.milvus.bulkwriter.LocalBulkWriter.lambda$commit$0(LocalBulkWriter.java:105)

Our hotfix for now is to add

configuration.set("fs.file.impl.disable.cache", "true");

in Buffer.java

Configuration configuration = new Configuration();
configuration.set("fs.file.impl.disable.cache", "true");

[yhmo]

I didn't see "fs.file.impl.disable.cache" in hadoop doc, it could be a flag to avoid multiple threads sharing the same cached object. If so, it indicates the LocalBulkWriter.flush() is called at the same time by multiple threads.

Is this error reproducible? Could you show me your client-side operations?

[opiolartb]

It's not about flush operation, it is related to securit manager removal in Java 24:

https://docs.oracle.com/en/java/javase/24/security/security-manager-is-permanently-disabled.html

That previous stack trace was from Milvus SDK 2.5.3, here is the stack trace for 2.5.8:

java.lang.UnsupportedOperationException: getSubject is not supported
    at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3884)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3874)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3662)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:541)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
    at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:273)
    at io.milvus.bulkwriter.writer.ParquetFileWriter.initWriter(ParquetFileWriter.java:59)
    at io.milvus.bulkwriter.writer.ParquetFileWriter.<init>(ParquetFileWriter.java:40)
    at io.milvus.bulkwriter.BulkWriter.createWriterByType(BulkWriter.java:132)
    at io.milvus.bulkwriter.BulkWriter.newFileWriter(BulkWriter.java:120)
    at io.milvus.bulkwriter.BulkWriter.<init>(BulkWriter.java:95)
    at io.milvus.bulkwriter.LocalBulkWriter.<init>(LocalBulkWriter.java:49)

Hadoop currently does not support new java (https://issues.apache.org/jira/browse/HADOOP-19486)

The hotfix works because of this line in hadoop:

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L551

[opiolartb]

And yes it is reproducible, just try to use LocalBulkWriter on java 24, no special steps needed

[yhmo]

I think it is ok to set "fs.file.impl.disable.cache = true" in the Buffer.java. Do you think there is any risk?

[opiolartb]

No, I think it's fine. In the current version it would be in ParquetFileWriter.java. Just hoping to have it fixed on Milvus side, so we don't have to hack it on our side.

[yhmo]

We don't intend to release new versions for v2.4.x, except for critical defects. I will add the line in the next minor version of v2.5.x. Do you think it makes sense?

[opiolartb]

Sure, that would be perfect, thanks

[yhmo]

A new version v2.5.9 has already been released with this fix.