fast-serialization icon indicating copy to clipboard operation
fast-serialization copied to clipboard

Published 20 hours ago • verified publisher icon RuedigerMoeller

@Conditional for String field

Open golivax opened this issue 7 years ago • 4 comments

Is it possible to use the @Conditional annotation in a String field? My class structure is something like this:

class X{
    List<A> listOfAs;
    String name;
}

class A{
    X parent;
    String veryLongString;
    List<B> listOfBs;
}

I would like to add @Conditional to the veryLongString field in class A. I tried to follow your examples, but I'm getting an exception. More specifically, just to try it out, I tried creating a call back that would always return false (i.e., it should never skip the field).

FSTObjectInput.ConditionalCallback conditionalCallback = new FSTObjectInput.ConditionalCallback() {
   @Override
   public boolean shouldSkip(Object halfDecoded, int streamPosition, Field field) {
    	return false;
    }
};

The exception is as follows:

2017-07-31 11:12:39 ERROR - Exception log
java.io.IOException: java.lang.RuntimeException: class not found CLASSNAME: loader:sun.misc.Launcher$AppClassLoader@18b4aac2
	at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:247) ~[fst-2.52.jar:?]
	at ca.queensu.sail.ah.framework.serial.Serializer.deserialize(Serializer.java:198) [classes/:?]
	at ca.queensu.sail.ah.framework.serial.Serializer.deserialize(Serializer.java:151) [classes/:?]
	at ca.queensu.sail.ah.framework.serial.Serializer.main(Serializer.java:327) [classes/:?]
Caused by: java.lang.RuntimeException: class not found CLASSNAME: loader:sun.misc.Launcher$AppClassLoader@18b4aac2
	at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:235) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:190) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTClazzNameRegistry.decodeClass(FSTClazzNameRegistry.java:173) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.coders.FSTStreamDecoder.readClass(FSTStreamDecoder.java:478) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readClass(FSTObjectInput.java:937) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:347) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:92) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:92) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTMapSerializer.instantiate(FSTMapSerializer.java:78) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:311) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:245) ~[fst-2.52.jar:?]
	... 3 more
Caused by: java.lang.ClassNotFoundException: 
	at java.lang.Class.forName0(Native Method) ~[?:1.8.0_131]
	at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_131]
	at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:197) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:190) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTClazzNameRegistry.decodeClass(FSTClazzNameRegistry.java:173) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.coders.FSTStreamDecoder.readClass(FSTStreamDecoder.java:478) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readClass(FSTObjectInput.java:937) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:347) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:92) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:92) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.serializers.FSTMapSerializer.instantiate(FSTMapSerializer.java:78) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:501) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:712) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:566) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:374) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:331) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:311) ~[fst-2.52.jar:?]
	at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:245) ~[fst-2.52.jar:?]
	... 3 more

I don't exactly understand why it is complaining about the loader:sun.misc.Launcher$AppClassLoader class. Maybe my scenario violates the pre-conditions you mentioned (to be honest, I don't exactly understand what they mean, sorry):

only applicable with unshared streams and full preregistration of all classes possibly referenced in a skipped subgraph.

If I change the callback to always return true, I get another exception (I can post it if necessary). What I would really want to accomplish is skipping the A.veryLongString field based on the value of A.parent.name.

In reality, the problem is not that veryLongString is actually that long. What happens is that I have a bunch of A objects with a bunch of X objects, and I only need veryLongString for very few As...

Thanks in advance. Please let me know if there is anything I could do to get around it or if you need more information about anything.

golivax avatar Jul 31 '17 15:07 golivax

Issue is, that @conditional might mess up the order classes occur in a serialized stream, I did not thought of this side effect when I introduced this annotations (like 5 years ago :) ). Anyway as its not covered by test it might simply be broken in the current release..

RuedigerMoeller avatar Jul 31 '17 15:07 RuedigerMoeller

I see, that makes sense. Do you think there's anything I can do get away with it? Like a custom serializer or something? Accomplishing this would speed up my deserialization process a lot (based on tests I've done). Thanks!

golivax avatar Jul 31 '17 15:07 golivax

Yes, you could use a custom serializer, but it will be somewhat tricky: You need to be able to calculate the endposition of the stream after having written it to the stream. At read time depending on your condition you either read the long string or call inputStream.getCodec().skip(XX) to just ignore the String. Unfortunately its hard to calculate the actual stream length of a given String in advance (optimizations, UTF-8), so you need a custom string-write loop.

pseudocode

    public void writeObject(FSTObjectOutput out, Object toWrite, FSTClazzInfo clzInfo, FSTClazzInfo.FSTFieldInfo referencedBy, int streamPosition)
            throws IOException;
{
  out.defaultWriteObject(mylist);
  byte b[] = longString.getBytes("UTF-8");
  out.writeInt(b.length());
  out.write(b);
}

    public void readObject(FSTObjectInput in, Object toRead, FSTClazzInfo clzInfo, FSTClazzInfo.FSTFieldInfo referencedBy)
        throws Exception;
{
  mylist = in.defaultReadObject();
  int len = in.readInt();
  if ( CONDITION ) {
    in.getCodec().skip(len);
  } else {
    byte b[] = new byte[len];
    in.read(b);
    mylongstring = new String(b,"UTF-8");
  }
}

RuedigerMoeller avatar Jul 31 '17 16:07 RuedigerMoeller

remove @Conditional as its limitations are too hard to understand / handle

RuedigerMoeller avatar Sep 15 '17 16:09 RuedigerMoeller