Changes of enum crashes storage

[JohannesRabauer]

Environment Details

• MicroStream Version: 05.00.02-MS-GA / 07.00.00-MS-GA
• JDK version: 17
• OS: Windows 10

Describe the bug

Given i have following enum:

public enum ExampleEnum
{
   FIRST_ENUM,
   SECOND_ENUM
}

Now i can save an instance of the following class:

public class PersistedClass
{
   ExampleEnum myInstance = ExampleEnum.SECOND_ENUM;
   ...
}

All well and good. What i did was, that i changed the Enum to following:

public enum ExampleEnum
{
   FIRST_ENUM,
   THIRD_ENUM,
   SECOND_ENUM
}

Granted this seems silly, but only in this example. In productive code this isn't something special.

If i reload my saved instance of PersistedClass and save it again, i get following error:

Exception in thread "main" one.microstream.persistence.exceptions.PersistenceExceptionConsistencyObjectId: Inconsistent Object id. Registered: 1000000000015841270, passed: 1000000000015841271
	at one.microstream.persistence.internal.DefaultObjectRegistry.internalValidateObjectNotYetRegistered(DefaultObjectRegistry.java:661)
	at one.microstream.persistence.internal.DefaultObjectRegistry.internalAddGetCheck(DefaultObjectRegistry.java:587)
	at one.microstream.persistence.internal.DefaultObjectRegistry.internalAddGet(DefaultObjectRegistry.java:669)
	at one.microstream.persistence.internal.DefaultObjectRegistry.optionalRegisterObject(DefaultObjectRegistry.java:502)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.getEffectiveInstance(BinaryLoader.java:321)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.getBuildInstance(BinaryLoader.java:658)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.lookupObject(BinaryLoader.java:340)
	at one.microstream.persistence.binary.types.Binary.collectElementsIntoArray(Binary.java:1613)
	at one.microstream.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.updateState(BinaryHandlerNativeArrayObject.java:134)
	at one.microstream.persistence.binary.java.lang.BinaryHandlerNativeArrayObject.updateState(BinaryHandlerNativeArrayObject.java:1)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.buildInstances(BinaryLoader.java:466)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.build(BinaryLoader.java:401)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.get(BinaryLoader.java:844)
	at one.microstream.persistence.binary.types.BinaryLoader$Default.loadRoots(BinaryLoader.java:888)
	at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.loadExistingRoots(EmbeddedStorageManager.java:340)
	at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.initialize(EmbeddedStorageManager.java:359)
	at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:248)
	at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:1)
	at one.microstream.storage.embedded.types.EmbeddedStorageFoundation.start(EmbeddedStorageFoundation.java:241)
	at one.microstream.storage.embedded.types.EmbeddedStorageFoundation.start(EmbeddedStorageFoundation.java:209)
...

How can i fix this behaviour?

I already discussed this very problem with @hg-ms (through internal issue MPS-14) but we found no suitable solution.

[hg-ms]

same as #381

[hg-ms]

What happened is that you created two versions of the ExampleEnum that have a conflict in their ordinals: The first version has the constant “SECOND_ENUM” with ordinal 1 The second version has constant “THIRD_ENUM” with ordinal 1

When updating the persisted old version to the new one the automatic legacy type handling unfortunately doesn’t recognize that conflict and allows to persist that “bad” enums without complaining. PR #435 addresses that issue and will introduce an exception before the storage gets corrupted.

Unfortunately, there is no simple option to repair such a corrupted storage automatically!

Repairing such a storage is not trivial and requires manual actions and quite a lot of expertise of Microstream…

Here are some hints: At first there are two kinds of corrupted states:

1. The enum has been changed but old values are not updated (stored again): Such a storage can still be loaded.
2. The old values are already updated (stored again): The storage can’t be loaded anymore and throws PersistenceExceptionConsistencyObjectId

In case one there are two options to repair the storage:

• export the storage to CVS, edit the affected enums and reimport the storage again.
• use a custom legacy type handler that bypasses the default legacy type handling of the affected enum and corrects the enum ordinals as needed by the application and perform the update (store again) after loading. Case two can be repaired by using a custom BinaryTypeHandler implementation that bypasses the loading issues and does the repairs.

Such a TypeHandler may look like:

public class BinaryHandlerRepair extends AbstractBinaryHandlerCustom<Aufzaehlung> {

	private static final long BINARY_OFFSET_ENUM_NAME = 0;
	private static final long BINARY_OFFSETORDINAL = 8;
	
	public BinaryHandlerRepair() {
		
		//The field definition must exactly match to the enum type definition in the storage type dictionary.
		
		super(Aufzaehlung.class,
			X.List(
			PersistenceTypeDefinitionMemberEnumConstant.New(Aufzaehlung.ONE.name()),
			PersistenceTypeDefinitionMemberEnumConstant.New(Aufzaehlung.THREE.name()),
			PersistenceTypeDefinitionMemberEnumConstant.New(Aufzaehlung.TWO.name()),
			PersistenceTypeDefinitionMemberFieldGenericSimple.New(
				String.class.getName(),
				"java.lang.Enum",
				"name",
				String.class,
				!String.class.isPrimitive(),
				BinaryPersistence.resolveFieldBinaryLength(String.class),
				BinaryPersistence.resolveFieldBinaryLength(String.class)
			),
			PersistenceTypeDefinitionMemberFieldGenericSimple.New(
				int.class.getName(),
				"java.lang.Enum",
				"ordinal",
				int.class,
				!int.class.isPrimitive(),
				BinaryPersistence.resolveFieldBinaryLength(int.class),
				BinaryPersistence.resolveFieldBinaryLength(int.class)
			)
		));
	}

	@Override
	public Object[] collectEnumConstants() {
		return Persistence.collectEnumConstants(this);
	}
	
	@Override
	public void iterateLoadableReferences(final Binary data, final PersistenceReferenceLoader iterator) {
		// not required here
	}

	@Override
	public void updateState(final Binary data, final Aufzaehlung instance, final PersistenceLoadHandler handler) {
		// not required here
	}

	@Override
	public boolean hasPersistedReferences() {
		return false;
	}

	@Override
	public boolean hasVaryingPersistedLengthInstances() {
		return false;
	}

	@Override
	public void store(final Binary data, final Aufzaehlung instance, final long objectId, final PersistenceStoreHandler<Binary> handler) {
		
		final long contentLength = Binary.referenceBinaryLength(1) + Integer.BYTES;
		
		data.storeEntityHeader(contentLength, this.typeId(), objectId);
		data.store_int(BINARY_OFFSETORDINAL, instance.ordinal());
		data.store_long(BINARY_OFFSET_ENUM_NAME, handler.apply(instance.name()));
	}

	@Override
	public Aufzaehlung create(final Binary data, final PersistenceLoadHandler handler) {
		
		/* Everything must be loaded here to avoid the
		 * PersistenceExceptionConsistencyObjectId exception,
		 * updateState() is to late...
		 * 
		 * The way the enum constant name is loaded is a hack that may
		 * no more work in future Microstream versions.
		 * 
		 */
		
		final long oidName = data.read_long(BINARY_OFFSET_ENUM_NAME);
		final int ordinal = data.read_int(BINARY_OFFSETORDINAL);
		
		final String name = (String)handler.getPersister().getObject(oidName);
		
		final Aufzaehlung retVal = Aufzaehlung.valueOf(name);
					
		System.out.println(
			"---------------------------------------------------------------------" + "\n" +
			"oid :" + data.getBuildItemObjectId() + "\n" +
			"read name: " + name + " ,oid: " + oidName + "\n" +
			"read ordinal: " + ordinal + "\n" +
			"result: " +  retVal + "\n" +
			"---------------------------------------------------------------------"
		);
		
		return retVal;
	}
}

[JohannesRabauer]

With your help i could solve the first case

1. The enum has been changed but old values are not updated (stored again): Such a storage can still be loaded.

I loaded the old database with a BinaryLegacyTypeHandler and solved the issue. Thanks!