Chronicle-Core

Chronicle-Core copied to clipboard

= Chronicle Core Chronicle Software :css-signature: demo :toc: macro :toclevels: 2 :icons: font

image:https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-core/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-core] image:https://javadoc.io/badge2/net.openhft/chronicle-core/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/index.html"] //image:https://javadoc-badge.appspot.com/net.openhft/chronicle-wire.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/chronicle-core] image:https://img.shields.io/github/license/OpenHFT/Chronicle-Core[GitHub] image:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link="https://chronicle.software/release-notes/"] image:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_Chronicle-Core&metric=alert_status[link="https://sonarcloud.io/dashboard?id=OpenHFT_Chronicle-Core"]

image::images/Core_line.png[width=20%]

toc::[]

== About

This library wraps up low level access to

• <<_system_properties_from_file,System properties from a file>>
• <<_off_heap_memory_access,Off Heap Memory Access>>
• <<_jvm_access_methods,JVM Access Methods>>
• <<_os_calls,OS Calls>>
• <<_resource_reference_counting,Resource Reference Counting>>
• <<_object_pools,Object Pools>>
• <<_class_local_caching,Class Local Caching>>
• <<_maths_functions,Maths Functions>> for casting types, rounding double, faster hashing.
• <<_serializable_lambdas,Serializable Lambdas>>
• <<_histogram,Histogram>> A high performance wide range histogram.
• <<_jlbh,JLBH>> Java Latency Benchmarking Harness.

== System properties from file

The Jvm class in its static initialiser will load system properties from a file called system.properties if it can find this file in the current directory. If it can't find this file in current directory it will look in .. too. This is a useful feature if you want to de-clutter your command line.

Chronicle Core class Jvm loads the file system.properties into the System's properties. To ensure it is loaded early enough, you can load in a static block

[source,java]

static { Jvm.init(); }

The choice of file to load can be overridden on the command line with -Dsystem.properties=my.properties

In link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/src/main/java/net/openhft/chronicle/core/Jvm.java[Jvm.java] it can be seen how to guarantee that JVM class is initialized before the system property is read. For example with Jvm.getInteger or Jvm.getLong.

A number of relevant system properties are listed in link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/systemProperties.adoc[systemProperties.adoc].

NOTE: If you specify system properties on the command line then they will override those in the system.properties file.

== ChronicleInit

The link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/src/main/java/net/openhft/chronicle/core/ChronicleInit.java[ChronicleInit] class allows the developer to hook in their own code to be run at startup before the Chronicle static initialisers are run. Chronicle static initialisers perform tasks such as loading system properties, so it is possible, for example, to override system properties using ChronicleInit. To this end, the system property "chronicle.init.runnable" should specify a fully qualified class name that will be run once before any system property is read by Chronicle code, allowing the class to set them to the desired values. The class should contain an empty static init() method that is called to trigger class load. The code should reside in a static block to be run once.

== Off Heap Memory Access

This allows you to access native memory using primitives and some thread safe operations.

[source,java]

Memory memory = OS.memory(); long address = memory.allocate(1024); try { memory.writeInt(address, 1); assert memory.readInt(address) == 1; final boolean swapped = memory.compareAndSwapInt(address, 1, 2); assert swapped; assert memory.readInt(address) == 2; } finally { memory.freeMemory(address, 1024); }

== JVM Access Methods

Check the JVM is running in debug mode

[source,java]

if (Jvm.isDebug()) { // running in debug.

Rethrow a checked exception as an unchecked one.

[source,java]

try { // IO operation } catch (IOException ioe) { throw Jvm.rethrow(ioe); }

Get a Field for a Class by name

[source,java]

Field theUnsafe = Jvm.getField(Unsafe.class, "theUnsafe"); Unsafe unsafe = (Unsafe) theUnsafe.get(null);

== OS Calls

Access to system calls

[source,java]

int processId = OS.getProcessId(); int maxProcessId = OS.getMaxProcessId(); int pageSize = OS.getPageSize(); boolean isWindows = OS.isWindows(); boolean is64bit = OS.is64Bit(); String hostname = OS.getHostName(); String username = OS.getUserName(); String targetDir = OS.getTarget(); // where is the target directory during builds.

Memory mapped files

[source,java]

FileChannel fc = new CleaningRandomAccessFile(fileName, "rw").getChannel(); // map in 64 KiB long address = OS.map(fc, MapMode.READ_WRITE, 0, 64 << 10); // use address OS.memory().writeLong(1024L, 0x1234567890ABCDEFL); // unmap memory region OS.unmap(address, 64 << 10);

== Deterministic Resource Management

Component which are closeable or reference counted can be released deterministically without waiting for a GC.

=== Closeable Resources

A Closeable resources has a simple lifecycle. It is open when created, and cannot be used once closed.

[source,Java]

public class AbstractCloseableTest {

@Test
public void close() {
    MyCloseable mc = new MyCloseable();
    assertFalse(mc.isClosed());
    assertEquals(0, mc.performClose);

    mc.throwExceptionIfClosed();

    mc.close();
    assertTrue(mc.isClosed());
    assertEquals(1, mc.performClose);

    mc.close();
    assertTrue(mc.isClosed());
    assertEquals(1, mc.performClose);
}

@Test(expected = IllegalStateException.class)
public void throwExceptionIfClosed() {
    MyCloseable mc = new MyCloseable();
    mc.close();
    mc.throwExceptionIfClosed();

}

@Test
public void warnAndCloseIfNotClosed() {
    Map<ExceptionKey, Integer> map = Jvm.recordExceptions();
    MyCloseable mc = new MyCloseable();
    mc.warnAndCloseIfNotClosed();
    Jvm.resetExceptionHandlers();
    assertEquals("Discarded without closing\n" +
                    "java.lang.IllegalStateException: net.openhft.chronicle.core.StackTrace: Created Here",
            map.keySet().stream()
                    .map(e -> e.message + "\n" + e.throwable)
                    .collect(Collectors.joining(", ")));
}

static class MyCloseable extends AbstractCloseable {
    int performClose;

    @Override
    protected void performClose() {
        performClose++;
    }
}

}

=== Resource Reference Counting

Use reference counting to deterministically release resources.

A reference counted resource can add reservations until closed.

[source,Java]

public class AbstractReferenceCountedTest {

@Test
public void reserve() {
    assertTrue(Jvm.isResourceTracing());
    MyReferenceCounted rc = new MyReferenceCounted();
    assertEquals(1, rc.refCount());

    ReferenceOwner a = ReferenceOwner.temporary("a");
    rc.reserve(a);
    assertEquals(2, rc.refCount());

    ReferenceOwner b = ReferenceOwner.temporary("b");
    rc.reserve(b);
    assertEquals(3, rc.refCount());

    try {
        rc.reserve(a);
        fail();
    } catch (IllegalStateException ignored) {
    }
    assertEquals(3, rc.refCount());

    rc.release(b);
    assertEquals(2, rc.refCount());

    rc.release(a);
    assertEquals(1, rc.refCount());
    assertEquals(0, rc.performRelease);

    rc.releaseLast();
    assertEquals(0, rc.refCount());
    assertEquals(1, rc.performRelease);
}

@Test
public void reserveWhenClosed() {
    MyReferenceCounted rc = new MyReferenceCounted();
    assertEquals(1, rc.refCount());

    ReferenceOwner a = ReferenceOwner.temporary("a");
    rc.reserve(a);
    assertEquals(2, rc.refCount());

    assertFalse(rc.isClosed());

    rc.closeable.close();

    assertEquals(2, rc.refCount());
    assertTrue(rc.isClosed());

    ReferenceOwner b = ReferenceOwner.temporary("b");
    try {
        rc.reserve(b);
        fail();
    } catch (IllegalStateException ignored) {
    }
    assertEquals(2, rc.refCount());

    assertFalse(rc.tryReserve(b));
    assertEquals(2, rc.refCount());

    rc.release(a);
    assertEquals(1, rc.refCount());
    assertEquals(0, rc.performRelease);

    rc.throwExceptionIfReleased();

    rc.releaseLast();
    assertEquals(0, rc.refCount());
    assertEquals(1, rc.performRelease);

    rc.throwExceptionBadResourceOwner();
    try {
        rc.throwExceptionIfClosed();

fail(); } catch (IllegalStateException ignored) {

    }
    try {
        rc.throwExceptionIfReleased();
        fail();
    } catch (IllegalStateException ignored) {

    }
}

@Test
public void throwExceptionBadResourceOwner() {
    MyReferenceCounted rc = new MyReferenceCounted();
    MyReferenceCounted rc2 = new MyReferenceCounted();
    rc.reserve(rc2);
    rc.throwExceptionBadResourceOwner();

    rc2.closeable.close();
    try {
        rc.throwExceptionBadResourceOwner();
        fail();
    } catch (IllegalStateException ignored) {
    }
    rc.release(rc2);
    rc.releaseLast();
}

@Test
public void throwExceptionIfClosed() {
    MyReferenceCounted rc = new MyReferenceCounted();
    rc.throwExceptionIfClosed();

    rc.closeable.close();
    try {
        rc.throwExceptionIfClosed();

       fail();
    } catch (IllegalStateException ignored) {

    }
}

static class MyReferenceCounted extends AbstractReferenceCounted {
    final AbstractCloseable closeable;
    int performRelease;

    public MyReferenceCounted() {
        this(new AbstractCloseableTest.MyCloseable());
    }

    public MyReferenceCounted(AbstractCloseable abstractCloseable) {
        super(abstractCloseable);
        closeable = abstractCloseable;
    }

    @Override
    protected void performRelease() {
        performRelease++;
    }
}

}

[source,java]

MappedFile mf = MappedFile.mappedFile(tmp, chunkSize, 0); MappedBytesStore bs = mf.acquireByteStore(chunkSize + (1 << 10));

assertEquals(2, mf.refCount()); assertEquals(3, bs.refCount()); assertEquals("refCount: 2, 0, 3", mf.referenceCounts());

mf.close(); assertEquals(2, bs.refCount()); assertEquals("refCount: 1, 0, 2", mf.referenceCounts()); bs2.releaseLast(); assertEquals(1, mf.refCount()); assertEquals(1, bs.refCount()); bs.releaseLast(); assertEquals(0, bs.refCount()); assertEquals(0, mf.refCount()); assertEquals("refCount: 0, 0, 0", mf.referenceCounts());

=== Releasing Resources

Releasing resources can be managed by starting the BACKGROUND_RESOURCE_RELEASER thread or alternatively it can be managed in a user defined thread. To start the BACKGROUND_RESOURCE_RELEASER thread, both system properties background.releaser and background.releaser.thread should be set to true. In this condition, the thread starts as a daemon thread and invokes BackgroundResourceReleaser.runReleaseResources().

If only background.releaser.thread is set to false, resources will still be queued for releasing, but they need to be released explicitly by calling BackgroundResourceReleaser.releasePendingResources().

If background.releaser is set to false regardless of background.releaser.thread, resources are not queued for release and release will be done synchronously (by calling the relevant close() function).

Calling BackgroundResourceReleaser.stop() releases pending resources and then stops the BACKGROUND_RESOURCE_RELEASER thread. To make sure the shutdown hook does not prevent classes from unloading, deregister the shutdown hook by calling PriorityHook.clear().

.Releasing Resources [%header,cols=3] |=== | background.releaser.thread | background.releaser | Release Behaviour | true | true | resources are queued and then released in the BACKGROUND_RESOURCE_RELEASER thread. | false | true | resources are queued but should be released in a user thread by calling BackgroundResourceReleaser.releasePendingResources(). | X | false | resources are not queued and are released synchronously. |===

== Object Pools

There is String and Enum object pools to turn a CharSequence into a String.

[source,java]

Bytes<?> b = Bytes.from("Hello World"); b.readSkip(6);

StringInterner si = new StringInterner(128); String s = si.intern(b); String s2 = si.intern(b); assertEquals("World", s); assertSame(s, s2);

== Class Local Caching

Add caching of a data structure for each class using a lambda

[source,java]

public static final ClassLocal<EnumInterner> ENUM_INTERNER = ClassLocal.withInitial(c -> new EnumInterner<>(c));

E enumValue = ENUM_INTERNER.get(enumClass).intern(stringBuilder);

== Maths Functions

Maths functions to support rounds

[source,java]

double a = 0.1; double b = 0.3; double c= Maths.round2(b - a); // 0.2 rounded to 2 decimal places

Checking type conversions

[source,java]

int i = Maths.toInt32(longValue);

== Serializable Lambdas

There is a number of FunctionalInterfaces you can utilise as method arguments. This allows implicitly making a lambda Serializable.

[source,java]

// in KeyedVisitable default <R> R applyToKey(K key, @NotNull SerializableFunction<E, R> function) {

// in code

String fullename = map.applyToKey("u:123223", u -> u.getFullName());

== Histogram

A high dynamic range histogram with tunable accuracy.

[source,java]

Histogram h = new Histogram(32, 4); long start = instance.ticks(), prev = start; for (int i = 0; i <= 1000_000_000; i++) { long now = instance.ticks(); long time = now - prev; h.sample(time); prev = now; } System.out.println(h.toLongMicrosFormat(instance::toMicros));

== JLBH

JLBH has moved home and now lives in its own project, see https://github.com/OpenHFT/JLBH[JLBH].

== Loop Block Monitor tool

The tool to summarise the thread stack traces is here.

net.openhft.chronicle.core.threads.MonitorProfileAnalyserMain