Add an optional logical offset to `BinaryReader`

[agatti]

Is your feature request related to a problem? Please describe.

Let BinaryReader have a logical offset to apply to the pointer index before reading.

I'm using Kaitai-Struct to generate complex data structure parsers, and I wrote a simple bridge class between BinaryReader and Kaitai's own intepretation of the same thing. Now, application code or other random files usually use either relative or zero-based offsets to reference other bits of data, but a non-trivial number of ROM images bake the absolute addresses in when it comes to data pointers. Manually-written parsers can accommodate for that, but generated parsers cannot really work since they do not have any idea of what's the start address to subtract from every offset.

Describe the solution you'd like

Add a getter/setter to BinaryReader to apply an optional offset to the pointer index, like getLogicalOffset/setLogicalOffset or similar. So for example something like:

BinaryReader reader = BinaryReader(provider, true);
reader.setLogicalOffset(0x10000000);
String name = reader.readAsciiString(0x10001000);

would read from offset 0x1000 from reader's provider and not from 0x10001000.

Describe alternatives you've considered

I originally wrote my own BinaryReader-derived class that overrode almost every method to keep track of the logical offset and that used to work, but sadly the changes in 01aadea22b968acf1d30260fae477b00ccfddc1a broke string reading for me. BinaryReader.readString and its associated methods are private and thus readAsciiString, readUnicodeString, readUtf8String, etc. cannot be made offset-aware. Granted, I can always copy and paste BinaryReader's implementation in my own code and change that, but I'd like to keep that as a last resort option if possible.

[dev747368]

Could you bake the logical offset magic into a ByteProvider wrapper?

[ghidra1]

Have you looked at using the BinaryReader.clone(long newIndex) method. A new reader instance would be used for anything needing a differerent "logical" offset.

BinaryReader reader = BinaryReader(provider, true);
BinaryReader reader2 = reader.clone(0x10000000);
String name = reader2.readAsciiString(0x10001000);

[agatti]

@dev747368 that's an option... I'll have to change a fair bit of code but I can deal with that.

@ghidra1 unless I'm doing things wrong here it doesn't work (code is in kotlin, but it shouldn't matter):

import ghidra.app.util.bin.BinaryReader
import ghidra.app.util.bin.ByteArrayProvider

fun main() {
    val reader1 =
        BinaryReader(
            ByteArrayProvider(byteArrayOf(0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08)),
            true
        )
    assert(reader1.readUnsignedInt(1L) == 0x05040302L) { "read at offset 1 failed." }
    val reader2 = reader1.clone(0x10000000)
    assert(reader2.readUnsignedShort(0x10000002L) == 0x0403) { "read at offset 0x10000002 failed" }
}

errors out with

Exception in thread "main" java.io.IOException: Invalid position, index: 268435458, max is: 8
	at ghidra.app.util.bin.ByteArrayProvider.assertValidIndex(ByteArrayProvider.java:125)
	at ghidra.app.util.bin.ByteArrayProvider.readBytes(ByteArrayProvider.java:138)
	at ghidra.app.util.bin.BinaryReader.readShort(BinaryReader.java:724)
	at ghidra.app.util.bin.BinaryReader.readUnsignedShort(BinaryReader.java:735)
	at TestKt.main(test.kt:12)
	at TestKt.main(test.kt)

That said, if there's no plan to add such a feature to BinaryReader I'll rework things on my end, no big deal. Thanks to both of you for the help!

[dev747368]

Yeah, it seems a bit too niche to add to BinaryReader itself.

I would copy/paste ByteProviderWrapper (about 130 lines of code), and add a setter to change the offset so you can manipulate the index value however you need in the various methods before calling the real provider.

Next step would to be create a something like:

public class SuperSpecialBinaryReader extends BinaryReader {
  private SuperSpecialByteProvider specialProvider;

  public SuperSpecialBinaryReader(ByteProvider provider, boolean isLittleEndian) {
    super( new SuperSpecialByteProvider(provider), isLittleEndian);

    this.specialProvider = (SuperSpecialByteProvider)getByteProvider();
  }

  public void setLogicalOffset(long offset) {
    specialProvider.setLogicalOffset(offset);
  }
 ...other methods...
}

and that should be it.

You could even get by without storing a typecast SuperSpecialByteProvider if you only have a method or two that needs it and are willing to cast it there before calling into the ByteProvider.

[ghidra1]

unless I'm doing things wrong here it doesn't work

@agatti sorry, I was confused by what you were asking for. It sounds like you are trying to use an alternative offset (i.e., logical) instead of the real offset. As @dev747368 suggested you would need a new BinaryReader implementation which would need to override all methods which specify an offset and translate a specified "logical" offset to an actual offset which would be conveyed to super.

Alternative to what @dev747368 suggested (BinaryReader uses term index instead of offset):

public class SuperSpecialBinaryReader extends BinaryReader {

  private long logicalBaseAdjustment;

  /**
   * Construct logical binary reader with an initial pointer index of 0.
   * @param provider byte source
   * @param isLittleEdian true if little-endian conversion, else big-endian
   * @param logicalIndex initial logical index which corresponds to pointer index of 0
   */
  public SuperSpecialBinaryReader(ByteProvider provider, boolean isLittleEndian, long logicalIndex) {
    super(provider, isLittleEndian);
    setLogicalIndex(logicalIndex);
  }

  /**
   * Set the logical index which corresponds to the current physical index position
   * ( see setPointerIndex(index) and getPointerIndex() ).  This method should be 
   * invoked after current pointer index has been established.
   * @param logicalIndex logical index value which correspond to current position
   */
  public void setLogicalIndex(long logicalIndex) {
    logicalBaseAdjustment = logicalIndex - getPointerIndex();
  }

  @Override
  public String readAsciiString(long logicalIndex) throws IOException {
    return super.readAsciiString(logicalIndex - logicalBasAdjustment);
  }

 // ...other methods...  will have to watchout for new BinaryReader methods which may get added over time
}