Limiting the size of a compound structure / enum that can be browsed by the reader

[Nabellaleen]

My input data is quite complex, but always has a format where structures are preceded by a byte giving the length of the structure to come.

Within a compound structure, the last fields are sometimes not supplied. The only way to know that reading is complete is to know that all the bytes indicated upstream have been consumed, and that the next structure has started.

Here is an example :

#[test]
fn test_uncomplete_structure() {
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Data {
        #[deku(read_all)]
        pub blocks: Vec<Block>,
    }
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Block {
        pub size: u8,
        pub data: BlockType,
    }
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    #[deku(id_type = "u8")]
    enum BlockType {
        #[deku(id = 1)]
        Block1(Block1),
        #[deku(id = 2)]
        Block2(Block2),
    }
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Block1 {
        pub field1_size: u8,
        pub field1: u16,
        pub field2_size: u8,
        pub field2: u64,
    }
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Block2;

    let buffer = &[
        // 1st Block, type 1
        4, // Size: 4
            1, // ID: 1
                2, 0x12, 0x34, // Field 1, size 2
                // No Field 2
        // 2nd Block, type 1
        6, // Size: Y
            1, // ID: 2
                1, 0x56, // Field 1, size 1 (casted into u16)
                2, 0x78, 0x9A // Field 2, size 2
    ];

    let (_rest, val) = Data::from_bytes((buffer, 0)).unwrap();
    assert_eq!(val.blocks.len(), 2);
    assert_eq!(val.blocks[0].size, 4);
    assert_eq!(val.blocks[1].size, 6);
}

How could I handle this case ? Is there a way to handle the second field of the first block ?

[Nabellaleen]

I've found a solution, by implementing a custom reader :

fn read_with_size<T, R: std::io::Read + std::io::Seek>(reader: &mut Reader<R>, size: usize) -> Result<T, DekuError> 
    where T: for<'a> DekuContainerRead<'a>
{
    let max_size = core::mem::size_of::<T>();
    let mut buf = vec![0; max_size];
    let buf: &mut [u8] = &mut buf;
    let ret = reader.read_bytes(size, buf)?;
    let (_rest, block) = match ret {
        ReaderRet::Bytes => {
            T::from_bytes((buf, 0))?
        },
        _ => return Err(DekuError::Parse("Unexpected result reading size bytes: got bits".into())),
    };
    Ok(block)
}

I use it this way :

#[derive(Debug, PartialEq)]
#[deku_derive(DekuRead)]
struct Block {
    pub size: u8,
    pub id: u8,
    #[deku(ctx = "*id,*size-1")] // size-1 to remove the ID size
    pub data: BlockType,
}
#[derive(Debug, PartialEq)]
#[deku_derive(DekuRead)]
#[deku(ctx = "id: u8, size: u8", id = "id")]
enum BlockType {
    #[deku(id = 1)]
    Block1(
        #[deku(reader = "read_with_size(deku::reader, size as usize)")]
        Block1
    ),
    #[deku(id = 2)]
    Block2(
        #[deku(reader = "read_with_size(deku::reader, size as usize)")]
        Block2
    ),
}

[Nabellaleen]

I think it could be nice to be able to :

• use bytes attribute, with a value from an expression
• use bytes_read attribute with structures implementing DekuRead

[wcampbell0x2a]

Hey! I added the bytes support for arbitrary bytes number from any token in this MR: https://github.com/sharksforarms/deku/pull/489

I solved your issue with the use of Option, skip, cond, and the new bytes. Although I admit this would be very verbose with structs with more then just a couple fields...

use deku::prelude::*;

fn main() {
    env_logger::init();
    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Data {
        #[deku(read_all)]
        pub blocks: Vec<Block>,
    }

    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Block {
        pub size: u8,
        pub id: u8,
        #[deku(ctx = "*id, *size as usize -1")] // size-1 to remove the ID size
        pub data: BlockType,
    }

    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    #[deku(ctx = "id: u8, size: usize", id = "id")]
    enum BlockType {
        #[deku(id = 1)]
        Block1(#[deku(ctx = "size")] Block1),
        #[deku(id = 2)]
        Block2(Block2),
    }

    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    #[deku(ctx = "size: usize")]
    struct Block1 {
        pub field1_size: u8,
        #[deku(bytes = "*field1_size as usize")]
        pub field1: u16,
        #[deku(skip, cond = "size <= 4")]
        pub field2_size: Option<u8>,
        #[deku(skip, cond = "size <= 4", bytes = "field2_size.unwrap() as usize")]
        pub field2: Option<u64>,
    }

    #[derive(Debug, PartialEq)]
    #[deku_derive(DekuRead)]
    struct Block2;

    let buffer = &[
        // 1st Block, type 1
        4, // Size: 4
        1, // ID: 1
        2, 0x12, 0x34, // Field 1, size 2
        // No Field 2
        // 2nd Block, type 1
        6, // Size: Y
        1, // ID: 1
        1, 0x56, // Field 1, size 1 (casted into u16)
        2, 0x78, 0x9A, // Field 2, size 2
    ];

    let (_rest, val) = Data::from_bytes((buffer, 0)).unwrap();
    assert_eq!(val.blocks.len(), 2);
    assert_eq!(val.blocks[0].size, 4);
    assert_eq!(val.blocks[1].size, 6);
}