PatternLanguage
PatternLanguage copied to clipboard
WerWolv
[Suggestion] Attribute to make local variables within structs
In functions (and the global scope), u32 var = 123; is equivalent to u32 var; var = 123;. In pattern structs, u32 var = 123; creates a local variable (not part of the pattern data) that can be freely modified, whereas u32 var; var = 123; is simply forbidden (as u32 var; is parsed as part of the pattern data, and therefore must not be modified) - there is a functional difference between initialised and uninitialised variables/patterns.
However, arrays can't be initialised like that; neither u32 var[3] = {10,20,30};, u32 var[3] = [10,20,30];, u32 var[3] = 10,20,30;, nor u32 var[3] = 10; are allowed. This means that it's impossible to declare local array variables (not part of the pattern data) within pattern structs. I would've suggested "let us initialise arrays", but the same applies to structs (e.g. struct MyStruct { u32 value; }; MyStruct var = {0}; doesn't work), and I can't think of a good way to initialise structs while declaring them.
Currently, to get around that, you have to first make a section, then place an array or struct in that section, and then fill out the array/struct values. Doing this can be inconvenient, fills the Sections tab with many different sections if you e.g. create a section like this in a struct and use that struct 20 times, and makes the [[export]] attribute not display it in the Pattern Data tab (as you instead have to open and look within the respective section's own patterns).
Therefore, I suggest an attribute or a keyword to make a local variable even when it's not initialised. For example, u32 var[3] [[local]]; would create a local array with 3 zeroes, and struct MyStruct { u32 value; }; MyStruct var [[local]]; would create a local struct with value set to zero.
(For structs, care should be taken to ignore @ for its members, except when it's placed in a section created within that struct or a sub-struct (or just except when it's in any section at all?). Structs that do something like struct MyStruct { u32 value1; if (value1 > 10) u32 value2; } won't work as expected as value1 would always be zero at that time, but it's up to pattern makers to not use local structs that require being placed in memory.)
TL;DR: An attribute/keyword to force a variable to be local would be nice, even if it's not initialised (which is currently impossible for arrays/structs). All values that would've been read from (non-section) memory should probably default to 0.
Structs can be initialized the same way as any other variable. This creates local variables too.
import std.io as io;
struct Test_Local {
u8 f;
};
Test_Local tl;
tl.f=8;
struct Test {
Test_Local ro=tl;
u8 f=tl.f;
};
Test ns@0;
io::print("ns: {{ \n ro = {},\n f = {} \n }}",ns.ro,ns.f);
To initialize arrays you have to enclose them in a struct and use the same method.
copying an array:
import std.io as io;
struct Test_Local {
u8 f[4];
};
Test_Local tl;
tl.f[0]=8;
tl.f[1]=9;
tl.f[2]=-3;
tl.f[3]=6;
struct Test {
Test_Local ro=tl;
u8 h;
};
Test ns@0;
io::print("ns: {{ \n ro = {} \n }}",ns.ro);
Structs can be initialized the same way as any other variable. This creates local variables too. [Pattern code] [...]
I'm going to assume that tl and ns.ro are the same struct, not "copies" of the same struct. In that case, that wouldn't work if I want to make an arbitrary amount of "copies" (e.g. read X from a file, create an X-sized array of structs, have each of those structs contain a local, unique array/struct for some decrypted data).
As a "simplified" example...
[Click to expand]
fn DecompressImage(ref auto input_array,ref auto output_array) {
output_array[3] = 0xFF; //Pretend that there's more than just this
};
struct InnerStruct {
u32 var_a;
u16 var_b[3];
};
struct CompressedStruct {
u8 fake_rgba_image[64] = {0}; //Should be local
u8 data[16] [[hex::visualize("bitmap",fake_rgba_image,4,4)]];
InnerStruct decompressed_struct = {0} [[export]]; //Should be local, and visible in "Pattern Data"
decompressed_struct.var_a = data[0]*100;
decompressed_struct.var_b[0] = data[1]*100;
DecompressImage(data,fake_rgba_image);
};
struct MainStruct {
InnerStruct uncompressed_data;
le u32 compressed_count;
CompressedStruct compressed_data[compressed_count];
};
MainStruct mainstruct @ $;
I could make a section for each fake_rgba_image and decompressed_struct above, but it would be nice (and reduce line count) to have local, unique arrays/structs within pattern structs in the same way that local, unique variables are possible. Plus, if decompressed_struct is placed in a section, then it won't be shown in the same "Pattern Data" tab as the main file/patterns.
(InnerStruct could be split into two versions, with the other version renamed to InnerStruct_Local and having u32 var_a; changed to u32 var_a = 0; to make a "fake local" struct, but that would duplicate code and wouldn't work if the struct contains an array.)
I'm going to assume that tl and ns.ro are the same struct, not "copies" of the same struct. In that case, that wouldn't work if I want to make an arbitrary amount of "copies" (e.g. read X from a file, create an X-sized array of structs, have each of those structs contain a local, unique array/struct for some decrypted data).
They are not the same struct. If you modify one, the other is not modified. When you use the assignment operator the pattern language creates a clone of the right hand side which is totally independent of the original data. The only difference between custom types and built-in types is that there is no notation to create literals of custom types or arrays (as in u8 a[3]={5,8,2}).
To make the point clear, if you want to make a variable local you assign a value to it. Then you are free to change the values to whatever you want. If a variable is not local but placed in the data, its value cannot be changed without changing the input file itself.
You can create the array of structs with local struct members using the same struct 'initializer' for all of them. That tags the data contained inside the local structs as local and henceforth make the data writable. Then you can go one at a time and reassign to each of the local structs their own portion of the decrypted data independently of each other so you end up with the x-sized array of structs and each of those structs contain a local, unique array/struct for some decrypted data.
Sections are nothing but additional memory made available for use. Global and local variables also use memory in addition to the memory used by the input file, they are just more convenient and easier to use. Local variable will not show on the pattern data window unless you export it, that is true no matter what the data type is.
(InnerStruct could be split into two versions, with the other version renamed to InnerStruct_Local and having u32 var_a; changed to u32 var_a = 0; to make a "fake local" struct, but that would duplicate code and wouldn't work if the struct contains an array.)
Code duplication can't be avoided. when you write u32 var[3] = {10,20,30}; you are duplicating code. You create local variables, but you still need to fill the values which is duplicating the code of writing the original values.
Even if the structs contain arrays the same thing works as my second example shows. Here is an example that shows the entire process. The pragma example is the data used to run it.
#pragma example 45 67 0E 98 79 04 88 39 98 88 90 00 90 9E 0E 09 00 99 09 00 9E E0 90 90 90 09 00 90 99 00 90 90
import std.io as io;
//You want to write
/*
struct RealStruct {
u8 magic;
u8 version;
u8 size;
u8 local[4] [[local]];
u8 real[5];
};
*/
// so that you can write values to local array. You can accomplish that now doing this
struct ArrayForLocals {
u8 a[4];
}[[format("format_array")]];
fn format_array(auto a) {
return io::format("[{:#x}, {:#x}, {:#x},{:#x}]",a.a[0],a.a[1],a.a[2],a.a[3]);
};
// I created this struct only to display the values in hexadecimal
struct ArrayForRealData {
u8 a[5];
}[[format("format_array2")]];
fn format_array2(auto a) {
return io::format("[{:#x}, {:#x}, {:#x}, {:#x}, {:#x}]",a.a[0],a.a[1],a.a[2],a.a[3],a.a[4]);
};
ArrayForLocals init; //the values here are not important. they will be overwritten.
struct RealStruct {
u8 magic;
u8 version;
u8 size;
ArrayForLocals local=init; // mark this as local.
ArrayForRealData real;
};
RealStruct rs[4]@0;
// No we can overwrite the local part using the real data
for (u32 i=0,i<4,i+=1) {
for (u32 j=0,j<4,j+=1)
rs[i].local.a[j] = rs[i].real.a[j] % rs[i].real.a[4];
io::print("{}",rs[i]);
}
result
I: struct RealStruct { magic = 152, version = 136, size = 144, local = [0x0, 0x0, 0x5,0x5], real = [0x0, 0x90, 0x9e, 0xe, 0x9] }
I: struct RealStruct { magic = 0, version = 153, size = 9, local = [0x0, 0xe, 0x50,0x0], real = [0x0, 0x9e, 0xe0, 0x90, 0x90] }
I: struct RealStruct { magic = 144, version = 9, size = 0, local = [0x0, 0x9, 0x0,0x0], real = [0x90, 0x99, 0x0, 0x90, 0x90] }
I: Pattern exited with code: 0
I: Evaluation took 0.0076147s
Note that each of the local variables are different and depend only on the values of their real array version.
They are not the same struct. If you modify one, the other is not modified. [...]
You're correct. :thumbsup: I assumed that I'd need to do something special to make a separate copy of "x", not just MyStruct y = x;, but you're right.
(It's still inconvenient that "x" needs to be declared somewhere first, though. It's also tedious when it comes to arrays (where the array needs to be within a struct) - especially if the array needs to have 0x100000 elements in one case and only needs 0x40 elements in another case, like a texture file containing multiple mipmaps/image resolutions that could be decoded into RGBA8 for visualization.)
[...] Code duplication can't be avoided. when you write u32 var[3] = {10,20,30}; you are duplicating code. You create local variables, but you still need to fill the values which is duplicating the code of writing the original values. [...]
By "duplicating code", I meant...[Click to expand]
struct MyStruct {
u32 abc;
u16 def;
};
struct Test {
bool var1;
if (var1) //Just to avoid "MyStruct var3 = var2;" being a valid solution
MyStruct var2;
MyStruct var3 [[local]]; //Should be local
//var3.abc = 10; //This doesn't seem to work right now...
};
Test data @ $;
...versus...
struct MyStruct {
u32 abc;
u16 def;
};
struct MyStruct_Local {
u32 abc = 0;
u16 def = 0;
} [[hidden]];
struct Test {
bool var1;
if (var1) //Just to avoid "MyStruct var3 = var2;" being a valid solution
MyStruct var2;
MyStruct_Local var3; //Should be local
//var3.abc = 10; //This doesn't seem to work right now...
};
Test data @ $;
...where MyStruct_Local is essentially a duplicate of MyStruct that just adds = 0 to the variable definitions, in order for them to be local.
You really dont need to declare a new structure to make it local, this code would do it as well:
struct MyStruct {
u32 abc;
u16 def;
};
MyStruct local;
struct Test {
bool var1;
if (var1) //Just to avoid "MyStruct var3 = var2;" being a valid solution
MyStruct var2; // "MyStruct var3 = var2;" will make var3 local, but not var2. but you dont need that
MyStruct var3 =local; // do this and now var3 is local
var3.abc = 10; //This should work now
};
Test data @ $;
The reason your version is not working is because var3 is not local until it is assigned a value itself, not its members. The locality of a variable is not something inherited, but belongs to the variable instance only. If the variable is a built-in type, then you can simply assign some literal and you don't need a dummy variable to make it local, but you can also make built-in variables local that way. There is no notation for struct or array literals so they need a dummy variable, but the extra notation is very small.
Having to declare some dummy variable is trivial even for arrays because the dummy variable doesn't need to be assigned a value for it to be an effective local variable maker. The rest is identical either way. new values would need to be assigned whether you use [[local]] or =dummy so there is only minimal inconvenience. is you need to initialize arrays of different sizes, create a dynamic array template class with the size as the template argument and instantiate that.
Now that I think about it, the fact it doesn't work may be a bug. i need to look more into it
Details
Im sorry, setting var3.abc=10 in the array will never work because at that point there is no instance of Test so there is no local variable. what you need to do is assign it after placing the Test variable; i.e.
struct MyStruct {
u32 abc;
u16 def;
};
MyStruct local;
struct Test {
bool var1;
if (var1) //Just to avoid "MyStruct var3 = var2;" being a valid solution
MyStruct var2; // "MyStruct var3 = var2;" will make var3 local, but not var2. but you dont need that
MyStruct var3 =local; // do this and now var3 is local
// var3.abc = 10; //This should work now
};
Test data @ $;
data.var3.abc = 10;
You really dont need to declare a new structure to make it local, this code would do it as well: [...]
Yes, but I didn't know that that was possible when I made this issue, and the "It's still inconvenient that "x" needs to be declared somewhere first" part of my previous message still applies.
[...] There is no notation for struct or array literals so they need a dummy variable, but the extra notation is very small. [...]
The extra notation/dummy variable is still inconvenient, considering how you don't need a dummy variable to make local variables of built-in types.
(Someday, u32 var[3] = {10,20,30}; or u32 var[3] = {0}; might work for local arrays, but we'd still need dummy variables for local structs, unlike everything else. Ignoring that I didn't know "dummy variables work for this", that's what this issue is for.)
Also, for arrays, wrapping them in a (global, dummy) struct means that I can't set their size at will. I would either need a separate struct and (at least) dummy variable for every array size (or for certain thresholds), or use a single "one-size-fits-all" struct with an array that's big enough for the biggest possible scenario. For example, if I want local arrays for every power of two up to 65536, I'd need either to figure out which of 17 dummy variables (one for each array size) to use where applicable, or to use a 65536-sized array 17 times when I only actually want a total of 131071 entries combined (wasting a lot of memory - especially if the array size is actually 65536*65536 instead of just 65536).
Now that I think about it, the fact [that setting var3.abc=10 in the array] doesn't work may be a bug. [...]
From what I can tell, period accessors just don't work properly on the left-hand side of assignments (which might explain half of https://github.com/WerWolv/ImHex/issues/1599 ?). If abc is a global struct, then abc.var += 1; doesn't work in the global scope nor in a struct, abc.var = 1; works in the global scope but not in a struct, and u32 def = abc.var + 1; and std::print("{}",abc.var + 1); works in the global scope and in a struct.
(That's not directly relevant to this issue, though, and I should report it as its own issue.)