dfinity
Reserved For Large Metadata Handling in Transaction Logs
We have identified that icrc1_supported_standards needs to be generalized into its own ICRC.
We may need to provide a hash or URL in block data so that large metadata can be reconstructed. This ICRC-61 number will be used for defining that standard.
To provide a standardized approach to handling metadata that is too large to fit inside of a standard ICRC-3 Value block, ICRC-61 defines the standard structure and expected behavior for consumers of the Block. The initial standards are purposely simple and MAY be extended through additional ICRC Extensions to target specific systems, standards, and protocols. Below is a comprehensive description of the ICRC-61 External Metadata Standard's specifications.
Behavior:
When a parser encounters a Map variant in a Value type that has a key that contains one or more of the following icrc61_{} keys, the parser should retrieve the file, parse the path if provided and replace the Map in the tree with a Value containing the retrieved data
A Text variant should be used for the retrieved data and the data should be represented as a dataURI. The mime type used can be overridden with the icrc61_mime key in the ICRC-61 map, otherwise, the mime-type returned from the remote server should be used.
SHA-256 should be the default hashing used if not alternative hash is provided in icrc61_hash_algo. The current standard does not define any other hashing algorithms to use at the is time.
The SHA-256 is supplied in icrc61_hash to validate the retrieved file if provided.
The standard does not explicitly define the names of all possible hash algorithms. Users SHOULD default to the names as defined in ISO and NIST standards.
Data Types
The ICRC-61 standard is agnostic to the data type that an ICRC-61 entry may point to.
The possible values in an ICRC-61 map are as follows.
Map {
icrc61_url?: Text; // Optional Path to external file. This item MUST exist unless the hash provided is compatible with IPFS
icrc61_hash?: Text; '8o3vfuaowrgva' // Optional hash of external file in Base64
icrc61_hash_algo?: Text; //Optional: Defaults to SHA-256, but user may override the hash function
icrc61_path?: Text // Optional path to be applied to the retrieved data. Example for JSON: 'a.c.[2].d' json value within above json
icrc61_mime?: Text; //Optional: override the mime type of the retrieved file
};
Examples
The following Block:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("file", #Map[
("icrc61_url", #Text("https://foo.bar/data.json"),
("icrc61_hash, #Text("UNhYo7NpAQ61sEpPbHXH4B5C4tLwF87qNaAg17Qn8Jw=")),
]),
("ts", #Nat(1739394848859))
])
]
Would be parsed to:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("file", #Text("data:application/json;charset=utf-8;base64,ew0KICAiZm9vIjoiYmFyIiwNCiAgImZpenoiOiJidXp6Ig0KfQ==)"),
("ts", #Nat(1739394848859))
])
]
Updated after NFT WG meeting on July 9, 2024
To provide a standardized approach to handling metadata that is too large to fit inside of a standard ICRC-3 Value block, ICRC-61 defines the standard structure and expected behavior for consumers of the Block. The initial standards are purposely simple and MAY be extended through additional ICRC Extensions to target specific systems, standards, and protocols. Below is a comprehensive description of the ICRC-61 External Metadata Standard's specifications.
Behavior:
When a parser encounters a Map variant in a Value type that has a key of icrc61_external, the parser should retrieve the file, parse the path if provided, and replace the Map in the tree with a Value containing the parsed retrieved data.
The current standard supports the following document object models for interpretation and replacement:
- json RFC-7159
The retrieved data should be converted to the ICRC-3 Value types as follows
Json Conversion
Default Conversion
- Strings - JSON strings are converted to ICRC3 Text elements:
"smith"
to
#Text("smith")
Note: If you want the full content of a file to replace the icrc61_external Map, you can use a dataURI in a string
-
Numbers - Numbers are converted depending on if they contain a decimal or negative symbol
- Natural - If the item has no negative symbol and no decimal it should be converted to an ICRC3 Nat.
- Integer - If the time has a negative symbol and no decimal it should be converted to an ICRC3 Int.
- Float - If the item has a decimal it should be converted to an ICRC3 Text.
-
Bool - Bools are converted to ICRC3 Text values with either "true" or "false" strings;
true
to
#Text("true");
-
Null - Null items are left out of the object map they are specified in. A null value at the root should remove the icrc61_external item from the containing structure.
-
Array - Arrays are replaced by ICRC3 Array Objects.
[1,2,3]
to
#Array([#Nat(1), #Nat(2), #Nat(3)]);
- Object - Objects are converted to ICRC3 Maps where the key is the 0 parameter as a string and the value is the value converted to ICRC3 according to this specification.
{
"foo": "bar",
"baz": 16
}
to
#Map([
("foo", #Text("bar")),
("bas", #Nat(16))
]);
Overrides
Reserved keys in the json should allow a user to more expressly define what datatype a value should be. These items should be in an object with only one property:
- "__icrc61_blob_hex" - convert this value to an ICRC3 Blob;
- "__icrc61_blob_base64" - convert this value to an ICRC3 Blob;
- "__icrc61_int" - convert this value to an ICRC3 Int
Candid Conversion
Candid Conversion should follow the following translations
- functions: Functions should be left out of the conversion
- reserved: reserved items should be left out of the conversion
- null : null items should be left out of the conversion
- empty: empty items should be left out of the conversion
- nat | nat8 | nat16 | nat32 | nat64: should be converted to a ICRC3 Nat.
- int | int8 | int16 | int32 | int64 : should be converted to an ICRC3 Int.
- float32 | float64: should be converted to an ICRC3 Text with the most descriptive possible string output
- bool : ICRC3 Text values with either "true" or "false" strings;
- text: ICRC3 Text value
- options: Option items that are null should be left out and if not-null should be converted without regard to the optionality according to this spec.
- vec: vectors should be converted to ICRC3 Arrays.
- blobs: blobs should be converted to ICRC3 Blobs.
- records | tuple: records should be converted to ICRC3 Maps where the key is the key hash in the record as a string. If no key is provided the text output of the index should be used ("0", "1", "2", etc.). Keys may be replaced with filed names if a did file is provided.
- variants: convert to a single member ICRC3 Map where the key is the variant hash. Keys may be replaced with filed names if a did file is provided.
- service reference: not converted
- principal reference: converted to an ICRC3 Blob type of the principal. (is this possible)
- type reference: not converted.
- imports: not converted: unclear if these are represented in the file.
XML conversion
TBD
Data Types
The possible values in an ICRC-61 map are as follows.
Map {
icrc61:external: Map :{
url: Text//required, the URL of the resource to be retrieved
sha256_hash?: Blob; //optional hash if needed. Base64; For URIs such as IPFS the hash may be implicit to the URL
path?: Text; //optional path to a place in a json file for which the parser should take as the root element for replacement
};
};
If you want your data converted to candid in a specific mannare you must define the specification and indicate that standard, otherwise, we assume the json mappings above.
Examples
The following Block:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("details", #Map[
("icrc61_external", #Map[
("icrc61_url", #Text("https://foo.bar/data.json"),
("icrc61_hash, #Text("M0AbXxs3iBNWiAzx8zc0q0o89rqAcqtEYwK6KMNfgBE=")),
];),
]),
("ts", #Nat(1739394848859))
])
]
With the file:
{
"foo" : "bar"
"baz" : 16
}
Would be parsed to:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("details", #Map([
("foo": #Text("bar"),
("baz": #Nat(16)]),
("ts", #Nat(1739394848859))
])
]
@sea-snake @dietersommer
I'm thinking that I need to take out the candid and XML for now...they are kind of messy...well even JSON is messy as I see a ton of applications where people will define metadata in Candid/ICRC16 and then it will have to be converted to JSON when will lose a good bit of definition unless we define a specific conversion and then back out the ICRC3 Value. Yuck.
#Map([
("name", #Text("Bob"),
("age", #Nat(45)),
("roles", #Array(#Text("admin"),#Text("reader"),#Text("writer")]),
("status", #Opt(#Text("active"))
];
to:
{
"name":"Bob",
"age": 42,
"roles": ["admin","reader","writer"],
"status" : "active"
}
or
{
"name":{
"Text": "Bob"
},
"age": {
"Nat": 42
},
"roles": {
"Array": [{
"Text": "Bob"
},
{
"Text": "Bob"
},
{
"Text": "Bob"
}],
"status" : {
"Opt": {
"Text": "active"
}
}
to
#Map([
("name", #Text("Bob"),
("age", #Nat(45)),
("roles", #Array(#Text("admin"),#Text("reader"),#Text("writer")]),
("status", #Text("active"))
];
or
#Map([
("name", #Map([("Text", #Text("Bob")]),
("age", #Map([("Nat", #Nat(45))],
("roles", #Map([("Array", #Array([
#Map([("Text", #Text("read")]),
#Map([("Text", #Text("write")]),
#Map([("Text", #Text("admin")])]),
("status", #Map(["Opt", #Map([("Text", #Text("active"))])]))
];
Super Yuck. I guess I'm going to have to define more overrides on the way out with magic strings so they can be imported properly.
Updated after 8/13/2024
| ICRC | Title | Author | Discussions | Status | Type | Category | Created |
|---|---|---|---|---|---|---|---|
| 61 | Large Block and Metadata Handling | Austin Fatheree (@skilesare), Thomas (@sea-snake),Dieter Sommer (@dietersommer) | https://github.com/dfinity/ICRC/issues/61 | Pre-Draft | Standards Track | 2024-08-15 |
To provide a standardized approach to handling metadata that is too large to fit inside of a Value or ICRC16 data item, ICRC-61 defines the standard structure and expected behavior for consumers of the Data. The initial standards are purposely simple and MAY be extended through additional ICRC Extensions to target specific systems, standards, and protocols. Below is a comprehensive description of the ICRC-61 Large Block and Metadata Handling's specifications.
Behavior:
When a parser of a Value type as defined in ICRC-3 or an ICRC16 type as defined in ICRC-16 encounters a Map Or Class(ICRC-16 only) variant in a Value type that has a key of icrc61:external, the parser should retrieve the file, parse the path if provided, and replace the Map in the tree with the data retreived.
The current standard supports the following document object models for interpretation and replacement:
Data Types
The possible values in an ICRC-61 map are as follows.
Parent Element
Type: Map
Members:
Required:
- Key: "icrc61:external"
- Value: Map - see Child Element
- Description - This element is replaced by the parser
ICRC-16 Extention
The type may be a Map or Class. In the case of a Class the parser MUST honor the immutable property of a Class and apply the same value to the replacement object.
Child Element
Type: Map
Members:
Required:
- Key: "url"
- Value: Text
- Description: A url pointing to the asset to be retrieved. A parser SHOULD support http, https, ipfs, or ic-http urls and MAY support more.
Optional:
- Key: "sha256_hash"
- Value: Blob
- Description: The hash can be provided if the url schema used does not provide a native hash value in it. The parser should verify the sha256_hash of the retrieved data.
Optional:
- Key: "path"
- Value: Text
- Description: The path can be provided if the object needs to point to a child element in the document. It should use CandyPath Syntax.
Candid Definition
Map (
vec {
record {
"icrc61:external";
Map (
vec {
record { "url" : Text; Text("{target_resource})}; //required, the URL of the resource to be retrieved
record { "sha256_hash" : Text; Blob("{hash}") }; //optional hash if needed. Base64; For URIs such as IPFS the hash may be implicit to the URL
record { "path" : Text; Text("{path}") }; //optional path to a place in the retrieved candid for which the parser should take as the root element for replacement
};
)
};
};
);
Examples
The following Block:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("details", #Map[
("icrc61_external", #Map[
("url", #Text("https://foo.bar/data.json"),
("hash, #Blob("/0a/0b/0c.../0e/0f")),
];),
]),
("ts", #Nat(1739394848859))
])
]
With the file:
#Map([
("foo", #Text("bar"),
("baz", #Nat(16)
]
)
Would be parsed to:
#Map[
("btype", #Text("60upload")),
("tx", #Map[
("details", #Map([
("foo", #Text("bar"),
("baz", #Nat(16)
]
),
("ts", #Nat(1739394848859))
])
]
Downgrades of ICRC-16 to Value
When a Value type points to an ICRC16 external data the ICRC16 Data should be downgraded according to the following specification:
///converts a candyshared value to the reduced set of ValueShared used in many places like ICRC3. Some types not recoverable
public func CandySharedToValue(x: CandyShared) : ValueShared {
switch(x){
//Text stays Text
case(#Text(x)) #Text(x);
//Map stays Map
case(#Map(x)) {
let buf = Buffer.Buffer<(Text, ValueShared)>(1);
for(thisItem in x.vals()){
buf.add((thisItem.0, CandySharedToValue(thisItem.1)));
};
#Map(Buffer.toArray(buf));
};
//Class is downgraded to Map
case(#Class(x)) {
let buf = Buffer.Buffer<(Text, ValueShared)>(1);
for(thisItem in x.vals()){
buf.add((thisItem.name, CandySharedToValue(thisItem.value)));
};
#Map(Buffer.toArray(buf));
};
//Int variants become Ints
case(#Int(x)) #Int(x);
case(#Int8(x)) #Int(Int8.toInt(x));
case(#Int16(x)) #Int(Int16.toInt(x));
case(#Int32(x)) #Int(Int32.toInt(x));
case(#Int64(x)) #Int(Int64.toInt(x));
case(#Ints(x)){
#Array(Array.map<Int,ValueShared>(x, func(x: Int) : ValueShared { #Int(x)}));
};
//Nat varians become Nats
case(#Nat(x)) #Nat(x);
case(#Nat8(x)) #Nat(Nat8.toNat(x));
case(#Nat16(x)) #Nat(Nat16.toNat(x));
case(#Nat32(x)) #Nat(Nat32.toNat(x));
case(#Nat64(x)) #Nat(Nat64.toNat(x));
case(#Nats(x)){
#Array(Array.map<Nat,ValueShared>(x, func(x: Nat) : ValueShared { #Nat(x)}));
};
//Bytes become a Blob
case(#Bytes(x)){
#Blob(Blob.fromArray(x));
};
//Array stays array
case(#Array(x)) {
#Array(Array.map<CandyShared, ValueShared>(x, CandySharedToValue));
};
//Blob stays Blob
case(#Blob(x)) #Blob(x);
//Bool becomes a single byte blob of 1(true) or 0(false)
case(#Bool(x)) #Blob(Blob.fromArray([if(x==true){1 : Nat8} else {0: Nat8}]));
//Float is converted to Text
case(#Float(x)){#Text(Float.format(#exact, x))};
//Floats array becomes an array of Texts
case(#Floats(x)){
#Array(Array.map<Float,ValueShared>(x, func(x: Float) : ValueShared { CandySharedToValue(#Float(x))}));
};
//Option Values - nulls are input as an empty Array, statefull values as a one item array
case(#Option(x)){ //empty array is null
switch(x){
case(null) #Array([]);
case(?x) #Array([CandySharedToValue(x)]);
};
};
//Principals are converted to a Blob via Principal.toBlob
case(#Principal(x)){
#Blob(Principal.toBlob(x));
};
//Set is converted to an array
case(#Set(x)) {
#Array(Array.map<CandyShared,ValueShared>(x, func(x: CandyShared) : ValueShared { CandySharedToValue(x)}));
};
//Value map is converted to an array of length two arrays
case(#ValueMap(x)) {
#Array(Array.map<(CandyShared,CandyShared),ValueShared>(x, func(x: (CandyShared,CandyShared)) : ValueShared { #Array([CandySharedToValue(x.0), CandySharedToValue(x.1)])}));
};
//case(_){assert(false);/*unreachable*/#Nat(0);};
};
};