xlc
Parse valid JSON array of objects and insert each object into a substrate pallet's StorageMap
@xlc i've been trying to use lite-json, and i noticed that you actually created it!
i'm want to use off-chain workers to query an external API endpoint in our code here where i've added the off-chain workers example code https://github.com/DataHighway-DHX/node/blob/luke/rewards-allowance-new-offchain/pallets/mining/rewards-allowance/src/lib.rs#L2684
i want to retrieve data in the following valid JSON format from the body of the http request response:
{
"data": [
{
"acct_id": "1234",
"mpower": "1",
"date_last_updated": "1636096907000"
},
{
"acct_id": "1234",
"mpower": "1",
"date_last_updated": "1636096907000"
}
]
}
then iterate through each object in the array and whilst doing so i'll populate a data structure with each object's values and insert it into the pallet's storage where for each object in the that array i create a key (which is a tuple that contains both the start of the current date that we received the response, and the account_id that was in the response), and a value (which just contains the other two values received including the account_id's mining power mpower and the date that date was last updated off-chain)
/// Recently submitted mPower data.
#[pallet::storage]
#[pallet::getter(fn mpower_of_account_for_date)]
pub(super) type MPowerForAccountForDate<T: Config> = StorageMap<_, Blake2_128Concat,
(
Date, // converted to start of date
T::AccountId,
),
(
u128, // mPower
Date, // date last updated off-chain
T::BlockNumber, // block receive using off-chain workers
),
>;
i'd also like to know how to serialize and deserialize that kind of object.
how may i do this with lite-json?
i was able to do it with serde_json here https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=7a066487d37870330444b0908115cacf
use serde::{Deserialize, Serialize};
#[cfg_attr(feature = "std", derive(Debug))]
#[derive(Default, Clone, PartialEq, Eq)]
pub struct MPowerPayload {
acct_id: String,
mpower: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct DataValue {
acct_id: String,
mpower: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct JData {
data: Vec<DataValue>,
}
fn main() {
let json_str = r#"{
"data": [
{ "acct_id": "50000000000000001", "mpower": "1" },
{ "acct_id": "50000000000000002", "mpower": "2" }
]
}"#;
let mpower_json_data: JData = match serde_json::from_str(json_str) {
Err(e) => {
println!("Couldn't parse JSON :( {:?}", e);
return;
},
Ok(data) => data,
};
println!("{:?}", mpower_json_data);
let mut mpower_data_vec = vec![];
for (i, v) in mpower_json_data.data.into_iter().enumerate() {
println!("i v {:?} {:?}", i, v);
let mpower_data_elem = MPowerPayload {
acct_id: v.acct_id.clone(),
mpower: v.mpower.clone(),
};
mpower_data_vec.push(mpower_data_elem);
}
}
but serde_json doesn't work in my pallet as it causes error:
error: duplicate lang item in crate `std` (which `serde` depends on): `oom`.
There are no any automatic serialize / deserialize support so you will need to construct JsonValue / match the generated JsonValue to work with the data.
For serde_json, you should just need to disable std somewhere.
i'm using lite-json again now because i can't figure out how to disable std to overcome that error with serde_json, if i use lite-json, i can get it to work when the value of the key in the JSON is just a string or number, but not when it's an vector.
for example, if i do the below, where the value of the key in the JSON file is an vector:
/// Payload used to hold mPower data required to submit a transaction.
#[cfg_attr(feature = "std", derive(Debug))]
#[derive(Encode, Decode, Default, Clone, PartialEq, Eq)]
pub struct MPowerPayload<U, V> {
pub account_id_registered_dhx_miner: U,
pub mpower_registered_dhx_miner: V,
}
type MPowerPayloadData<T> = MPowerPayload<
<T as frame_system::Config>::AccountId,
u128,
>;
let mpower_data = r#"{
"data": [
{ "acct_id": "50000000000000001", "mpower": "1" },
{ "acct_id": "50000000000000002", "mpower": "1" }
]
}"#;
let mpower_json_data = lite_json::parse_json(mpower_data);
let mut mpower_data_vec: Vec<MPowerPayloadData<T>> = vec![];
let mpower_array = match mpower_json_data.ok()? {
JsonValue::Object(obj) => {
let (_, v) = obj.into_iter().find(|(k, _)| k.iter().copied().eq("data".chars()))?;
match v {
JsonValue::Array(vec) => vec,
_ => return None,
};
},
_ => return None,
};
for (i, obj) in mpower_array.into_iter().enumerate() {
println!("mpower_array obj {:?} {:?}", i, obj);
let mpower_data_elem: MPowerPayloadData<T> = MPowerPayload {
account_id_registered_dhx_miner: obj.acct_id.clone(),
mpower_registered_dhx_miner: obj.mpower.clone(),
};
mpower_data_vec.push(mpower_data_elem);
}
it appears to assign a value of type std::vec::Vec<lite_json::JsonValue> to mpower_array and then gives the following error instead of returning the vector that i can then iterate
error[E0599]: the method `into_iter` exists for unit type `()`, but its trait bounds were not satisfied
--> pallets/mining/rewards-allowance/src/lib.rs:2761:42
|
2761 | for (i, obj) in mpower_array.into_iter().enumerate() {
| ^^^^^^^^^ method cannot be called on `()` due to unsatisfied trait bounds
|
= note: the following trait bounds were not satisfied:
`(): Iterator`
which is required by `(): IntoIterator`
`&(): Iterator`
which is required by `&(): IntoIterator`
`&mut (): Iterator`
which is required by `&mut (): IntoIterator`
Can you create a snippet with https://play.rust-lang.org to reproduce the error?
Can you create a snippet with https://play.rust-lang.org to reproduce the error?
i've reproduced the error in the latest commit in this https://github.com/DataHighway-DHX/node/pull/238 (branch 'luke/rewards-allowance-new-offchain')
unfortunately i can't replicate it using https://play.rust-lang.org/, because you can only use the top % of most used crates there, and lite_json doesn't appear to be in that and thus isn't available on the playground.
i wonder if it'd be worth requesting funding from the Kusama treasury to increase the amount of crates you can use there in that Rust-specific playground. an alternative is to use https://playground.substrate.dev/, but it takes can take much longer to replicate just a Rust error.