fluvio-jolt
fluvio-jolt copied to clipboard
Rust implementation of Java Jolt JSON to JSON transformation library
Fluvio Jolt JSON library
Overview
JSON to JSON transformation where the "specification" for the transform is itself a JSON document.
Port of Java Jolt library written in Rust.
Usage Example
Add fluvio-jolt
crate to your Cargo.toml
file:
[dependencies]
fluvio-jolt = { version = "0.3"}
Then, for example, if you want to repack your JSON record, you can do the following:
use serde_json::{json, Value};
use fluvio_jolt::{transform, TransformSpec};
let input: Value = serde_json::from_str(r#"
{
"id": 1,
"name": "John Smith",
"account": {
"id": 1000,
"type": "Checking"
}
}
"#).unwrap();
let spec: TransformSpec =
serde_json::from_str(r#"[
{
"operation": "shift",
"spec": {
"name": "data.name",
"account": "data.account"
}
}
]"#).unwrap();
let output = transform(input, &spec);
assert_eq!(output, json!({
"data" : {
"name": "John Smith",
"account": {
"id": 1000,
"type": "Checking"
}
}
}));
Supported Operations
-
shift
: copy data from the input tree and put it the output tree -
default
: apply default values to the tree -
remove
: remove data from the tree
See SPEC.md
for more info on specifics of execution order and DSL grammar.
Specification
Composes a list of operation specifications. Each operation has its own DSL (Domain Specific Language) in order to facilitate its narrow job.
use fluvio_jolt::TransformSpec;
let spec: TransformSpec =
serde_json::from_str(r#"[
{
"operation": "shift",
"spec": {
"name": "data.name",
"account": "data.account"
}
}
]"#).unwrap();
Shift
operation
Specifies where the data from the input JSON should be placed in the output JSON, or in other words, how the input JSON/data should be shifted around to make the output JSON/data.
At a base level, a single shift
operation is a mapping from an input path to an output path,
similar to the mv
command in Unix, mv /var/data /var/backup/data
.
The input path is a JSON tree structure, and the output path is flattened "dot notation" path notation.
For example, given this simple input JSON:
{ "id": 1, "name": "John Smith", "account": { "id": 1000, "type": "Checking" } }
A simple spec could be constructed by copying that input, and modifying it to supply an output path for each piece of data:
{ "id": "data.id", "name": "data.name", "account": "data.account" }
would produce the following output JSON:
{ "data" : { "id": 1, "name": "John Smith", "account": { "id": 1000, "type": "Checking" } } }
Wildcards
The shift
specification on the keys level supports wildcards and conditions:
1. *
- match everything
2. name1|name2|nameN
- match any of the specified names
&
Wildcard
&
lookup allows referencing the values captured by the *
or |
.
&(x,y)
means go up the path x levels and get the yth match from that level.
0th match is always the entire input they and the rest are the specific things the *
s matched.
&
== &(0)
== &(0,0)
and &(x)
== &(x,0)
It allows for specs to be more compact. For example, for this input:
{ "id": 1, "name": "John Smith", "account": { "id": 1000, "type": "Checking" } }
to get the output:
{ "data" : { "id": 1, "name": "John Smith", "account": { "id": 1000, "type": "Checking" } } }
the spec with wildcards would be:
{ "*": "data.&0" }
If you want only id
and name
in the output, the spec is:
{ "id|name": "data.&(0)" }
&
wildcard also allows to dereference any level of the path of given node:
{ "foo": { "bar" : { "baz": "new_location.&(0).&(1).&(2)" // &(0) = baz, &(1) = bar, &(2) = foo } } } }
for the input:
{ "foo": { "bar": { "baz": "value" } } }
will produce:
{ "new_location": { "baz": { "bar": { "foo": "value" } } } }
$
Wildcard
$
wildcard allows accessing matched keys from the path and use them on the right hand side.
See tests in tests/java/resources/shift
for usage examples.
@
Wildcard
@
wildcard allows accessing values of matched keys from the path and use them on the right hand side.
See tests in tests/java/resources/shift
for usage examples.
Default
operation
Applies default values if the value is not present in the input JSON.
For example, given this simple input JSON:
{ "phones": { "mobile": 01234567, "country": "US" } }
with the following specification for default
operation:
{ "phones": { "mobile": 0000000, "code": "+1" } }
the output JSON will be:
{ "phones": { "mobile": 01234567, "country": "US", "code": "+1" } }
As you can see, the field mobile
remains not affected while the code
has a default '+1' value.
Remove
operation
Removes content from the input JSON. The spec structure matches the input JSON structure. The value of fields is ignored.
For example, given this simple input JSON:
{ "phones": { "mobile": 01234567, "country": "US" } }
you can remove the country
by the following specification for remove
operation:
{ "phones": { "country": "" } }
the output JSON will be:
{ "phones": { "mobile": 01234567 } }
Contributing
If you'd like to contribute to the project, please read our Contributing guide.
License
This project is licensed under the Apache license.