How to: Serialize, Deserialize, Access values

[davidglavas]

I'm trying to perform the following 3 operations:

• serialize a plaintext json document (plaintext json -> ion binary)
• access values in the ion binary document
• deserialize an ion binary document (ion binary -> plaintext json)

For example:

#include <iostream>
#include <string>    

int main(int, char**) {

   // valid plaintext JSON document
   std::string testJSON = "{ \"id\" : { \"numID\" : 12345, \"strID\" : \"onetwothreefourfive\", \"doubleID\" : 1.2345}, \"hello\" : \"world\", \"age\" : 42, \"cats\" : [ \"first cat\", \"second cat\", \"third cat\" ] }";

   // TODO somehow use ion functions to perform the 3 operations:

   // 1. Serialize
   ionObject serializedDoc = ion.serialize(testJSON);

   // 2. Access values
   // access the top-level value associated with key "age" (42 in this case)
   int age = serializedDoc.accessValue("age");

   // 3. Deserialize
   std::string originalJSON = ion.deserialize(serializedDoc);

   std::cout << originalJson << std::endl;
}

Is there a straightforward way to do these 3 operations for any given plaintext JSON document?

[tgregg]

@davidglavas You may find the cookbook useful. For straight conversion between formats, see also ion_writer_write_all_values, which re-writes the document the reader is configured to parse using the writer's configuration.

[davidglavas]

@tgregg Thank you for the links. In the cookbook there is a "Migrating JSON data to Ion" section which sounds like what I'm looking for (convert plaintext JSON files to Ion so I can more quickly access values in the JSON document).

But there is no code example in the cookbook, is there some example that shows how to convert an std::string someJSONFile to an ion object?

[tgregg]

@davidglavas There is no example in the "Migrating JSON data to Ion" section because Ion readers can parse JSON natively with no extra configuration (Ion's text format is a superset of JSON, so JSON is valid Ion). There is no need to convert to Ion first before using Ion readers to parse the data. Try starting with the "Reading and Writing Ion Data" section of the cookbook.

[davidglavas]

@tgregg

There is no need to convert to Ion first before using Ion readers to parse the data.

If I pass the JSON text to the ion reader then it needs to parse the JSON text which is what I thought was slow and we were trying to avoid. I thought that the whole point of having a binary format to represent JSON files is to convert the JSON text once into the binary format, which can then be used to efficiently extract values (much faster than from the JSON text because of for example size-prefixes).

So to maximize the speed at which I read values from the JSON text, I don't want to re-parse the entire JSON text with every access, I want to convert the JSON text to the ion binary format once, which can be parsed more efficiently so that I can use it for many different accesses.

So the code example I'm looking for would do this conversion from JSON text to the ion binary format which I can then use for quick accesses without having to re-parse the JSON text.

Did I miss anything?

[tgregg]

@davidglavas You didn't miss anything. I didn't realize that your use case involved reading the same data multiple times.

Convert JSON data to binary Ion like this. I stitched together examples from the cookbook and added some logic to avoid having to guess the size of the output buffer. Note: I didn't compile this, but it should be pretty close to runnable.

#include <stdlib.h>
#include "ionc/ion.h"

#define ION_OK(x) if (x) { printf("Error: %s\n", ion_error_to_str(x)); return x; }

int main(int argc, char **argv) {
    const char* json_text = "{\"hello\": \"world\"}";

    hREADER reader;
    ION_READER_OPTIONS reader_options;
    hWRITER writer;
    ION_WRITER_OPTIONS writer_options;
    ION_STREAM *output_stream;

    // Configure the reader and writer options
    memset(&reader_options, 0, sizeof(ION_READER_OPTIONS));
    memset(&writer_options, 0, sizeof(ION_WRITER_OPTIONS));
    writer_options.output_as_binary = true;

    // Initialize the reader.
    /* Note: there is a variant that allows reading over an ION_STREAM, which enables
     * reading directly from a file, user-provided handler, and more. See ion_reader.h
     * and ion_stream.h.
     */
    ION_OK(ion_reader_open_buffer(&reader,
                                  (BYTE *)json_text,
                                  (SIZE)strlen(json_text),
                                  &reader_options));

    // Initialize the output stream and writer.
    /* Note: there are variants that enable writing directly to file, a user-provided
     * handler, and more. See ion_stream.h.
     */
    ION_OK(ion_stream_open_memory_only(&output_stream));
    ION_OK(ion_writer_open(&writer, ion_stream, &writer_options));

    // Convert the JSON data to binary Ion.
    ION_OK(ion_writer_write_all_values(writer, reader));

    // Close the writer and collect the data from the output stream.
    ION_OK(ion_writer_close(writer));
    POSITION output_stream_length = ion_stream_get_position(output_stream);
    ION_OK(ion_stream_seek(output_stream, 0));
    BYTE *binary_ion = (BYTE *)(malloc((size_t)output_stream_length));
    SIZE binary_ion_length;
    ION_OK(ion_stream_read(output_stream, binary_ion, (SIZE)output_stream_length, &binary_ion_length));
    ION_OK(ion_stream_close(output_stream));
    if (binary_ion_length != (SIZE)output_stream_length) {
        ION_OK(IERR_INVALID_STATE);
    }

    // Do something with the data...
    // ...

    free(binary_ion);
}

[davidglavas]

@tgregg Thank you I will try it out. Your example would be what I call "Serialization" (turning a plaintext JSON document into the binary representation).

Now for the accesses, is there some helper function for accessing top-level and nested values without having to use reader.stepIn(); and reader.next();?

So for example, for this JSON document:

{
  "shopper": {
    "Id": "4973860941232342",
    "Context": {
      "CollapseOrderItems": false,
      "IsTest": false
    }
  },
  "SelfIdentifiersData": {
    "SelfIdentifierData": [
      {
        "SelfIdentifierType": {
          "SelfIdentifierType": "111"
        }
      },
      {
        "SelfIdentifierType": {
          "SelfIdentifierType": "2222"
        }
      }
    ]
  }
}

I would provide a path (sequence of strings) and get back the value in the JSON document behind that path. So for example:

The path [ shopper -> Id -> targetValue ] points to the string "4973860941232342"

The path [ SelfIdentifiersData -> SelfIdentifierData -> array_idx: 0 -> targetValue ] points to the object { "SelfIdentifierType": { "SelfIdentifierType": "111" } }.

The path [ SelfIdentifiersData -> SelfIdentifierData -> array_idx: 0 -> SelfIdentifierType -> targetValue ] points to the object { "SelfIdentifierType": "111" }.

The path [ SelfIdentifiersData -> SelfIdentifierData -> array_idx: 0 -> SelfIdentifierType -> SelfIdentifierType -> targetValue ] points to the string "111"

Is there some function to perform accesses this way?

[tgregg]

@davidglavas Yes, the path extractor API. See ion_extractor.h.

The best place to find some quick examples at this point is probably the tests: test_ion_extractor.cpp.

[davidglavas]

@tgregg Nice, thank you. I'll try using the tests.

It would be really useful to have such an example in the cookbook where for a given JSON string it is converted into the ion binary representation and then efficient accesses are performed with the extractor API.

[tgregg]

@davidglavas I agree.

Relevant issues: https://github.com/amzn/ion-docs/issues/98 https://github.com/amzn/ion-docs/issues/99