Macros ready for experimental use?

[barries]

We're very happily using ion-rs's v1.1 binary format for logging, along with ion cat and our own log decoder.

One of the reasons for using our own log decoder is that we started before macros were a thing, so every log entry is an array:

[ timestamp, thread_id, entity_id, entry_kind, ...variable amount of data ]

We'd like to use macros so we could encode the above as a struct with field names like timestamp, etc, so we can have nearly the same space efficiency on disk as with the array and have more user friendly output. For example, iot cat output would be more self describing (and easier to convert to JSON), and our local Rustaceans and Pythonistas could work with structs or HashMap<>s and dicts instead of arrays w/ index constants.

Are the macro APIs and implementations far enough along for me to prototype this? If so, where should I look to learn how?

Many thanks for an excellent format and set of tools.

[zslayton]

Are the macro APIs and implementations far enough along for me to prototype this? If so, where should I look to learn how?

The low-level APIs are implemented/functional, but we're going to be adding higher-level APIs that are less error prone in the coming weeks. Since you're likely to have a fixed set of macros (at least initially?), I think you could get started with the low-level APIs without too much trouble and migrate the higher-level ones later.

Note that Ion 1.1 is in its final comment period and it's likely there will be some small changes to the format once the FCP closes on March 12th. Any log files you write today may not be readable after those changes are merged.

All of that to say: I think it makes sense to start experimenting/developing with macros now if you're ok discarding all of your logs when the spec is finalized.

Looking around at our existing examples/benchmarks, I don't think any of them are a good guide to follow. I'll try to find a time later today to add one to the examples directory.

[barries]

I'm very comfortable with the rapid pace of development and it seems like the timing of the changes will dovetail with our project's progress well.

Thanks again!

[zslayton]

In the process of updating our examples I realized there's one more feature that needs to be implemented before you can do this easily. I'm going to prioritize working on that and will update the write_log_events example in the process.

[zslayton]

When #938 is merged, it will be possible to easily define and use macros from the user-facing Writer type. See examples/write_log_events.rs in that PR for sample code.

Still TODO:

• Binary expression groups are not yet supported; trying to instantiate a binary expression group writer will lead to a todo!() panic.
• Guard rails on the e-expression writers that surface an IonError if/when you call methods that are not compatible with the signature of the macro you're invoking.

In sum, when you go to experiment, stick to required parameters using tagged types (e.g. (x y z), not (flex_uint::x+ y* z?)) and test your output to make sure it's working as expected.

I'll be addressing these items next. However, I'm going to be away from work for a few weeks and may not land a PR before I leave. Please continue reach out via issues if you have questions or need help, my teammates will be able to assist.

[popematt]

@barries, you might want to see https://github.com/amazon-ion/ion-docs/issues/397. We have been running into issues implementing the current Ion 1.1 specification, and we are likely to simplify things. How extensively have you started using Ion 1.1 macros? Feel free to jump in on the other issue too.

[barries]

Thanks, @popematt; based on the earlier responses, I deferred using macros. Good luck getting 1.1 support integrated!