Support `importbytes`

[sbarzowski]

It should be pretty easy to provide an import variant which returns an array of bytes (we can even store it efficiently under the hood, at least for go-jsonnet). This would allow working with binary files at all - there is currently no good way to import non-UTF8 files. For example loading binary representations of cryptographic keys is impossible without external preprocessing.

[brancz]

I have exactly this use case. I'm not too familiar with the code base, but with some hints I could give this a shot (unless you know exactly how to do it and would it would be quick :) ).

[sbarzowski]

That's really cool. I'll collect the pointers to the relevant parts of the code soon.

We'll need it to do it for both C++ and Go implementations, but even the solution for one of them would be really helpful (I will implement the other one myself then). We'll also need to update the spec.

[brancz]

I know close to 0 C++, but go I should be able to do :)

[brancz]

It's not much, but this is how far I got without pointers: https://github.com/brancz/go-jsonnet/tree/importbytes

I guess the biggest question I have is where the actual functionality is implemented. If I'm not mistaken, we may need to change or introduce a new way of reading files entirely?

[sbarzowski]

If I'm not mistaken, we may need to change or introduce a new way of reading files entirely?

No. Fortunately, we already have the importer API which handles arbitrary byte sequences. The Unicode restriction appears only when it's converted to Jsonnet string.

Here are the relevant pieces layer by layer:

1. Creating AST a) AST node: https://github.com/google/go-jsonnet/blob/v0.17.0/ast/ast.go#L419 b) Tokenizer (lexer): https://github.com/google/go-jsonnet/blob/v0.17.0/internal/parser/lexer.go#L581 and https://github.com/google/go-jsonnet/blob/v0.17.0/internal/parser/parser.go#L873 c) Parser: https://github.com/google/go-jsonnet/blob/v0.17.0/internal/parser/parser.go#L1108 d) Desugarer (AST postprocessing): https://github.com/google/go-jsonnet/blob/v0.17.0/internal/program/desugarer.go#L431
2. Interpretation (evaluating AST): https://github.com/google/go-jsonnet/blob/v0.17.0/interpreter.go#L484
3. Importer support (the "meat", actually handling the import): https://github.com/google/go-jsonnet/blob/v0.17.0/imports.go#L134

I pointed to the importString which should be very similar in most cases. The last part (3) is where the generic data is converted to a Jsonnet value. We just need to convert data to valueArray of numbers.

Jsonnet arrays contain lazy values (thunks). It's necessary to convert each byte to a number, wrap it in valueNumber and then wrap it each in a thunk.

(This representation is of course inefficient. It can be improved, but that's out of scope for importbytes change).

Please let me know if you have any questions or if I can help.

[johnbartholomew]

I guess this request is covered by importbin which was added in 2022: https://github.com/google/jsonnet/pull/976, https://github.com/google/go-jsonnet/pull/584

Edit: And in Rust, https://github.com/CertainLach/jrsonnet/pull/77