Improve documentation for `import_callback`

[ceridwen]

I've been struggling to understand how import_callback works using the documentation. It says that it can implement library search paths, which is the functionality I need. The documentation in the old webpage at least discusses what it needs to be called with and returns:

Another keyword argument import_callback can be used to pass a callable to trap the Jsonnet import constructs. This allows, e.g., reading files out of archives or implementing library search paths. The supplied function must take two string arguments (directory of the current file and the string given to the import construct, which can usually be concatenated to form an actual path but that is up to you). It returns a tuple of two strings, the first being the actual path of the file and the second being the content. The actual path is required so that imports can be resolved within the imported file.

The new page only refers to "see example in python/", which I think(?) refers to Github. Adding a link would have helped me find that faster. I looked at the example, though, and I still don't understand how to use it to set up library search paths correctly. I tried and failed to use PDB and print to figure out what import_callback is being called with and what it's returning, but both failed.

I have a layout like:

foo.py
jsonnet/
jsonnet/bar.jsonnet
jsonnet/baz.libsonnet

bar.jsonnet contains local metadata = import 'baz.libsonnet';. I want to run python foo.py, which contains _jsonnet.evaluate_snippet on the contents of bar.jsonnet and have the import work correctly.

I tried to have import_callback return jsonnet/ and the contents of baz.libsonnet, but I'm still getting import errors, so I've obviously misunderstood. Can someone clarify the documentation, please?

[sbarzowski]

Thanks for reporting the problem.

Please take a look at the example I posted on Stack Overflow: https://stackoverflow.com/questions/49949418/how-to-correctly-call-jsonnet-with-imports-from-python/49954607#49954607

Could you post the code that didn't work? It will be easier for us to tell what is confusing and make it clear in the docs.

I looked at the example, though, and I still don't understand how to use it to set up library search paths correctly.

Library search paths are just paths you check in your custom importer. There is no set up on the jsonnet API side, it's just the matter of how you parametrize your custom importer function. (To clarify, the library paths in C api are only a configuration option of the default formatter and don't apply to custom importers).

[ceridwen]

My last pieces of nonworking code looked like:

def import_callback(directory, file_name):
    print('DIRECTORY', directory)
    print('FILE NAME', file_name)
    try:
        with open(str(pathlib.Path.cwd() / 'templates' / file_name)) as f:
            print(str(pathlib.Path.cwd() / 'templates' / file_name))
            return str(pathlib.Path.cwd() / 'templates' / file_name), f.read()
    except IOError:
        with open('./' + file_name) as f:
            return './' + file_name, f.read()

jsonnet.evaluate_snippet(
            'jsonnet/',
            snippet,
            tla_vars=jsonnet_variables,
            import_callback=import_callback)

I went through many iterations before these, though.

I still don't think I really understand what import_callback and what arguments it's passed. If you have jsonnet that doesn't correspond to any actual file, you have to make up a fake file name and pass it to evaluate_snipper()? Does the filename that's passed to evaluate_snippet get passed to import_callback at all?

[sbarzowski]

I still don't think I really understand what import_callback and what arguments it's passed.

import_callback (aka importer) is a way to provide files to jsonnet from any source. It may be a filesystem, but it can also be a python dict or a database or whatever. Our only assumption is that the files are identified by paths. Your hierarchy can be completely made up, but every file/snippet must be there.

import_callback(from_dir, path)

• if path is absolute it returns a file identified by path.
• if path is relative then it returns a file identified by path, usually relative to from_dir, but it may also be relative to some library paths. By "returns a file" I mean file contents and its absolute path.

BTW I view import_callback as a somewhat subtle feature for special cases. So if you are doing something which you feel should be simple and need to use import_callback, then there is probably a better way (or we should add it 😀).

If you have jsonnet that doesn't correspond to any actual file, you have to make up a fake file name and pass it to evaluate_snipper()?

Short answer: yes Long answer: every imported piece of jsonnet has an associated identifier, a path which may or may not correspond to the actual filesystem. You can for example get it from an in-memory dict or from database or over the network. It is important that even snippets have this path, because imports inside the snippet may be relative to the snippet's "fake" path.

Does the filename that's passed to evaluate_snippet get passed to import_callback at all?

Yes, the path is passed to import_callback when you import stuff from the snippet. import_callback is not called for the snippet itself - you already have it (contents and path). However if you import stuff from the snippet, import callback is called to fetch the imported file and it needs to know from where it is requested.

Please let me know if it's any clearer.

[sparkprime]

I see we did lose some content in the new page http://jsonnet.org/ref/bindings.html

That stuff was supposed to make it into the "language reference" since it's not actually Python specific, the behavior comes from the C code being wrapped. But that's not written yet. I'll make sure it's obvious.

In addition to what sbarzowski said, the "filename" of the Jsonnet snippet is also used in error tracebacks. Its directory is used to resolve relative imports from the snippet to elsewhere.

[indigoviolet]

FWIW, I ran into this identical state of documentation just now. It might be worth just linking to this issue from the documentation in the meantime.