[Feature] Add ~/.jq.d to default module search path

[systemmonkey42]

trafficstars

I have a number of "safe" macro's I load in ~/.jq because it will autoload and is super convenient for interactive use. Unfortunately this makes it impossible to have a fixed location in $HOME for dynamically loaded macros.

If I use the -L option, the standard search path is cleared, so its not as helpful as it sounds.

Can the directory ~/.jq.d be added to the start of the default module search path, so that fixed location is available for dynamic loading which does not require root privileges or conflict with autoloaded macros?

Also, but less important, is it possible to add $XDG_CONFIG_HOME support? Something like adding ~/.config/jq/jq (yeah, its a mess, configs in $XDG_CONFIG_HOME tend to drop the leading .) as an autoading module, and adding ~/.config/jq or ~/.configs/jq/lib to the module search path? I try to move configs to ~/.config whenever possible to reduce clutter in $HOME.

Thanks.

[emanuele6]

~/.jq can be a directory with .jq files in it instead of a file.

[systemmonkey42]

~/.jq can be a directory with .jq files in it instead of a file.

My apologies for not being explicit with my meaning.

~/.jq is BOTH an "on-startup-macro-file" and an "on-demand-module-directory". Having a single name for two entirely different use cases is a problem.

I would like a number of macros autoloaded on startup (hence ~/.jq the file), and a number of modules available "on demand" when they are needed (hence ~/.jq the directory).

Unless I'm mistaken, I cannot create a file and a directory of the same name, so I'm suggesting adding ~/.jq.d to the module search path so both features can be made available. (~/.jq as a directory should continue working if its used)

Sorry for not simply saying that in the beginning.

[wader]

~/.jq can be a directory with .jq files in it instead of a file.

Sure it should work? i can't manage to get it working. Also tried main.jq which seem to be a thing but with no success. Looking at the code around here https://github.com/jqlang/jq/blob/5029328d35f3e60037970d27f350a742af41aa02/src/linker.c#L178-L203 i get a feeling if ~/.jq is a directory stat will succeed and not end up trying main.jq and .jq. I'm also confused by the comments Try ${search_dir}/$(dirname ${rel_path})/jq/main.jq and Try ${search_dir}/${rel_path}/$(basename ${rel_path}).jq but don't see any dirname-ish call?

Get a feeling all this could be clarified a bit in the documentation also :)

[emanuele6]

Hmm, you are right; having .jq files in ~/.jq is different from having a ~/.jq that is a file.

Only if ~/.jq is a file, its content is sourced: https://github.com/jqlang/jq/blob/5029328d35f3e60037970d27f350a742af41aa02/src/linker.c#L423-L431

~/.jq as a directory is just a directory that happens to be in the default search path, it has no other special behaviour.

I don't know what the main.jq thing is, ~~it looks like a change a made after jq 1.6 was released, and never documented.~~ It was actually introduced in 1.6, but never documented.

[systemmonkey42]

Also tried main.jq which seem to be a thing but with no success. Looking at the code around here

100% if the ~/.jq directory could contain an autoloading "main" module I would be very happy.

The main.jq business is to support loading ./MODULE/jq/main.jq when you call include "MODULE". This allows main.jq to import more files and have them sourced locally.

EDIT: I take that back. A quick test shows when module "X" calls include "Y", Y is sourced locally (relative to my currect directory) instead of locally to "X". This is a bug, but its tangential to this thread.

[systemmonkey42]

This is a bug, but its tangential to this thread.

To clarify what I mean by this..

As a test, I created the file ~/.jq/test/jq/main.jq which simply contains include "second";

This is the result of some linuxy magic show how "second" is located.

$ strace -f jq -n 'include "test"; .'|&grep -E 'test|second'
newfstatat(AT_FDCWD, "./test.jq", 0x7ffc384aa420, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./test/jq/main.jq", 0x7ffc384aa420, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./test/test.jq", 0x7ffc384aa420, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/user/.jq/test.jq", 0x7ffc384aa420, 0) = -1 ENOENT (No such file or directory)
readlink("/home/user/.jq/test", 0x7ffc384a9ab0, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/user/.jq/test/jq", 0x7ffc384a9ab0, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/user/.jq/test/jq/main.jq", 0x7ffc384a9ab0, 1023) = -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/home/user/.jq/test/jq/main.jq", {st_mode=S_IFREG|0644, st_size=19, ...}, 0) = 0
openat(AT_FDCWD, "/home/user/.jq/test/jq/main.jq", O_RDONLY) = 3
read(3, "\ninclude \"second\";\n", 4096) = 19
newfstatat(AT_FDCWD, "./second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./second/jq/main.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./second/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/user/.jq/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/user/.jq/second/jq/main.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/user/.jq/second/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/jq/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/jq/second/jq/main.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/jq/second/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/second/jq/main.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "./../lib/second/second.jq", 0x7ffc384aa250, 0) = -1 ENOENT (No such file or directory)
write(2, "jq: error: module not found: sec"..., 37jq: error: module not found: second

Note that it will search for ./second instead of ~/.jq/test/jq/second.jq...

Since it is being included by "test/jq/main.jq" is should be sourced there first.

This will help ensure any package containing multiple files can get sourced correctly without namespacing issues.

[aberezin]

+1 for feature request

[kirkoman]

Found related past issues #1501 and #2208.