Documentation clarification: -l is useable in require statements, but -L and -I are not

[psath]

Summary of Problem

According to C interop docs you can use -L and -I flags in require statements to indicate non-standard header and library search directories, but on main (cb81008e74565fa) for

module myFoo {
  use CTypes;
  require "-L../../path/to/userspace/lib";
  require "-luserlib";
  ...
}

I am getting

wrapper.chpl:4: error: file '-L../../path/to/userspace/lib' does not have a recognized suffix

It does not complain about -l statements, though the linker obviously fails later because it doesn't know where to look for them..

As an aside, is there/should there be support for --isystem?

[psath]

In case it matters, I am using system "LLVM 17" from ROCm 6.1.0 with the AMD GPU backend, using cherry-picks from @milthorpe's branch discussed in #24497 and a couple minor tweaks to deal with ROCm changes/reversions between 6.0.2 and 6.1.0. However I'm not doing anything on the GPU (yet) and it seems orthogonal.

[lydia-duncan]

I'm not seeing any tests in our testing system of trying to use -L and -I in require statements themselves. Looking at the wording, I think it might be trying to say that you need to use the Chapel compiler's version of those flags, but I'm not sure. All the ones I see of -luserlib have -I and -L in their compilation flags, not in require statements as well. @bradcray - do you remember if this was something we intended to have work with require statements?

[bradcray]

@psath : Thanks for pointing out this discrepency. I believe that this may be a case where the implementation is correct (or at least, "as intended") and the documentation is out-of-date, and strongly suspect that the mistake is mine.

As I recall, when we were originally designing and implementing the require statement, we'd included support for -I and -L flags; but in early uses of it, there was concern within the team that directories/paths shouldn't be embedded into source code and should always be outside in Makefiles or build scripts. I remember a debate along the lines of "Well, isn't it the programmer's choice whether to make their code potentially brittle in that way or not?" but think we ended up deciding not to include such support in the source, at least for the time being, to be conservative.

So I think the "easy" fix for this issue would be to remove the mention of the support from the docs. But I'd be curious whether you think it is appropriate (or maybe "isn't inappropriate") to support file locations in source using require with -I/-L flags. I suspect it would not be difficult to re-enable support for the -I/-L options, if we decided that it was the right thing to do.

W.r.t. -isystem, that's not a flag I was familiar with (had to Google it to see what it was) and I don't recall us ever discussing whether we should support it. If you'd like to advocate for it, I think it'd be reasonable to file a feature request for it, and I suspect this too would be easy to implement (essentially as a pass-through to the back-end compiler invocation).

[psath]

Ok thanks for the clarification that it is a documentation issue, Feel free to re-tag/title as appropriate

I've already managed to throw them in the Makefile. I think the philosophical standpoint of using require for "what the module needs", but leaving the [machine-dependent] "where to find it" in the Makefile is reasonable.

I think we can table -isystem. I have no imminent need and can open a new issue if so. The only cases I've had to personally use it are when some system header I can't modify is using angle braces to #include or (more often) #include_next something that has different versions in system and user space, and I need to adjust the search order to hit the userspace version... think "inattentive admins and urgent need for newer version of libFoo".

[bradcray]

@psath: Thanks for the feedback. I hadn't looked at the docs prior to my previous comment. Is the part you're referring to, the following (emphasis mine):

.. code-block:: chapel

require "foo.h", "foo.c";

This has an effect similar to adding foo.h and foo.c to the Chapel compiler's command line. Filenames are interpreted as expressing a path relative to the directory in which the source file lives. You can also use the compiler's -I and -L flags to indicate search directories for headers or library files.

If so, do you think the following wording clarifies things sufficiently (where I'm thinking I'll also move it below the bit that has the "-lfoo" example since talking about -L above seems premature before showing the -lfoo case):

You can specify where the compiler can find header files and libraries specified in require statements using its command-line -I and -L flags, respectively.

[psath]

I think my root confusion stemmed from the fact that require "-luserlib" is putting a compiler flag into the source code. While the current text doesn't imply that any arbitrary flag could be placed there, the explicit mention of -I and -L path flags, and the necessary interrelatedness seemed intentional enough to draw my conclusion. Essentially it sounded like require is a way to specify to the compiler certain include/linker information within the source code.

You can specify where the compiler can find header files and libraries specified in require statements using its command-line -I and -L flags, respectively.

So I'm not sure this phrasing is explicit enough to not leave that opening for misunderstanding (because -l is ultimately a command-line flag, so why can't I use other related flags in the same place?). How about some variation on:

Any headers or libraries specified in require statements must be locatable by the backend compiler's search paths. If additional directories are required, the corresponding C/++ -I and -L flags must be added to the chpl compiler's invocation, outside the .chpl source file.

[bradcray]

That totally makes sense. Thinking a bit more about why we expressed libraries as require "-luserlib"; rather than, say, require "userlib.a";, I think it came down to wanting to avoid locking .a vs. .so assumptions into the source code. That said, the reliance on -l is admittedly a little weird, and the confusion around "What other flags can I put in here?" is understandable.

I like your suggestion, thanks for making it!

[bradcray]

@psath : It took me awhile to get back to this, but I've made some changes taking inspiration from your suggestion above and think we're in a strictly better place now. Thanks for bringing this potential for confusion to our attention!