including externs definitions in "codeql test"

[franktip]

Would it be possible to give the “codeql test” command an option to include externs and/or make this something that one can specify in a config file? Right now, the approach we have taken is to copy/symlink externs definitions into the directory containing test files, but this solution seems unsatisfactory/fragile, particularly if one wants to use the standard externs definitions.

My question relates to JavaScript, but the same issue presumably arises for other languages.

[hmakholm]

Currently we have very limited means for passing through options from the codeql command line to the language-specific parts of database creation. There's a design in the works to remedy that in a structured way, which will hopefully land over the summer.

For the specific point about externs, perhaps someone in @github/codeql-javascript can comment?

[p0]

I briefly discussed this with @github/codeql-javascript. It seems like we would like to assume the externs are always available in the database, and so I think we should make a pre-finalise script that extracts them.

[asgerf]

Sorry for taking this long getting back to you @franktip. I'm afraid your workaround will have to do.

For a while we were considering extracting externs in tests by default, but have decided against it for two reasons:

• Tests will take longer. Even a few seconds/test is noticable.
• Tests will find results inside the externs, which is surprising, and many of our tests would need to be updated to exclude such results.

[franktip]

Hi Asger,

I understand that you don't want to include the externs by default for performance reasons, but would it be possible to make it an option that users need to specify/enable explicitly in a config file? I would really appreciate it if you could come up with a solution so that I won't have to copy/symlink the externs definitions in my tests.

Let me know what you think.

best regards,

-Frank

[asgerf]

Right, adding an option for it would indeed be very useful. I'll re-open the issue, but resolution will unfortunately have to wait until the method for passing options has landed (see @hmakholm's comment above).

Currently we have very limited means for passing through options from the codeql command line to the language-specific parts of database creation. There's a design in the works to remedy that in a structured way, which will hopefully land over the summer.