Running fontbakery using subprocess results in error

[aaronbell]

Observed behaviour

I created a python script that runs fontbakery on a set of fonts and makes use of the subprocess.check_call() command: subprocess.check_call(["fontbakery", "check-universal", path+"/*"+extension, "--loglevel", "WARN", "--ghmarkdown", "_TEST/"+outputName+".md"])

The testing process runs (it shows the progress bar), and exports the markdown file, but then errors out:

Traceback (most recent call last):
  File "fontbakery.py", line 54, in <module>
    subprocess.check_call(["fontbakery", "check-universal", path+"/*"+extension, "--loglevel", "WARN", "--ghmarkdown", "_TEST/"+outputName+".md"])
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['fontbakery', 'check-universal', path+"/*"+extension, '--loglevel', 'WARN', '--ghmarkdown', '_TEST/"+outputName+".md']' returned non-zero exit status 1.
(fontbakery) Aarons-iMac: aaronbell$ 

Implementing try: except: avoids the issue, but this doesn't seem ideal.

Expected behaviour

Fontbakery should complete and not return an error :)

[arrowtype]

This may be related to the fact that running FontBakery in a shell script with set -e (stop on errors) will stop the script after FB completion. So, if I include FontBakery in a shell script (as anything but the final command) I have to add in a set +e just before.

I have always assumed that FontBakery must be returning its checks readout as an error status 1 notification, though I don't exactly understand the mechanisms behind that or if there is a reason that it does so.

[graphicore]

@arrowtype

I have always assumed that FontBakery must be returning its checks readout as an error status 1 notification, though I don't exactly understand the mechanisms behind that or if there is a reason that it does so.

See:

https://github.com/googlefonts/fontbakery/blob/0e672ee0bbb56311794d7bff57ff26d272938625/Lib/fontbakery/commands/check_profile.py#L365-L366

That's I think the line, when you get ERROR or FAIL statuses it returns as a failed command. Depending on your context, this could be not what you want. Maybe it should be changed in general or we add a command line flag to make it controllable.

I think, especially if we had a way to convert edge case FAILs into e.g. SKIPs, failing the command on a FAIL could be helpful for some use cases, e.g. in a script that does CI/publishing and relies on passing a profile before continuing.

[aaronbell]

I think a command line flag would make the most sense. I can see where having the error response could be useful, but if you aren't planning on doing anything with that info (since you'll be looking at the font bakery report itself), then it would be nice to be able to ignore it, especially when batching a bunch of reviews.

[arrowtype]

Came back to this, hoping there might be an answer or fix.

I’d still appreciate being able to run batches of checks via a shell script, without getting a failed status returned.

I know there’s a lot going on, so this is really just a "bump" to this idea. Thanks!

[arrowtype]

Later update: it seems like the newish argument --error-code-on solves this. For instance, adding --error-code-on FATAL seems to only return Error 1 if a FATAL result is encountered. It looks like the level can be one of DEBUG, PASS, SKIP, INFO, WARN, FAIL, FATAL, ERROR.