google
Issue 57808: libssh: Coverage build failure
Couple of weeks ago we got an issue about coverage build failure:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57808
The log does not show anything useful and if I see right the only failure is the following part in the end of the log:
Starting Step #16
Step #16: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #16: CommandException: No URLs matched: /workspace/srcmap.json
Finished Step #16
ERROR
ERROR: build step 16 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1
which I have no idea what does it mean nor if that is something that can be fixed from our side. Around the time when the issue was filled, there were no changes to the repository master branch for more than one week.
Is it some issue on the oss-fuzz side or did I miss some part of the coverage log?
This is strange: I am unable to reproduce this build failure locally via:
python infra/helper.py build_image libssh
python infra/helper.py build_fuzzers --sanitizer address libssh
Recent build logs seem fine, too.
Requesting help from @oliverchang.
@Alan32Liu this is a coverage build failure. Our UI https://oss-fuzz-build-logs.storage.googleapis.com/index.html#libssh is reporting a failure (I think the UI is semi broken though, it shows me the fuzzer build logs instead of the coverage build ones).
^above bug I pointed out seemed to go away. Here is failure https://oss-fuzz-build-logs.storage.googleapis.com/log-b4654afd-feb1-4412-a4d9-2e931bf6da1c.txt
Thanks! Re. the UI issue, do we use an identical URL (https://oss-fuzz-build-logs.storage.googleapis.com/index.html#libssh) for both the fuzzer build and coverage build (and the introspector build)?
Thanks! Re. the UI issue, do we use an identical URL (https://oss-fuzz-build-logs.storage.googleapis.com/index.html#libssh) for both the fuzzer build and coverage build (and the introspector build)?
Yup
Got another ping today about the failed coverage, but I still do not see any failure in the coverage build log that I could fix. Can you clarify if there is something I can do or if this is some infra issue?
@Alan32Liu I think the srcmap command is failing. Can you check this out please?
A quick update:
srcmap works fine in gcr.io/oss-fuzz/libssh when I tested it locally.
The command prints the same outputs and saves the file in the indented directory.
In fact, the following content was generated by the cat /workspace/srcmap.json command, proving the existence of the file:
Step #2 - "srcmap": {
Step #2 - "srcmap": "/src/libssh": {
Step #2 - "srcmap": "type": "git",
Step #2 - "srcmap": "url": "https://git.libssh.org/projects/libssh.git",
Step #2 - "srcmap": "rev": "c3aa0cb18225f2f9fcefd96aad327c6ffbb2d77c"
Step #2 - "srcmap": }
Step #2 - "srcmap": }
I will check if some code accidentally removed that file later.
Three days ago, the coverage build was successful. Not sure if anything changed since then on your side. Lets keep it open if it will re-open or if we will get a new reports.
Thanks, that's good to hear!
I did not change anything, though, because I could not figure out why that srcmap.json was missing in the first place.
I don't think we have deployed any new ClusterFuzz commit.
But there are changes to OSS-Fuzz.
I am a bit inclined to close it and re-open it if that (or similar) error happens. WDYT @jonathanmetzman ?
As I was affraid, the new issue was automatically filled again with the same symptoms: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=58788
Interesting, I don't think anything has changed. Would this be non-deterministic somehow? I will continue to look into this when I have time.
And another one after the previous one was closed:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60875#c1
The build log does not show anything usable either ...
ping? Any update on this?
I just noticed one more exception coming from most of the steps:
Step #6: CommandException: 1 files/objects could not be removed.
any idea what could this mean? Is there a way to get more verbose logs to see what was the actual object?
We are spammed with the mails about coverage build failure for well over 6 months every week.
Ping? Any update 6 months later? The previous message is still present in the coverage builds, but I do not see what is the its cause.
We are being spammed by the bugzilla every week about this: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67123
This is interesting, because the last passing coverage build had libfuzzer failures: https://oss-fuzz-build-logs.storage.googleapis.com/log-f3cd498e-85bc-409e-bb32-f2f32b9e75d0.txt
Step #5: Error occured while running ssh_client_config_fuzzer:
Step #5: INFO: Running with entropic power schedule (0xFF, 100).
Step #5: INFO: Seed: 3134367418
Step #5: MERGE-OUTER: 3646 files, 0 in the initial corpus, 0 processed earlier
Step #5: MERGE-OUTER: attempt 1
Step #5: INFO: Running with entropic power schedule (0xFF, 100).
Step #5: INFO: Seed: 3134395183
Step #5: INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
Step #5: MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge117.txt'
Step #5: MERGE-INNER: 3646 total files; 0 processed earlier; will process 3646 files now
Step #5: #1 pulse exec/s: 0 rss: 31Mb
Step #5: #2 pulse exec/s: 0 rss: 31Mb
Step #5: #4 pulse exec/s: 0 rss: 31Mb
Step #5: #8 pulse exec/s: 0 rss: 31Mb
Step #5: #16 pulse exec/s: 0 rss: 31Mb
Step #5: #32 pulse exec/s: 0 rss: 31Mb
Step #5: #64 pulse exec/s: 0 rss: 31Mb
Step #5: #128 pulse exec/s: 0 rss: 31Mb
Step #5: #256 pulse exec/s: 0 rss: 31Mb
Step #5: #512 pulse exec/s: 0 rss: 31Mb
Step #5: /bin/sh: 1: Syntax error: EOF in backquote substitution
Step #5: /bin/sh: 1: /: Permission denied
Step #5: /bin/sh: 1: /root: Permission denied
Step #5: /bin/sh: 1: “: not found
Step #5: /bin/sh: 1: : not found
Step #5: /bin/sh: 1: ip: not found
Step #5: #1024 pulse exec/s: 0 rss: 31Mb
Step #5: ALARM: working on the last Unit for 101 seconds
Step #5: and the timeout value is 100 (use -timeout=N to change)
Step #5: MS: 0 ; base unit: 0000000000000000000000000000000000000000
Step #5: 0x69,0x6e,0x63,0x6c,0x75,0x64,0x65,0x9,0x2f,0x2a,0x2f,0x2a,0x32,
Step #5: include\011/*/*2
Step #5: artifact_prefix='./'; Test unit written to ./timeout-f2505435d3d085b2d7fcd123bf0ddcf4f9259180
Step #5: Base64: aW5jbHVkZQkvKi8qMg==
Step #5: ==125== ERROR: libFuzzer: timeout after 101 seconds
...
I am not an OSS-Fuzz maintainer, so I don't have any further insight, but my recommendation would be to fix the outstanding non-coverage related issues listed in https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Alibssh&can=2
Ping? Any update 6 months later? The previous message is still present in the coverage builds, but I do not see what is the its cause.
We generally don't have the bandwidth to investigate the unique behavior of individual projects, there's over 1000 in OSS-Fuzz. It looks like this srcmap file is written to /workspace early on in the build process and later gets deleted. Any idea why this happens? It seems to be unique to libssh.
This is interesting, because the last passing coverage build had libfuzzer failures: https://oss-fuzz-build-logs.storage.googleapis.com/log-f3cd498e-85bc-409e-bb32-f2f32b9e75d0.txt
Thank you for having a look into this! This input is a string include./*/*2, which is parsed by the libssh config parser attempting to read the content of the the / directory of the container where it runs, which can result in any unexpected issues.
As a workaround, we can disable some subset of the code attempting to touch the content of the container while parsing one configuration file with some limitation on what will actually be tested.
I am not an OSS-Fuzz maintainer, so I don't have any further insight, but my recommendation would be to fix the outstanding non-coverage related issues listed in https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Alibssh&can=2
I think I already tried that, but these are some pattern matching stuff that I was not able to limit reasonably more, but I will certainly try again.
It looks like this srcmap file is written to /workspace early on in the build process and later gets deleted. Any idea why this happens? It seems to be unique to libssh.
Only case I can think of that could cause it would be if the fuzzers would be running there with inputs that would contain execution of rm srcmap. Given the richness of the OpenSSH configuration file and matching syntax, this is what would cause execution of this, which is certainly not ideal and might be a good candidate to skip at least during the fuzzing builds.
Today's was passing as well (I guess intermittently as well): https://oss-fuzz-build-logs.storage.googleapis.com/log-9cc490a8-416f-4dd0-9105-91f0bac01ce6.txt
Thank you for having a look into this. Could it be an issue I describe and would it be a good idea to prevent the fuzzers to run exec like this? If so, I have a MR that should avoid this
https://gitlab.com/libssh/libssh-mirror/-/merge_requests/509
I suspect this could be something like this, but I do not want to shovel through the corpus.
If you think it could help, we can try to merge it, provide this flag in the fuzzers build and see if it will help.
https://gitlab.com/libssh/libssh-mirror/-/merge_requests/509
Yes, it makes sense to exclude arbitrary commands such as rm -rf / from the fuzz input from execution. (I can't comment on the exact changes in the merge request, though)
There was some discussion around detecting stuff like this, but I think it went stale: https://github.com/google/clusterfuzz/issues/2750#issuecomment-1221885545
I suspect this could be something like this, but I do not want to shovel through the corpus.
If you want to test it, you can also temporarily include the patch and flag in the oss-fuzz project folder only, then see if it works, then merge it, then remove the temporary test.