coveralls-public icon indicating copy to clipboard operation
coveralls-public copied to clipboard

Published 20 hours ago • verified publisher icon lemurheavy

Merging covered relevant lines from different jobs

Open aenachescu opened this issue 3 years ago • 18 comments

If I have a build with more jobs the number of relevant lines is not computed as it should be. As far as I understand a line is considered relevant if it is a relevant line in all jobs. Shouldn't a line considered relevant if it is a relevant line in at least one job? For example: Job1: https://coveralls.io/jobs/72975052/source_files/4640888347 Relevant lines: 32, 36, 49, 52, 54, 56, 58, 60, 92, 94, 109, 111, 127, 129 Job2: https://coveralls.io/jobs/72975096/source_files/4640912663 Relevant lines: 32, 36, 66, 69, 71, 73, 75, 77, 96, 98, 100, 102, 109, 111, 127, 129 Final result: https://coveralls.io/builds/36073657/source?filename=test/check_build_configuration/main.c Relevant lines: 32, 36, 109, 111, 127, 129

I would expect the relevant lines to be: 32, 36, 49, 52, 54, 56, 58, 60, 66, 69, 71, 73, 75, 77, 92, 94, 96, 98, 100, 102, 109, 111, 127, 129.

Is there any solution to fix my problem?

aenachescu avatar Jan 05 '21 16:01 aenachescu

@aenachescu I agree. Relevant lines should be considered relevant in all jobs, but only if all things remain equal, which is to say, if the same coverage library is being used in both jobs, instrumenting the same code files in the same way.

Is it possible that is different across your two jobs?

If not, then the next step will be to re-run the previous job (Job #27) in verbose mode so we can see exactly what's being sent to the coveralls API.

Coveralls does not determine what lines of code are considered relevant. Your test coverage library does that, first, then CI sends us that determination in the form of your coverage report. We simply read in the coverage report and log it for each job, each build. For a given build, we do combine the coverage reports for each job, but, again, we don't determine which lines are relevant.

If you re-run Job #27 in verbose mode, you should find that in the generated JSON (and potentially, one step before in the generated LCOV), the number of relevant lines has already been determined, and I suspect it will be as you describe above / as follows:

/test/check_build_configuration/main.c:

  • Job 27.1: 14 relevant lines
  • Job 27.3: 16 relevant lines

It is very strange to me that the combined result indicates only 6 relevant lines: https://coveralls.io/builds/36073657

Screen Shot 2021-01-05 at 1 17 20 PM

The fact that this result is the union of relevant lines across the two jobs makes me think we're getting covered lines in place of relevant lines on the incoming reports.

That's why it will help to see verbose output from the CI build logs for both of those jobs (27.1 & 27.3).

The method to invoke verbose mode depends on what language integration you're using.

Per our C/C++ listings, I'll assume yours is: cpp-coveralls. Per usage instructions there, it seems you can pass the --verbose flag to the coveralls command.

afinetooth avatar Jan 05 '21 21:01 afinetooth

Is it possible that is different across your two jobs?

No, in the first job I test the code using gcc and in the second job I test the code using g++ compiler.

In order to report the coverage to coveralls I use coveralls-lcov tool. I passed --verbose flag to coveralls-lcov and I got these json outputs:

  • Job #30.1 {"service_name":"travis-ci","service_job_id":"469101862","git":{"head":{"id":"72d9b9702faffa9b04b834413395c9dff061d4ea\n","committer_email":"[email protected]\n","committer_name":"Alin Enachescu\n","author_email":"[email protected]\n","author_name":"Alin Enachescu\n","message":"print coveralls-lcov output\n"},"remotes":[],"branch":"test-coveralls"},"source_files":[{"name":"test/check_build_configuration/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n\nint main()\n{\n // print operating system name\n#ifdef NBP_OS_LINUX\n printf(\"linux \");\n#elif defined NBP_OS_WINDOWS\n printf(\"windows \");\n#elif defined NBP_OS_MAC\n printf(\"mac \");\n#elif defined NBP_OS_CUSTOM\n printf(\"custom \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_OS_LINUX\n\n // print compiler name and version\n#ifdef NBP_COMPILER_GCC\n printf(\"gcc-\");\n\n#if NBP_COMPILER_VERSION == 5\n printf(\"5 \");\n#elif NBP_COMPILER_VERSION == 6\n printf(\"6 \");\n#elif NBP_COMPILER_VERSION == 7\n printf(\"7 \");\n#elif NBP_COMPILER_VERSION == 8\n printf(\"8 \");\n#elif NBP_COMPILER_VERSION == 9\n printf(\"9 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_VERSION\n\n#elif defined NBP_COMPILER_GXX\n printf(\"g++-\");\n\n#if NBP_COMPILER_VERSION == 5\n printf(\"5 \");\n#elif NBP_COMPILER_VERSION == 6\n printf(\"6 \");\n#elif NBP_COMPILER_VERSION == 7\n printf(\"7 \");\n#elif NBP_COMPILER_VERSION == 8\n printf(\"8 \");\n#elif NBP_COMPILER_VERSION == 9\n printf(\"9 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_VERSION\n\n#elif defined NBP_COMPILER_CLANG\n printf(\"clang-\");\n#elif defined NBP_COMPILER_CUSTOM\n printf(\"custom \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_GCC\n\n // print standard\n#ifdef NBP_LANGUAGE_STANDARD_C99\n printf(\"-std=c99 \");\n#elif defined NBP_LANGUAGE_STANDARD_C11\n printf(\"-std=c11 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP03\n printf(\"-std=c++03 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP11\n printf(\"-std=c++11 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP14\n printf(\"-std=c++14 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP17\n printf(\"-std=c++17 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_LANGUAGE_STANDARD_C99\n\n // print platform\n#ifdef NBP_PLATFORM_32BIT\n printf(\"-m32 \");\n#elif defined NBP_PLATFORM_64BIT\n printf(\"-m64 \");\n#else\n printf(\"unknown\");\n#endif // end if NBP_PLATFORM_32BIT\n\n // print sanitizer\n#ifdef NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n printf(\"address \");\n#elif defined NBP_TEST_MODE_THREAD_SANITIZER_ENABLED\n printf(\"thread \");\n#elif defined NBP_TEST_MODE_LEAK_SANITIZER_ENABLED\n printf(\"leak \");\n#elif defined NBP_TEST_MODE_UB_SANITIZER_ENABLED\n printf(\"ub \");\n#endif // end if NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\n printf(\"\\n\");\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,20,null,null,null,20,null,null,null,null,null,null,null,null,null,null,null,null,20,null,null,4,null,4,null,4,null,4,null,4,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,10,null,10,null,null,null,null,null,null,null,null,null,null,null,null,null,null,10,null,10,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,20,null,20,null]},{"name":"test/check_leak_sanitizer/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#ifdef NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\n#ifdef NBP_LANGUAGE_CPP\nextern \"C\" {\n#endif // end if NBP_LANGUAGE_CPP\n\nconst char* __asan_default_options()\n{\n return \"detect_leaks=0\";\n}\n\n#ifdef NBP_LANGUAGE_CPP\n}\n#endif // end if NBP_LANGUAGE_CPP\n\n#endif // end if NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\nint main()\n{\n void* ptr = malloc(100);\n printf(\"value = %p\\n\", ptr);\n ptr = NULL;\n\n printf(\"check_leak_sanitizer completed successfully\\n\");\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,20,null,20,20,20,null,20,null,20,null]},{"name":"test/tmp1/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#if NBP_VERSION > NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg1 = \"greater than 1.0.0\";\n#else\nconst char* msg1 = \"not greater than 1.0.0\";\n#endif\n\n#if NBP_VERSION >= NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg2 = \"greater than or equal to 1.0.0\";\n#else\nconst char* msg2 = \"less than 1.0.0\";\n#endif\n\n#if NBP_VERSION > NBP_MAKE_VERSION(0, 9, 9)\nconst char* msg3 = \"greater than 0.9.9\";\n#else\nconst char* msg3 = \"not greater than 0.9.9\";\n#endif\n\nint* ptr = NULL;\n\nint main()\n{\n printf(\n \"nbp version = %s\\n\"\n \"nbp version = %x\\n\",\n NBP_VERSION_STR,\n NBP_VERSION);\n\n printf(\"message#1 = %s\\n\", msg1);\n printf(\"message#2 = %s\\n\", msg2);\n printf(\"message#3 = %s\\n\", msg3);\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,20,null,20,null,null,null,null,null,20,20,20,null,20,null]},{"name":"test/tmp2/main.c","source":"/*\nNo Bugs in Production (NBP) Library Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#if NBP_VERSION > NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg1 = \"greater than 1.0.0\";\n#else\nconst char* msg1 = \"not greater than 1.0.0\";\n#endif\n\n#if NBP_VERSION >= NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg2 = \"greater than or equal to 1.0.0\";\n#else\nconst char* msg2 = \"less than 1.0.0\";\n#endif\n\n#if NBP_VERSION > NBP_MAKE_VERSION(0, 9, 9)\nconst char* msg3 = \"greater than 0.9.9\";\n#else\nconst char* msg3 = \"not greater than 0.9.9\";\n#endif\n\nint main(int argc, const char** argv)\n{\n if (argc != 2) {\n return 0;\n }\n\n printf(\n \"nbp version = %s\\n\"\n \"nbp version = %x\\n\",\n NBP_VERSION_STR,\n NBP_VERSION);\n\n printf(\"message#1 = %s\\n\", msg1);\n printf(\"message#2 = %s\\n\", msg2);\n printf(\"message#3 = %s\\n\", msg3);\n\n printf(\"%s\", argv[1]);\n\n return 1;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,20,null,20,0,null,null,20,null,null,null,null,null,20,20,20,null,20,null,20,null]}],"parallel":false}
  • Job #30.3 {"service_name":"travis-ci","service_job_id":"469101864","git":{"head":{"id":"72d9b9702faffa9b04b834413395c9dff061d4ea\n","committer_email":"[email protected]\n","committer_name":"Alin Enachescu\n","author_email":"[email protected]\n","author_name":"Alin Enachescu\n","message":"print coveralls-lcov output\n"},"remotes":[],"branch":"test-coveralls"},"source_files":[{"name":"test/check_build_configuration/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n\nint main()\n{\n // print operating system name\n#ifdef NBP_OS_LINUX\n printf(\"linux \");\n#elif defined NBP_OS_WINDOWS\n printf(\"windows \");\n#elif defined NBP_OS_MAC\n printf(\"mac \");\n#elif defined NBP_OS_CUSTOM\n printf(\"custom \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_OS_LINUX\n\n // print compiler name and version\n#ifdef NBP_COMPILER_GCC\n printf(\"gcc-\");\n\n#if NBP_COMPILER_VERSION == 5\n printf(\"5 \");\n#elif NBP_COMPILER_VERSION == 6\n printf(\"6 \");\n#elif NBP_COMPILER_VERSION == 7\n printf(\"7 \");\n#elif NBP_COMPILER_VERSION == 8\n printf(\"8 \");\n#elif NBP_COMPILER_VERSION == 9\n printf(\"9 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_VERSION\n\n#elif defined NBP_COMPILER_GXX\n printf(\"g++-\");\n\n#if NBP_COMPILER_VERSION == 5\n printf(\"5 \");\n#elif NBP_COMPILER_VERSION == 6\n printf(\"6 \");\n#elif NBP_COMPILER_VERSION == 7\n printf(\"7 \");\n#elif NBP_COMPILER_VERSION == 8\n printf(\"8 \");\n#elif NBP_COMPILER_VERSION == 9\n printf(\"9 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_VERSION\n\n#elif defined NBP_COMPILER_CLANG\n printf(\"clang-\");\n#elif defined NBP_COMPILER_CUSTOM\n printf(\"custom \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_COMPILER_GCC\n\n // print standard\n#ifdef NBP_LANGUAGE_STANDARD_C99\n printf(\"-std=c99 \");\n#elif defined NBP_LANGUAGE_STANDARD_C11\n printf(\"-std=c11 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP03\n printf(\"-std=c++03 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP11\n printf(\"-std=c++11 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP14\n printf(\"-std=c++14 \");\n#elif defined NBP_LANGUAGE_STANDARD_CPP17\n printf(\"-std=c++17 \");\n#else\n printf(\"unknown \");\n#endif // end if NBP_LANGUAGE_STANDARD_C99\n\n // print platform\n#ifdef NBP_PLATFORM_32BIT\n printf(\"-m32 \");\n#elif defined NBP_PLATFORM_64BIT\n printf(\"-m64 \");\n#else\n printf(\"unknown\");\n#endif // end if NBP_PLATFORM_32BIT\n\n // print sanitizer\n#ifdef NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n printf(\"address \");\n#elif defined NBP_TEST_MODE_THREAD_SANITIZER_ENABLED\n printf(\"thread \");\n#elif defined NBP_TEST_MODE_LEAK_SANITIZER_ENABLED\n printf(\"leak \");\n#elif defined NBP_TEST_MODE_UB_SANITIZER_ENABLED\n printf(\"ub \");\n#endif // end if NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\n printf(\"\\n\");\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,null,null,40,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,null,8,null,8,null,8,null,8,null,8,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,10,null,10,null,10,null,10,null,null,null,null,null,null,20,null,20,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,40,null]},{"name":"test/check_leak_sanitizer/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#ifdef NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\n#ifdef NBP_LANGUAGE_CPP\nextern \"C\" {\n#endif // end if NBP_LANGUAGE_CPP\n\nconst char* __asan_default_options()\n{\n return \"detect_leaks=0\";\n}\n\n#ifdef NBP_LANGUAGE_CPP\n}\n#endif // end if NBP_LANGUAGE_CPP\n\n#endif // end if NBP_TEST_MODE_ADDRESS_SANITIZER_ENABLED\n\nint main()\n{\n void* ptr = malloc(100);\n printf(\"value = %p\\n\", ptr);\n ptr = NULL;\n\n printf(\"check_leak_sanitizer completed successfully\\n\");\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,40,40,40,null,40,null,40,null]},{"name":"test/tmp1/main.c","source":"/*\nNo Bugs in Production (NBP) Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#if NBP_VERSION > NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg1 = \"greater than 1.0.0\";\n#else\nconst char* msg1 = \"not greater than 1.0.0\";\n#endif\n\n#if NBP_VERSION >= NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg2 = \"greater than or equal to 1.0.0\";\n#else\nconst char* msg2 = \"less than 1.0.0\";\n#endif\n\n#if NBP_VERSION > NBP_MAKE_VERSION(0, 9, 9)\nconst char* msg3 = \"greater than 0.9.9\";\n#else\nconst char* msg3 = \"not greater than 0.9.9\";\n#endif\n\nint* ptr = NULL;\n\nint main()\n{\n printf(\n \"nbp version = %s\\n\"\n \"nbp version = %x\\n\",\n NBP_VERSION_STR,\n NBP_VERSION);\n\n printf(\"message#1 = %s\\n\", msg1);\n printf(\"message#2 = %s\\n\", msg2);\n printf(\"message#3 = %s\\n\", msg3);\n\n return 0;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,24,null,null,null,16,null,40,40,40,null,40,null]},{"name":"test/tmp2/main.c","source":"/*\nNo Bugs in Production (NBP) Library Library\nhttps://github.com/aenachescu/nbplib\n\nLicensed under the MIT License <http://opensource.org/licenses/MIT>.\nSPDX-License-Identifier: MIT\nCopyright (c) 2019-2020 Alin Enachescu <https://github.com/aenachescu>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n*/\n\n#include \"nbp.h\"\n\n#include <stdio.h>\n#include <stdlib.h>\n\n#if NBP_VERSION > NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg1 = \"greater than 1.0.0\";\n#else\nconst char* msg1 = \"not greater than 1.0.0\";\n#endif\n\n#if NBP_VERSION >= NBP_MAKE_VERSION(1, 0, 0)\nconst char* msg2 = \"greater than or equal to 1.0.0\";\n#else\nconst char* msg2 = \"less than 1.0.0\";\n#endif\n\n#if NBP_VERSION > NBP_MAKE_VERSION(0, 9, 9)\nconst char* msg3 = \"greater than 0.9.9\";\n#else\nconst char* msg3 = \"not greater than 0.9.9\";\n#endif\n\nint main(int argc, const char** argv)\n{\n if (argc != 2) {\n return 0;\n }\n\n printf(\n \"nbp version = %s\\n\"\n \"nbp version = %x\\n\",\n NBP_VERSION_STR,\n NBP_VERSION);\n\n printf(\"message#1 = %s\\n\", msg1);\n printf(\"message#2 = %s\\n\", msg2);\n printf(\"message#3 = %s\\n\", msg3);\n\n printf(\"%s\", argv[1]);\n\n return 1;\n}\n","coverage":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,40,null,40,0,null,null,24,null,null,null,16,null,40,40,40,null,40,null,40,null]}],"parallel":false}

aenachescu avatar Jan 06 '21 08:01 aenachescu

Hi @aenachescu, thanks for the details.

So, at a high level, I'm afraid I need to kick the issue back to you.

In other words, there's nothing specific to Coveralls that seems to be acting here. We simply consume the coverage reports we receive, via POSTs to our API, and, from the incoming JSON shown above, you can see the issue starts there.

That is to say that, between the two jobs, for the same file, test/check_build_configuration/main.c, the number of relevant lines cited in each job's report are different. Job 30.1 reports 14 relevant lines, while Job 30.3 reports 16 relevant lines.

This indicates the origin of the issue is with the coverage library / coverage tooling native to your project. In our experience, if the same code is being tested by the same tests, using the same coverage library, there shouldn't be any difference in what that coverage library determines to be the relevant lines for a given file, regardless of the number of passes.

I would like to be more helpful, and for some projects I could be, but the nature of your project falls outside my domain of expertise. I would need you to answer a bunch of questions I have about your project and, even then, I think it's unlikely I'll be able to identify the source of the discrepancy.

That said, I'm willing to help, so, if you'd like more help, let me know and I'll start with my questions.

afinetooth avatar Jan 06 '21 20:01 afinetooth

Hi @afinetooth Somehow the origin of the issue is from gcov. I mean, there are a lot of issues when you try to compute the coverage for template functions/classes in C++ or for code under ifdef (i dont know if you are familiar with C/C++). In my case, I have some code that is compiled only if the compiler is g++ and other code that is compiled only if the compiler is gcc. And when the compiler is g++, the code that is compiled only if the compiler is gcc wont be seen as relevant code because it is not included in the binary. Same when the compiler is gcc. And I can understand that...

But my question is why you do intersection on relevant lines and not union? I see in the json a coverage array that contains null or the number of hits for nth line. In my case there are 2 json outputs and each one has a source file test/check_build_configuration/main.c and this file has a coverage array in each job. Isn't possible to do union on these arrays?

aenachescu avatar Jan 06 '21 21:01 aenachescu

@aenachescu aha. No, I don't have that expertise, but that was going to be my guess, that somehow the different compilers render different lines of the original source relevant / irrelevant. So that helps to know, and seems the likely source of discrepancy.

Your point---why intersection and not union---is actually something I don't understand, that is happening on our side. Please let me examine it and get back to you. I suspect that we expect relevant lines to be relevant (non-null) across all jobs and, therefore, only sum hits for lines that are non-null across all jobs. (We re-build that coverage hash for each file before calculating aggregate coverage.) Let me verify.

afinetooth avatar Jan 06 '21 23:01 afinetooth

I just noticed this problem appears to still happening on the coveralls reports for my latest build: https://coveralls.io/builds/46667478

But there's a lot going on here to try to unpack:

  • Each of these jobs is from a different OS configuration. So there is some code that is only present for certain OS (hence why we are sending the reports for all jobs)
  • Lines 247-255 are a few of the key lines to look at between the various reports, as that is Windows-only code
  • The value reported back on the PR status was 91.597%, and links back to the last job to finish (Windows)
  • The "true" value for coverage on this build is probably 90.16%
  • The summary view shows 89.89% covered
  • I have the parallel jobs flag set in the uploader

Is there an update on this?

vtjnash avatar Feb 18 '22 05:02 vtjnash

Hi @vtjnash.

So at this point, I noticed that you've run at least one more build against that PR and, as designed, the latest PR Comment replaced the previous one. It has coverage at 92% and looks like this:

Screen Shot 2022-02-21 at 12 43 15 PM

That badge also links back to the more recent build: https://coveralls.io/builds/46695736

So I'm going to try to address the same concerns for that build, but I need some help.

You said:

  • The "true" value for coverage on this build is probably 90.16%

Can you clarify why you think that? And what you believe coverage is on the new build?

I noticed you also said:

The summary view shows 89.89% covered

Which I take to be this view from the old build:

Screen Shot 2022-02-21 at 1 04 03 PM

Don't let the value of any one file---even in a project with just one file---mislead you regarding aggregate coverage for the project. There are a couple of reasons why the coverage of one file may not lead intuitively to the aggregate coverage, or, in the case of this one-file project, might not equal the aggregate coverage:

  1. First, the coverage shown for individual files is limited to line coverage, while some projects have reports that also include branch coverage. I don't believe yours does, but this is a common source of confusion.
  2. Second (and likely applicable to your use case), the coverage for the file is derived from several jobs, each of which may have conflicting coverage data for the file. In this case, until confirmed by a review of each coverage report, we can probably assume the source of the discrepancy is what @aenachescu points to above, that our algorithm appears to take the intersection and not union of relevant lines across all "instances" of the same file.

I have been able to confirm that this is designed into the algorithm, and not just an unhandled corner case.

Comments on the aggregation method explain the choice:

If a line is not relevant in one file it should be irrelevant in all. Fix for some coverage libraries which fail to report correct relevancy when file has no coverage

I'm not 100% sure what "fail to report correct relevancy" and "when file has no coverage" mean, or how often such cases occur, but I've asked that question and submitted the situation described in this issue as a conflicting use case that may warrant more nuanced approach.

Thinking about workarounds until such time that this could be addressed...

At the moment, all I can come up with is to:

  1. Rewrite your file in a way that would avoid discrepancies in relevant lines for the main body of your code. Not sure how, but perhaps by splitting it, or by rewriting the statements on the offending line(s) that would force them to always be considered relevant.
  2. Look into your coverage library and see if there's a setting that allows you to force a line to be considered relevant (or perhaps a comment on the line that the library would recognize).

afinetooth avatar Feb 22 '22 17:02 afinetooth

Looking at codecov on the same commit, and examining the build logs for individual jobs to see that 92% is overestimating the total (missing some lines that are only relevant to some platforms, but uncovered on all platforms) and 89% is underestimating (counting as uncovered some lines that are only relevant to other platforms). So 90% seemed a fair estimate for the aggregation of the lines covered/relevant when aggregating running tests across all platforms.

Given the reported value there, I think the aggregate coverage in the comment might be wrong due to: 3. Thirdly, the aggregate coverage is not computed at all when updating the PR comment, but simply uses the value of the last job run, even if using the parallel flag.

vtjnash avatar Feb 22 '22 18:02 vtjnash

Looking at codecov on the same commit, and examining the build logs for individual jobs to see that 92% is overestimating the total (missing some lines that are only relevant to some platforms, but uncovered on all platforms) and 89% is underestimating (counting as uncovered some lines that are only relevant to other platforms). So 90% seemed a fair estimate for the aggregation of the lines covered/relevant when aggregating running tests across all platforms.

Ok, got it. Thanks.

Given the reported value there, I think the aggregate coverage in the comment might be wrong due to: 3. Thirdly, the aggregate coverage is not computed at all when updating the PR comment, but simply uses the value of the last job run, even if using the parallel flag.

Sorry, can you please clarify where "there" is when you say:

Given the reported value there [...]

Can you also explain why you called this (3)?

  1. Thirdly, the aggregate coverage is not computed at all when updating the PR comment, but simply uses the value of the last job run, even if using the parallel flag.

Is it meant to be another reason (besides the two I mention above) why the file coverage may differ from the aggregate coverage?

Also, AFAIK this is not correct, unless I misunderstand the context:

the aggregate coverage is not computed at all when updating the PR comment, but simply uses the value of the last job run, even if using the parallel flag.

I think I might know what you mean. In actuality, coverage is re-computed for every PR commit. There is another issue there that is masking that.

If you're talking about the Status Updates for your PR, the issue is that you should either be seeing status updates for all jobs + a final (aggregate) status update for the build, OR just a status update for the build, depending on your settings. But due to the bug, for some reason, in your PR I am seeing two (2) status updates, one for your final job + one for the overall build. Aside from the bug, though, the status update for the build should always show the aggregate coverage and link back to the build page, which represents the aggregate coverage.

If you're talking about the PR comment on your PR, the badge image should represent aggregate coverage and always links back to the build page, which represents aggregate coverage.

As it does here, for me: https://github.com/vtjnash/Pidfile.jl/pull/11

Screen Shot 2022-02-22 at 11 27 43 AM

Screen Shot 2022-02-22 at 11 28 03 AM

afinetooth avatar Feb 22 '22 19:02 afinetooth

Look into your coverage library and see if there's a setting that allows you to force a line to be considered relevant (or perhaps a comment on the line that the library would recognize).

Yeah, this is not ideal, but some of the builds are from old versions of the build tool, which have less reliable information on relevant lines. I perhaps should only upload builds for the latest version, as intersecting them in this way is leading to even worse reported statistics.

Is it meant to be another reason (besides the two I mention above) why the file coverage may differ from the aggregate coverage?

Yes, I am just guessing there at why it seems to agree exactly with the last build, but disagree with the other places that this is being reported.

I am not sure why that status reporting is odd, since I have the official uploader here, and thought this was the expected configuration: https://github.com/vtjnash/Pidfile.jl/blob/4466af1b9abe1484ed493502a8f1501bc550fc3e/.github/workflows/CI.yml#L53-L62

          parallel: true
  finish:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - name: Coveralls Finished
      uses: coverallsapp/github-action@master
      with:
        github-token: ${{ secrets.github_token }}
        parallel-finished: true

vtjnash avatar Feb 22 '22 19:02 vtjnash

First, your CI config looks fine. Thanks for sharing it.

Yes, I am just guessing there at why it seems to agree exactly with the last build, but disagree with the other places that this is being reported.

I see what you mean. I had not noticed that before, only that the figure correlated with the overalls build.

It does seem odd that the last build would be identical to the overall coverage for the build, and not be closer to an average of all the jobs.

I re-ran the jobs for your previous build---this one: https://coveralls.io/builds/46667478

And got something different for the build coverage %:

Instead of 91.6% it now reads 89.888%, which seems closer to an appropriate coverage given the constituent jobs:

Screen Shot 2022-02-23 at 2 13 27 PM

I don't know why it happened, but it suggests a one-off issue.

To validate, I re-ran the jobs on the newer build as well: https://coveralls.io/builds/46695736

And similarly, the overall coverage changed again, this time to 91.346%.

So, if it is a one-off, it still happened consistently for two builds in a row.

To test further, I re-ran the jobs for the later build again (and again) to see if I continued getting the same overall coverage %---which should be what always happens---and I did consistently get the same result, 91.346%.

So I think there's an issue at play having to do with your build not closing and calculating aggregate coverage, but just taking the last build for the aggregate coverage.

I haven't seen this before and, like I said, I see nothing in your CI config that indicates why it would happen.

Can you verify if this is the case for any other builds?

afinetooth avatar Feb 23 '22 22:02 afinetooth

P.S. I tried it on two more of your PR builds (the last build for each PR):

  • https://coveralls.io/builds/44414158 (for PR 9)
  • https://coveralls.io/builds/37426114 (for PR 7)

And aggregate coverage for both builds changed slightly after I re-ran the jobs:

https://coveralls.io/builds/44414158 (for PR 9)

  • Before: Screen Shot 2022-02-23 at 2 28 25 PM
  • After: Screen Shot 2022-02-23 at 2 29 14 PM

And:

https://coveralls.io/builds/37426114 (for PR 7)

  • Before: Screen Shot 2022-02-23 at 2 30 07 PM
  • After: Screen Shot 2022-02-23 at 2 30 36 PM

I'm looking into what could have happened here. Will feed back asap.

afinetooth avatar Feb 23 '22 22:02 afinetooth

Hi, @vtjnash. I can't find any more cases of PR builds that display the behavior where overall coverage matches the coverage of the last job.

For instance, this build for PR 5 looks normal, so apparently the issue doesn't go that far back.

Unfortunately, we don't keep the data on builds that are replaced by subsequent builds (which is what happens when I manually re-run a build), so I will need you to inform me the next time you see the issue so I can examine your new build. (I probably got a little overzealous when I re-ran the builds for PRs 9 & 7.)

As far as possible causes, I did notice that, across these three (3) builds for PR 9, you have a varying number of parallel jobs. Specifically:

  • https://coveralls.io/builds/44276748 (4 jobs)
  • https://coveralls.io/builds/44322054 (5 jobs)
  • https://coveralls.io/builds/44414149 (3 jobs + 1 failed job)

Do you think it's possible that's due to an errant or duplicate job?

I looked to your latest build to see what your "canonical" number of builds should be and see that there are seven (7) there: https://coveralls.io/builds/46695736

That can be perfectly OK, I'm just wondering if you're always getting the number of jobs you're expecting.

afinetooth avatar Feb 24 '22 20:02 afinetooth

Yeah, there is not much activity on that repo, so I don't have more examples. I recently changed the number of build configurations (removing x86, adding Windows), so that is why you are probably seeing varying numbers of jobs.

vtjnash avatar Feb 24 '22 20:02 vtjnash

Got it. Yeah, I see the repo is stable and updates are infrequent. Well, we can keep this open, or you can re-open it if it happens again. Or email us at [email protected] and reference this issue. We'll try to find out what's happening.

afinetooth avatar Feb 24 '22 23:02 afinetooth

This new report seems even stranger: https://coveralls.io/builds/48419690. It reports an overall status of 0 out of 0 lines covered, despite each individual job covering 80% of the file. And for status results it posted these 2 badges (corresponding to the "total" and to the last run to finish, respectively).

image

vtjnash avatar Apr 20 '22 15:04 vtjnash

Hi, @vtjnash.

I re-ran the build: https://coveralls.io/builds/48419690

And see new results: Screen Shot 2022-04-20 at 10 45 18 AM

Screen Shot 2022-04-20 at 10 45 45 AM

Screen Shot 2022-04-20 at 10 45 35 AM

And in the PR: https://github.com/vtjnash/Pidfile.jl/pull/16

Screen Shot 2022-04-20 at 10 46 33 AM

Screen Shot 2022-04-20 at 10 46 54 AM

I can't speak to why the original build may have failed, but over the past 3-4 days we were affected by random, intermittent failing requests to the Github API. The usual effect was that PRs were not receiving status updates, which doesn't seem to be the situation in your case. But I do wonder if the underlying issue led to your build calculating coverage incorrectly / missing source files.

The problems mentioned above have resolved sometime in the last 24-hours, so please let me know if you continue to see any further issues like this and that may negate my theory.

afinetooth avatar Apr 20 '22 17:04 afinetooth