citgm-smoker fails when CITGM passes

[al-k21]

I am seeing citgm-smoker builds failing when module is marked is flaky.

For example:

Browserify is flaky on win32 - https://github.com/nodejs/citgm/blob/master/lib/lookup.json#L119

05:13:28 warn:                     | npm ERR! Test failed.  See above for more details.                                                                                     
05:13:28 info: done                | The smoke test has passed.
05:13:28 info: duration            | test duration: 234008ms
05:13:28 Build step 'Execute shell' marked build as failure
05:13:28 Recording test results
05:13:28 Notifying upstream projects of job completion
05:13:28 Finished: FAILURE

CITGM is passing while the build is marked as failed - citgm-smoker-nobuild#276 Is this intentional or does this requires investigation? If it is intentional, I don't really see the point of a flaky tag. Would it be more reasonable to have it yellow?

[MylesBorins]

Only citgm-all will respect the flaky tag rn afaik.

On Wed, Mar 7, 2018, 2:34 AM Oleksandr Kushchak [email protected] wrote:

I am seeing citgm-smoker https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker-nobuild builds failing when module is marked is flaky.

For example:

Browserify is flaky on win32 - https://github.com/nodejs/citgm/blob/master/lib/lookup.json#L119

05:13:28 warn: | npm ERR! Test failed. See above for more details. 05:13:28 info: done | The smoke test has passed. 05:13:28 info: duration | test duration: 234008ms 05:13:28 Build step 'Execute shell' marked build as failure 05:13:28 Recording test results 05:13:28 Notifying upstream projects of job completion 05:13:28 Finished: FAILURE

CITGM is passing while the build is marked as failed - citgm-smoker-nobuild#276 https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker-nobuild/MACHINE=win2016-vs2017/276/console Is this intentional or does this requires investigation? If it is intentional, I don't really see the point of a flaky tag. Would it be more reasonable to have it yellow?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/citgm/issues/554, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV4NUb367Gg9fk-fwqaVqcmYSD-cvks5tb7eugaJpZM4SgNu8 .

[gibfahn]

Only citgm-all will respect the flaky tag rn afaik.

Sounds like a feature-request 😁