cypress
cypress copied to clipboard
cypress-io
Capture + Display all logs for everything that happens in Cypress
Problem
Many of our users express the desire to see console.log statements when running headlessly. Typically you'd be able to see these when you're working in Dev Tools (in headed GUI mode).
We can pretty easily display console.log statements by mixing them into stdout when running headlessly, but we don't think this is really a "true" solution. It's kind of a hack and it still doesn't really tell you "everything" that's happening. For instance, when a console.log happens, how would you know when it happened in relation to everything else that's going on in a test?
So, we think to truly solve this problem, we need a comprehensive way of displaying not just console.log statements but also capturing:
- Network Traffic
- Commands
- Retries
- Under the hood Cypress wizardry
- DOM snapshots on failure
By providing all of this, the entire picture of what happened during a test is revealed and now you actually have even more information at your disposal than even what the GUI headed version provides you.
Solution
We're going to make these logs available both in our Dashboard, and in the Desktop App itself.
Once we capture and parse these logs, it will unlock an enormous amount of power in Cypress. We'll also be able to analyze the logs and deliver things like:
- Analytics into your tests
- Comparing a failing test to a previously passing one
- Diffing the logs for changes in response bodies, etc
- Understanding why tests are slow
- Discovering command anti patterns (like explicit waits in your code)
The logs could also be interactive - for instance they should "sync" up to the video and enable you to playback everything that actually happened in a Cypress test.
Here is an example comp of what we're going for.
We're still working on communicating the start and end of logs, and also communicating that a log event acts as "a stream" but that events are connected to other events.
For instance when a HTTP request starts - that's a 'start' event. When the response comes back, thats a separate 'end' event. However these are connected and you likely need to be able to view the data for both events from either the start or end event.
Another example:
A command starts. It retries 65x times. It eventually finishes. The duration of the retry events, the reason its retrying, and the eventual delta between a start and an end needs to be communicated intelligently.
These log files end up being a massive amount of data, but we already have the infrastructure in place to capture, parse and deliver them. At this point we're just iterating on a UI that's intuitive and doesn't overwhelm the user.
We're going to be providing a new Logs tab, but also for individual test failures, we'll just be providing the logs for that one independent test.
Feedback
...is welcome ;-)
@brian-mann excellent. Really looking forward to getting this. We're spending lots of time trying to debug why our compiled JS is not working when served static via Cypress. We cannot replicate anywhere else, so one peek at the logs would like solve everything!
@brian-mann I've been banging my head against a wall trying to upgrade to cypress 0.20.1. It's been days - things work fine in the GUI but fail from the console. And not being able to log anything is infuriating. After super old school commenting out code one line at a time until it passes again to isolate the line, I've found that it is cy.request that fails. But I don't know how to fix it yet, and I can't log anything so have no idea what the issue is. Adding all the features you noted in this ticket sounds fine, but we need a simple way to debug locally, in the command line itself. How can we get console.log (or cy.log) to show in stdout when running headlessly? We're blocked from using Cypress right now until we fix that because our CI is failing now due to this other error, which is supposedly fixed in 0.20 https://github.com/cypress-io/cypress/issues/517
@lukemadera have you tried using DEBUG variable to see messages from Cypress? https://docs.cypress.io/guides/guides/debugging.html#Debug-the-Command-Line
- Is it failing locally or only in CI?
- If failing locally be sure to run in
--headedmode so you can see what's going on. - You can also switch to using
--browser chromefrom the CLI (in local and in CI) if it fails only in electron. Electron is technically a different browser from your normal Chrome. - The key here is getting it to be reproduced locally. You can try running Chrome from the GUI, but using the Dev Tools to throttle back the network, and also decrease the CPU so that it mimics a much slower computer. That can sometimes highlight a race condition you are not accounting for.
- As per getting events out of the browser - that is not necessarily a straightforward thing. What exactly would you like us to send to
stdout? Internal information about what Cypress is doing? You will see hundreds if not thousands of events being fired over the course of a run. And that's just what Cypress is doing. Nothing from your application like network requests, console output, etc would be sent unless we chose to do that. - I would like to run an experiment here. Open up Cypress in GUI mode and then open the console and follow the instructions here: https://docs.cypress.io/api/events/catalog-of-events.html#Logging-All-Events.
That will send every internal event that Cypress fires to the console. Look at this and tell me if it is helpful. If the answer is yes, we could pretty easily get that out of the browser and into stdout. If it is not helpful then we'd need to figure out what is. Browsers are enormously complex and trying to distill what's happening down into text is not that simple. What exactly do you want to see? What would be helpful?
Also one last important piece of information. Why aren't you just recording your runs? Even though its headless you can see exactly what is going on since we capture a movie and screenshots. What about that information is not enough to see what's going on and be able to reproduce locally?
Or at the very least why aren't you watching the video from headless runs or looking at screenshots?
@bahmutov Yes I've tried debug. That gives general info, not info I generate and want to see.
@brian-mann thanks for the speedy response. There are 2 failures / issues right now. A. on CI, v0.19.x no longer runs (headless) B. locally, v0.20.x runs in GUI but fails in headless.
-
All the tests pass in the GUI so unless I'm missing something, the error won't happen in headed mode, so I can only debug in headless mode.
-
Again, it's a general request issue (I think - still need logging), not a browser issue.
-
I can reproduce it locally 100% of the time, in headless mode. Works 100% of the time in GUI mode. These are tests that worked fine for months, and they work in v0.19. Just not when upgrading to v0.20.x
-
I'm simply looking to be able to do
console.log()and have it show up on the command line when I do./node_modules/.bin/cypress run --spec cypress/integration/locations/locations-list.js -
while all that data MAY be helpful, right now I just need ONE (or a few) pieces of data, that I choose from
console.log() -
I'm not even getting to the screenshots or videos really, and I've already figured out the problem (it's in our data setup phase, trying to login via cy.request). It's not an actual test that is failing, we're not even making it to the tests. So I'm not sure how videos / screenshots would help. It's literally the first step that's failing, in setup, before tests even start.
If this is still unclear I can record a video of the test in the GUI and the test in the command line - one finishes and passes, one just gets stuck and never does anything (or times out).
Update: here's the video: http://recordit.co/DdfW8hgK74
Okay let me clear up a few things.
When we say GUI mode we're talking about launching cypress from cypress open. In that mode the tests never "complete" and you don't get anything on stdout.
When you run from the CLI via cypress run then the tests complete, you get stdout and an exit code.
Cypress itself does work differently in these two modes. When running from the CLI it makes internal optimizations to be able to run all the tests - ie it will run faster than the GUI mode. The GUI mode is focused on debuggability and interaction.
My point is this: using --headed from the CLI simply shows you the browser. It will behave identically to running headless. One shows the Electron browser window, and the other doesn't. That's it.
In fact you could just launch Chrome via the CLI with --browser chrome and then you'll be using the same two browsers in GUI mode and in CLI mode with the only difference that one runs to completion, the other doesn't.
The screenshots and video would still help in this situation, albeit I understand the need for console.log. At the very least they would show you precisely which step and command is failing without you having to guess in the dark.
Update: per your video - just add the --headed option or use --browser chrome to see what's going on. If you choose not to do that, you could still use the video Cypress captures when you run headlessly so you could still see what happened in that mode.
Also at this very moment you could do this trick here: https://github.com/cypress-io/cypress/issues/300#issuecomment-321587149
That will get all console.log's to show up in the command log, which will stringify and print their arguments. You could then watch the video / screenshot to see what the information is.
That's what I would do today when running in CI. Of course locally just show the browser and iterate on it there.
I opened a couple new issues to do what you're asking with exporting the console output, and some improvements to the debugging the driver.
YES! The --headed works and I can see console.log output, thanks so much @brian-mann ! It does still close automatically at the end, which means I can't see the logs persisted, so I have to add a wait at the end. Any way to prevent the browser from auto-closing?
However, I'm still stuck at the same cy.request line. It just doesn't ever complete. In digging into it, cy.request works fine if it's called BEFORE cy.readFile. If it's used after (within the .then()), it just never returns.
In trying to reproduce it, if I simplify things it seems to work, so I'm not clear on what the issue is.
If I just do a request (using the npm request module instead of cy.request, I get a CORS error).`
Luckily, the request I'm attempting is to logout so in this case I can just skip that request all together, and then things work fine. Now tests appear to be passing again.. Locally at least. Going to try in CI now.
Not sure if the issue is "fixed" but it's working for me now it seems. Thanks for all the speedy responses and help!
If you want to iterate without it close, just open up Electron in the GUI. Just select that browser instead of Chrome. You may see the same issue.
As I mentioned if that's the case just switch to booting chrome from the CLI --browser chrome
This would be a really nice addition. Right now my tests are failing on the CI and it looks like it has something to do with cookies but because Cypress is basically a black box right now on CI I have no way of figuring out what's going on. Hope this feature will arrive soon, keep up the good work 👏🏼
Related to #699 and #700.
Those will land pretty soon and should help alleviate these problems.
Just want to throw my 2 cents in, I'm also having an issue with (what I can only imagine is cookie-related) the CI build vs local cypress open run that @rovansteen is having, and not having a way to see what network requests are made is making it challenging to debug.
So far I'm loving Cypress though, you folks are doing amazing work and I'm more than willing to fork over our company credit card for a paid plan in the future!
Hi, loving Cypress. But I am curious. Are these issues still being addressed?
I see a lot of comments that these issues are being worked on, like: "x and y. Those will land pretty soon". But this seems to have been in development for nearly a year now.
Respectfully, if we can't debug our CI than this isn't really a viable product for the long term. And we would rather know sooner than later since E2E tests are very costly.
@brian-mann do you have any update on this one? In October you said soon. I know your backlog is very long but CI logs are quite important. Every time there is an issue on the CI we need to run the tests again locally in order to get the logs. Thank you for your understanding 🙏
I'm wondering if anything is happening here, it's way too quite for a feature all projects need at the point where they have to maintain tests and the CI console output and the test results can't tell what went wrong except the last exception message.
Sounds like a hack but if you
- start an HTTP server in the background with a log endpoint (the endpoints simply writes everything on its console)
- patch the cy.log() command to make a request to your endpoint with the arguments it got
You should at least see the cy.log messages. It would be much more elegant if it goes through the existing cypress internal websocket though.
Anyone here tried something like this before?
For everyone who wants to get the cy.log and other Cypress commands this might be helpful https://github.com/bahmutov/cypress-failed-log
First, thanks for all the good work!
But I would address this particularly issue again.
To give a daily example. We couldnt reproduce a 'bug' in our application in local development and our CI server continuously failed. We were not able to find why, because we were not able to inspect the console errors (with an error when a network request failed). After a lot of research we discovered that our CI environment was not able to connect to a specific endpoint.
So you cannot guarantee that cypress test running locally will behave exact the same as on your CI.
It was a little bit frustrating, because in the summary of this ticket there was a line
We can pretty easily display console.log statements by mixing them into stdout when running headlessly
and that simple functionality would helped us so much.
As kind of a hack we now have a feature flag when running E2E on the CI that when having a console error, the page will output the error.
/* This part is to put console errors in the HTML when running cypress tests */
var console = (function(oldCons){
return {
log: function(text){
oldCons.log(text);
},
info: function (text) {
oldCons.info(text);
},
warn: function (text) {
// $FlowFixMe
document.getElementById('root').innerHTML = text
oldCons.warn(text);
},
error: function (text) {
// $FlowFixMe
document.getElementById('root').innerHTML = text
oldCons.error(text);
}
};
}(window.console));
if (window.envConfig.cypressConsole == 'true') {
window.console = console;
}
Since 3.0, we're collecting more data related to the timings, body, and hooks run for recorded tests.
This issue has been put on hold as there was a great deal of work to be done in the server to gather all of the data necessary. It is still on our roadmap for the future - and there has been some work done on it.
Yeah, if it's so easy to mix cy.log into the console output, then give us a CLI config option for outputting that to the console. Don't just leave us helpless for more than a year working on the perfect solution.
We just had this issue, where tests pass locally, but fail on CI. We have no way of knowing why they fail as we just get a White Screen of Death and no console.log output. I'm at a loss for any way to know what error is being thrown as I can't even reproduce it locally. Any other tips for jury-rigging console logging on Travis?
