public
public copied to clipboard
wallabyjs
Command line runner, CI server support
I'd like to execute my tests from the command line. Didnt find any way to do this with wallaby, which is quite weird. Are you planning to support only IDEs/editors?
Wallaby.js main idea is to integrate with editors, run tests for the code that you change and display the results in the editor. Providing a command line runner is relatively easy, but why would you want to use it instead of a grunt/gulp task for example?
Thanks for the quick response Artem.
Actually I care about executing tests in the terminal, so gulp/grunt would work for me as well. While I think that IDE/editors support is great, it creates a dependency on them (and different people use different editors). Also it can't be automated on a CI which expects a json output of the tests results for example.
I know I cant set it up manually (which I usually do), but wallaby seemed to offer all the features out of the box (mocha, babel and webpack in my case). I just miss the most basic one ;)
I have to admit it's a bit unexpected feature request for a commercial tool like wallaby, as one of its strong (and competitive) parts is tight integration with editors and running tests incrementally. There are many free and OSS alternatives to just run all tests from CLI or on CI server. In any case, I'm open for all ideas that are useful for wallaby user, adding "enhancement" label.
You are focusing so much on the IDE awesomeness that you sometimes forget the Wallaby is also a lot faster than i.e. Karma, thanks to web workers. So in a CI environment Wallaby in command line mode would be a good friend too ;).
Just so I am understanding the documentation, the only way to run Wallaby.js is through an IDE/editor of some sort?
If you're using emacs, Vim, or anything not support today, you are out of luck?
Thanks
I also would like to run the tests with Wallaby on the command line for Continuous Integration. This would save me the configuration work for other tools like Karma and Jasmine CLI runner.
:+1:
I'm investigating Wallaby as a possible alternative test runner at work and was very surprised to see there is no way to run it on the command line! It was the first thing I looked for.
It seems that your have to use a different test runner for the command line, with its own version of the test configuration that must be maintained and kept in sync with the Wallaby config (with the risk that tests pass locally but not on CI due to the config getting out of sync or due to differences in how the test runners work).
Is this the official way we are expected to use Wallaby, or have a missed something!
@karl Correct, for now wallaby.js is not a CI tool. Its focus is to help you to be more productive when writing your code (by running tests, but the emphasis is on the development time experience).
It can't guarantee that your tests will pass on CI server, in all browsers, across all devices that your web app intends to support, etc. We'll get there sooner or later, but for now if you need to run your tests on CI server, in different browsers or on different devices, I'd recommend to use another solutions. BTW, with the testRunerConfig module you may re-use a big part of your config for different test runners.
@ArtemGovorov I would just like to let you know that configuring wallaby worked great while setting up karma for our scenario involved a good deal of frustration. I would love to be able to use your tool on our CI server provided you support it from a technical and licensing point of view.
I have to throw in my support for a Wallaby CLI as well. I've been struggling more and more with Karma lately, as I write more and more ES6 and TS code, using more and more modern features, frameworks, etc. Karma is falling behind, not keeping up with the times, and has really just become much more of a hinderance than a compliment to my daily routines.
I am really impressed with what Wallaby offers, the IDE integration and real time, continuous instrumentation of my code is pretty nice. However, at the end of the day...it all has to run through CI. That, at the moment, means extra work to keep Karma working, which is a lot harder than keeping Wallaby working.
Having a Wallaby CLI would definitely reduce the daily overhead of keeping CI rolling.
+1
For us we just find it a pig to get tests running consistently. We have tests working in Karma that don't in Wallaby, and vice versa. The more time we have to spend making 'good tests' run in another environment the more time we've wasted fixing actual issues or writing more tests.
@IPWright83 Not related to the feature request, but if you create a sample of tests working in Karma that don't in Wallaby, I'm happy to have a look and find out why it's happening (perhaps different test runner is used, in which case you may try Electron or the same PhantomJs version as used in Karma).
At the moment it's worse the other way around. Tests failing in wallaby are due to es6 (Symbol not being defined) and I guess a bad configuration.
On 12 Sep 2016 14:08, "Artem Govorov" [email protected] wrote:
@IPWright83 https://github.com/IPWright83 Not related to the feature request, but if you create a sample of tests working in Karma that don't in Wallaby, I'm happy to have a look and find out why it's happening (perhaps different test runner is used, in which case you may try Electron https://wallabyjs.com/docs/integration/electron.html or the same PhantomJs version https://wallabyjs.com/docs/integration/phantomjs2.html as used in Karma).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallabyjs/public/issues/72#issuecomment-246341492, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6N_9nQ2M1Sdbtku1uPAYuAcpLWslKks5qpU7GgaJpZM4EBv_x .
@IPWright83
Symbol not being defined
It definitely sounds like the default runner (PhantomJs 1.9.8) limitation. Try using the Electron runner, it's similar as running your code in the latest Chrome. Another solution is to use some ES6 polyfill, for example npm i babel-polyfill plus add it to the files list.
While people are waiting for this, you might be interested in this setup I have going where I run Wallaby.js locally and Gulp + Karma + Mocha on CI.
https://github.com/xolvio/qualityfaster/tree/dev
Hope that helps someone
I can understand the desire to maintain a narrow focus on your goals of seamless, high-quality editor support and speed. However, by ignoring CI, a huge portion of your market is going to be forced to write the same complex, convoluted build and test scripts they are desperately trying to avoid by switching to Wallaby, because many companies have CI requirements for tests and coverage as a part of their deployment pipeline. And frankly, these requirements make sense to me, since they are globally visible.
Additionally, being able to use Wallaby as a simple, headless runner for free would be a very useful feature. I see this being beneficial in two scenarios:
- Companies who use contractors, who are required to write unit tests, but who cannot really be forced to cough up the money for all editor licenses.
- Open source developers who would dramatically increase the uptake of this tool. Capturing more market share could really drive demand within the orgs that these developers belong to, and I don't think it would take away from the excellent benefits of having tight IDE integrations. Think of this like JetBrains Community Edition products.
Just a thought, on the last points. But with regard to lack of CI: For our company this makes the proposition of buying a bunch of licenses much harder to justify. I'll be evaluating this tool, and it's going to be hard to recommend without this feature.
@bfricka Thanks for the feedback. We are planning to support the CI scenario, it's our next major goal right after we provide/improve test debugging experience.
Given that CI support is happening in wallaby sooner or later, it's worth mentioning that switching to wallaby for CI builds will obviously be much easier and straightforward for those companies that are already using wallaby and have it set up and running on dev machines.
+1.
We also want to be able to use wallabyjs as a command line runner in our CI environment
+1. And option to generate lcov file. But ... It might be more problematic than a simple test runner
I bought wallabyjs about 6-8 months ago because I loved it. Work on the other hand turned me down because of this exact issue. This issue would sell licenses. Is there any way I can contribute? I would really like to see this.
