Very slow tests when the browser is minimized

[sqal]

Hi. I hope this is a good place to report an issue that I am having. Ok so as I said in the title, I have a problem with very slow tests when chrome browser is minimized or have open Karma DEBUG RUNNER tab. Take a look at these times:

minimized browser

Chrome 54.0.2840 (Windows 10 0.0.0): Executed 19 of 19 SUCCESS (8.607 secs / 1.204 secs)
TOTAL: 19 SUCCESS

open browser

Chrome 54.0.2840 (Windows 10 0.0.0): Executed 19 of 19 SUCCESS (0.406 secs / 0.272 secs)
TOTAL: 19 SUCCESS

The difference is pretty huge. I would like to know why is this happening and if I can fix this. Is this some bug or have i set something wrong in my karma config? (although i don't know what it would be). But there is my config:

import path from 'path';
import Config from 'webpack-config';
import yargs from 'yargs';

const { watch } = yargs.argv;
const projectRoot = path.resolve(__dirname, '../../');

const webpackConfig = new Config().extend({
  'config/webpack.config.base': (config) => {
    // no need for app entry during tests
    delete config.entry;

    // make sure isparta loader is applied before eslint
    config.module.preLoaders = config.module.preLoaders || [];
    config.module.preLoaders.unshift({
      test: /\.js$/,
      loader: 'isparta',
      include: path.resolve(projectRoot, 'src'),
    });

    // only apply babel for test files when using isparta
    config.module.loaders.some((loader) => {
      if (loader.loader === 'babel') {
        loader.include = path.resolve(projectRoot, 'test/unit');
        return true;
      }

      return false;
    });

    return config;
  },
}).merge({
  resolve: {
    alias: {
      vue: 'vue/dist/vue.js',
      src: path.join(projectRoot, 'src'),
    },
  },
  devtool: 'cheap-module-source-map',
  vue: {
    loaders: {
      js: 'isparta',
    },
  },
});

const reporters = [
  { type: 'text-summary' },
];

if (!watch) {
  reporters.unshift({ type: 'lcov', subdir: '.' });
}

module.exports = function setConfig(config) {
  config.set({
    browsers: ['Chrome'], // 'PhantomJS'
    frameworks: ['mocha', 'sinon-chai'],
    reporters: ['spec', 'coverage'],
    files: [
      './index.js',
    ],
    preprocessors: {
      './index.js': ['webpack', 'sourcemap'],
    },
    webpack: webpackConfig,
    webpackMiddleware: {
      noInfo: true,
    },
    coverageReporter: {
      dir: './coverage',
      reporters,
    },
    singleRun: !watch,
    captureTimeout: 10000,
  });
};

// package.json

  "devDependencies": {
    "autoprefixer": "^6.5.3",
    "babel-core": "^6.18.2",
    "babel-loader": "^6.2.7",
    "babel-plugin-transform-class-properties": "^6.19.0",
    "babel-plugin-transform-flow-strip-types": "^6.18.0",
    "babel-plugin-transform-object-rest-spread": "^6.19.0",
    "babel-plugin-transform-runtime": "^6.15.0",
    "babel-preset-latest": "^6.16.0",
    "babel-preset-node6": "^11.0.0",
    "babel-register": "^6.18.0",
    "browser-sync": "^2.18.1",
    "browser-sync-spa": "^1.0.3",
    "chai": "^4.0.0-canary.1",
    "cross-env": "^3.1.3",
    "css-loader": "^0.25.0",
    "css-mqpacker": "^5.0.1",
    "cssnano": "^3.8.0",
    "del": "^2.2.2",
    "eslint": "3.10.2",
    "eslint-config-airbnb-base": "^10.0.1",
    "eslint-import-resolver-webpack": "^0.7.0",
    "eslint-plugin-flowtype": "^2.25.0",
    "eslint-plugin-html": "^1.6.0",
    "eslint-plugin-import": "^2.2.0",
    "file-loader": "^0.9.0",
    "flow-bin": "^0.35.0",
    "function-bind": "^1.1.0",
    "gulp": "github:gulpjs/gulp#4.0",
    "gulp-postcss": "^6.2.0",
    "gulp-sass": "^2.3.2",
    "gulp-sourcemaps": "^2.2.0",
    "html-webpack-plugin": "^2.24.1",
    "isparta-loader": "^2.0.0",
    "json-loader": "^0.5.4",
    "karma": "^1.3.0",
    "karma-chrome-launcher": "^2.0.0",
    "karma-coverage": "^1.1.1",
    "karma-mocha": "^1.3.0",
    "karma-phantomjs-launcher": "^1.0.2",
    "karma-sinon-chai": "^1.2.4",
    "karma-sourcemap-loader": "^0.3.7",
    "karma-spec-reporter": "0.0.26",
    "karma-webpack-with-fast-source-maps": "^1.10.0",
    "mocha": "^3.1.2",
    "phantomjs-prebuilt": "^2.1.13",
    "postcss-loader": "^1.1.1",
    "require-dir": "^0.3.1",
    "sass-loader": "^4.0.2",
    "sinon": "^2.0.0-pre.4",
    "sinon-chai": "^2.8.0",
    "style-loader": "^0.13.1",
    "svg-sprite-loader": "0.0.31",
    "svgo-loader": "^1.1.2",
    "url-loader": "^0.5.7",
    "vue-loader": "^9.9.5",
    "vue-style-loader": "^1.0.0",
    "webpack": "2.1.0-beta.22",
    "webpack-config": "^6.2.1",
    "webpack-dev-middleware": "^1.8.4",
    "webpack-hot-middleware": "^2.13.2",
    "yargs": "^6.4.0"
  }

[delasteve]

Originally thought it was due to the test runner I was using, but after further investigation, it's definitely looking like a karma / Chrome issue. I haven't tested other browsers to see if this appears with them.

With browser window in foreground:

Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 900 of 900 SUCCESS (15.354 secs / 15.257 secs)

With browser in background:

Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 1 of 1 SUCCESS (1.713 secs / 0.278 secs)
Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 2 of 2 SUCCESS (3.374 secs / 1.885 secs)
Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 3 of 3 SUCCESS (3.143 secs / 3.137 secs)
Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 4 of 4 SUCCESS (3.653 secs / 3.649 secs)
Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 5 of 5 SUCCESS (5.01 secs / 5.004 secs)

...

Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 90 of 90 SUCCESS (1 min 35.316 secs / 1 min 34.14 secs)

As you can see, it increases ~1sec per test in watch mode with the browser in the background.

To test, I used the following (a generated test from the Angular CLI project, looped):

for (let i = 0; i < 900; i++) {
  it(`should create the app${i}`, async(() => {
    const fixture = TestBed.createComponent(AppComponent);
    const app = fixture.debugElement.componentInstance;
    expect(app).toBeTruthy();
  }));
}

It should be noted I am using zone.js's patch for jasmine and async from @angular/core/testing. That said, I ran a few more tests with and without async. Here are the results:

Foreground

• With async

  Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 900 of 900 SUCCESS (8.084 secs / 7.947 secs)
  

• Without async

  Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 900 of 900 SUCCESS (6.681 secs / 6.568 secs)
  

Background

• With async (had to cancel)

  Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 145 of 900 SUCCESS (0 secs / 2 mins 31.986 secs)
  

• Without async

  Chrome 55.0.2883 (Mac OS X 10.12.0): Executed 900 of 900 SUCCESS (46.051 secs / 44.849 secs)
  

Tests used:

With async

for (let i = 0; i < 90; i++) {
  it(`test: ${i}`, async(() => {
    expect(true).toEqual(true);
  }));
}

Without async

for (let i = 0; i < 90; i++) {
  it(`test: ${i}`, () => {
    expect(true).toEqual(true);
  });
}

// related issue link https://github.com/angular/angular-cli/issues/4071

[hansl]

Chrome started throttling background tabs/browsers more aggressively. See https://www.chromestatus.com/feature/6172836527865856

[rogeriopvl]

@hansl the aggressive throttling is only enabled by default on Chrome 57. The versions in question are 54 and 55.

Still, I'm pretty sure this is not directly related with karma-chrome-launcher, but I'll investigate a bit further to confirm this.

[gclayburg]

I ran into this issue with Chrome 57.0.2987.

For me my karma tests would finish almost immediately, unless I tried to run them using the DEBUG tab in chrome. The same tests took almost a minute to complete. not good.

karma-chrome-launcher was not the problem, but it did offer a solution for me. I added this to my karma.conf.js:

        customLaunchers: {
            Chrome_without_background_throttle: {
                base: 'Chrome',
                flags: ['--disable-background-timer-throttling', '--disable-renderer-backgrounding'],
            }
        },

and then started karma like this:

$ karma start src/test/javascript/karma.conf.js --debug --browsers=Chrome_without_background_throttle --autoWatch=true

At least for me, I needed both of these chrome flags to prevent Chrome from throttling the background tab.