happypack
happypack copied to clipboard
Published
20 hours ago •
amireh
amireh
Happypack worker processes use very little CPU; small improvement to build speed
Node 8.4.0 (on macOS) Webpack 3.5.5 Happypack 4.0.0-beta.5 2,302 modules
Build time without Happypack: 51s Build time with Happypack: 41s
Looking at the macOS Activity Monitor during my build, the spawned workers use very little CPU. Either that means my loaders are I/O bound, or there's something not working quite right.
Is this how Happypack normally runs, or is there something unusual here?
Happypack debug log: https://gist.github.com/sentience/20f06d0eea5158f61b0d1811b2b7f434
I know your first question is going to be can I share a repo that reproduces the issue, but I'm afraid I can't share it publicly. If you really want to get a hold of our codebase to investigate, let me know and we can look at getting you temporary access under NDA.
Hmmm. Thank you for the detailed report! That doesn't look right to me either. I can test on a macbook and hopefully I'll see something similar (but most of my coworkers use macs and it works as expected in the office.)
Can you tell me which loaders are being applied to the source files? Or if possible share webpack's config?
@amireh
Same situation for me on Windows. I have a huge project, using Happypack 4.0.1 for babel-loader and eslint-loader. 4 threads, shared pool. Here's the biggest thing: caching is enabled for both babel and eslint.
With caching is enabled, only the first node thread actually works. It's at 35-40% CPU usage, other 3 threads are at 3%. Starting webpack takes 50-60 seconds.
With caching disabled for babel or eslint, most of the threads start to really work. CPU usage distributes something like 25%-20%-18%-20%, overall CPU usage can go up to 95% which is good. But starting webpack now takes 1m 20s if I disable caching for eslint, and 1m 30s if I disable caching for babel. 2 minutes if caching is disabled for both.
So either Happypack doesn't work when caching is enabled at all, or simply doesn't offer the functionality of helping those modules resolve their cache. And I don't really think I'll have a reason to use Happypack if it's slower than simply setting 'cache: true' :( It's sad because I really like the idea of Happypack
I'd take this with a grain of salt, but perhaps what's going on when caching is enabled is that we're becoming I/O bound instead of CPU bound so only one thread ends up doing the work. 1 or 4 or 100 threads doing I/O would effectively have the same throughput since they all end up delegating to the kernel/OS.
But again, this is just a wild guess until I have time to actually look into this.
Is this demonstrable on a smaller scale? Meaning, is babel-loader always using 1 CPU when caching is enabled for, say, 20-30 files? We also need to see if this happens on other platforms like Linux.
It would be tremendously helpful to get some repro steps.
I don't think it's limited by I/O. I'm running it on an SSD and in Task Manager disk usage is never above 10%.
I don't really have a fully set up project with 20-30 files here, so I can't test it. Steps to reproduce are pretty simple: have a project with webpack, babel-loader and happypack. Start it with options: { cacheDirectory: true } and with options: { cacheDirectory: false }, check CPU/IO usage.
Also for me...
HDD is always at 0%. Twelve core CPU has only one core loaded. Lot's of node.exe spawned. I am testing this on SPIN Dashboard which is more or less big project with ton's of react components. It builds around 50-60 seconds on my desktop no matter with or without the happypack.
Apologies for my "dirty" code below - it was prototyping in hurry...
{
"name": "spin-react",
"version": "1.0.0",
"description": "SPIN Dashboard rewritten using React",
"main": "index.js",
"engines": {
"node": ">=4.2.0",
"npm": "^3.0.0"
},
"scripts": {
"clean": "rimraf dist",
"compile": "better-npm-run compile",
"lint": "eslint src tests server",
"lint:fix": "npm run lint -- --fix",
"start": "better-npm-run start",
"dev": "better-npm-run dev",
"dev:no-debug": "npm run dev -- --no_debug",
"test": "better-npm-run test",
"test:dev": "npm run test -- --watch",
"deploy": "better-npm-run deploy",
"deploy:dev": "better-npm-run deploy:dev",
"deploy:prod": "better-npm-run deploy:prod",
"codecov": "cat coverage/*/lcov.info | codecov"
},
"betterScripts": {
"compile": {
"command": "babel-node --extensions \".ts,.js,.tsx,.jsx\" bin/compile",
"env": {
"DEBUG": "app:*"
}
},
"dev": {
"command": "nodemon --exec babel-node bin/server",
"env": {
"NODE_ENV": "development",
"DEBUG": "app:*"
}
},
"deploy": {
"command": "npm run clean && npm run compile",
"env": {
"DEBUG": "app:*"
}
},
"deploy:dev": {
"command": "npm run deploy",
"env": {
"NODE_ENV": "development",
"DEBUG": "app:*"
}
},
"deploy:prod": {
"command": "npm run deploy",
"env": {
"NODE_ENV": "production",
"DEBUG": "app:*"
}
},
"start": {
"command": "babel-node --extensions \".ts,.js,.tsx,.jsx\" bin/server.ts",
"env": {
"DEBUG": "app:*"
}
},
"test": {
"command": "babel-node ./node_modules/karma/bin/karma start build/karma.conf",
"env": {
"NODE_ENV": "test",
"DEBUG": "app:*"
}
}
},
"author": "Webkom.co",
"license": "",
"dependencies": {
"@owczar/spin-template": "^1.5.2",
"assign-deep": "^0.4.5",
"autocast": "0.0.4",
"better-npm-run": "0.0.8",
"bootstrap-sass": "^3.3.7",
"chartist": "^0.9.7",
"css-loader": "^0.23.0",
"cssnano": "^3.3.2",
"debug": "^2.2.0",
"deep-assign": "^2.0.0",
"draft-js": "^0.10.1",
"extract-text-webpack-plugin": "^1.0.0",
"faker": "^3.1.0",
"file-loader": "^0.8.4",
"font-awesome": "^4.6.3",
"fs-extra": "^0.30.0",
"highcharts": "^4.2.6",
"highcharts-more": "^0.1.2",
"history": "^2.0.0",
"html-webpack-plugin": "^2.7.1",
"imports-loader": "^0.6.5",
"ip": "^1.1.2",
"ismobilejs": "^0.4.0",
"isomorphic-fetch": "^2.2.1",
"json-loader": "^0.5.7",
"koa": "^2.0.0-alpha.3",
"koa-connect-history-api-fallback": "^0.3.0",
"koa-convert": "^1.2.0",
"koa-proxy": "^0.6.0",
"koa-static": "^2.0.0",
"koa2-cors": "^2.0.3",
"left-pad": "^1.1.0",
"moment": "^2.18.1",
"node-sass": "^4.7.2",
"node-uuid": "^1.4.7",
"normalize.css": "^4.1.1",
"numeral": "^1.5.3",
"object-hash": "^1.1.4",
"pace": "github:HubSpot/pace#v1.0.2",
"peity": "^3.2.0",
"perfect-scrollbar": "^0.6.13",
"rc-slider": "^6.1.2",
"react": "^15.0.0",
"react-bootstrap": "^0.30.10",
"react-bootstrap-table": "^3.4.6",
"react-bootstrap-typeahead": "^1.4.0",
"react-chartist": "^0.10.2",
"react-color": "2.13.4",
"react-dates": "^12.3.0",
"react-dnd": "^2.4.0",
"react-dnd-html5-backend": "^2.4.1",
"react-dom": "^15.0.0",
"react-draft-wysiwyg": "^1.10.7",
"react-dropzone": "^3.13.3",
"react-dual-listbox": "^1.1.0",
"react-grid-layout": "^0.13.9",
"react-highcharts": "^8.3.3",
"react-highlight-words": "^0.6.0",
"react-image-cropper": "^1.1.0",
"react-image-holder": "^2.0.1",
"react-interval": "^1.3.1",
"react-moment-proptypes": "^1.3.0",
"react-nestable": "^1.0.15",
"react-notification-system": "^0.2.11",
"react-notification-system-redux": "^1.1.4",
"react-overlays": "^0.6.3",
"react-redux": "^4.0.0",
"react-router": "^2.2.0",
"react-router-bootstrap": "^0.23.0",
"react-router-redux": "^4.0.0",
"react-select": "^1.0.0-rc.3",
"react-sparklines": "^1.5.2",
"react-text-mask": "^2.0.0",
"react-toastr": "^2.8.2",
"react-toggle": "^3.0.1",
"react-treebeard": "^1.1.4",
"reactcss": "^1.2.2",
"recharts": "^0.21.2",
"redux": "^3.0.0",
"redux-thunk": "^2.0.0",
"rimraf": "^2.5.1",
"sass-loader": "^6.0.6",
"string-replace-loader": "^1.0.2",
"style-loader": "^0.13.0",
"text-mask-addons": "^3.3.0",
"tinycolor2": "^1.4.1",
"truncate": "^2.0.0",
"underscore": "^1.8.3",
"url-loader": "^0.5.6",
"velocity-animate": "^1.2.3",
"webpack": "^3.11.0",
"yargs": "^4.0.0"
},
"devDependencies": {
"@babel/core": "^7.0.0-beta.40",
"@babel/node": "^7.0.0-beta.40",
"@babel/plugin-transform-runtime": "^7.0.0-beta.40",
"@babel/polyfill": "^7.0.0-beta.40",
"@babel/preset-env": "^7.0.0-beta.40",
"@babel/preset-react": "^7.0.0-beta.40",
"@babel/preset-stage-3": "^7.0.0-beta.40",
"@babel/preset-typescript": "^7.0.0-beta.40",
"@babel/register": "^7.0.0-beta.40",
"@babel/runtime": "^7.0.0-beta.40",
"@types/debug": "^0.0.30",
"@types/html-webpack-plugin": "^2.30.2",
"@types/ip": "^0.0.30",
"@types/koa": "^2.0.44",
"@types/koa-static": "^4.0.0",
"@types/koa2-cors": "^2.0.1",
"@types/webpack": "^3.8.8",
"@types/yargs": "^11.0.0",
"babel-loader": "8.0.0-beta.0",
"cache-loader": "^1.2.0",
"chai": "^3.4.1",
"chai-as-promised": "^5.1.0",
"chai-enzyme": "^0.4.0",
"cheerio": "^0.20.0",
"codecov": "^1.0.1",
"enzyme": "^2.0.0",
"eslint": "^2.4.0",
"eslint-config-standard": "^5.1.0",
"eslint-config-standard-react": "^2.2.0",
"eslint-plugin-promise": "^1.0.8",
"eslint-plugin-react": "^5.0.0",
"eslint-plugin-standard": "^1.3.1",
"happypack": "^4.0.1",
"karma": "^0.13.21",
"karma-coverage": "^1.0.0",
"karma-mocha": "^1.0.1",
"karma-mocha-reporter": "^2.0.0",
"karma-phantomjs-launcher": "^1.0.0",
"karma-webpack-with-fast-source-maps": "^1.9.2",
"mocha": "^2.2.5",
"nodemon": "^1.8.1",
"phantomjs-prebuilt": "^2.1.3",
"postcss": "^6.0.19",
"postcss-loader": "^2.1.0",
"react-addons-test-utils": "^15.0.0",
"redbox-react": "^1.2.2",
"sinon": "^1.17.3",
"sinon-chai": "^2.12.0",
"stats-webpack-plugin": "^0.6.2",
"string-replace-loader": "^1.0.2",
"thread-loader": "^1.1.4",
"typescript": "^2.7.2",
"webpack-dev-middleware": "^2.0.5",
"webpack-hot-middleware": "^2.21.0"
}
}
import webpack from "webpack";
//import cssnano from "cssnano";
import HtmlWebpackPlugin from "html-webpack-plugin";
//import ExtractTextPlugin from "extract-text-webpack-plugin"
import path from "path";
// import StatsPlugin from "stats-webpack-plugin";
import HappyPack from "happypack";
import { config } from "./../config/config";
import _debug from "debug";
const happyThreadPool__ = HappyPack.ThreadPool ({ size: 12 });
const debug = _debug ("app:webpack:config")
const paths = config.utils_paths
const {__DEV__, __PROD__, __TEST__} = config.globals;
debug ("Create configuration.")
const webpackConfig: webpack.Configuration = {
name: "client",
target: "web",
devtool: config.compiler_devtool,
resolve: {
extensions: [".js", ".jsx", ".json"],
modules: [
paths.client (),
path.resolve ("./src"),
"node_modules"
]
},
profile: true
}
webpackConfig.module = {} as webpack.NewModule;
//webpackConfig.module.rules = [] as webpack.OneOfRule [];
// ------------------------------------
// Entry Points
// ------------------------------------
const APP_ENTRY_PATHS = [
"@babel/polyfill",
paths.client("main.js")
];
webpackConfig.entry = {
init: [paths.client("init.js")],
template: paths.client("template.js"),
app: __DEV__
? APP_ENTRY_PATHS.concat (`webpack-hot-middleware/client?path=${config.compiler_public_path}__webpack_hmr`)
: APP_ENTRY_PATHS,
vendor: config.compiler_vendor
};
// ------------------------------------
// Bundle Output
// ------------------------------------
webpackConfig.output = {
filename: `[name].[${config.compiler_hash_type}].js`,
path: paths.dist (),
publicPath: config.compiler_public_path
}
// ------------------------------------
// Plugins
// ------------------------------------
webpackConfig.plugins = [
new webpack.optimize.ModuleConcatenationPlugin(),
new webpack.DefinePlugin (config.globals),
new HtmlWebpackPlugin ({
template: paths.client("index.html"),
hash: false,
favicon: paths.client("static/favicon.png"),
filename: "index.html",
inject: "body",
minify: {
collapseWhitespace: true
}
}),
new HappyPack ({
id: "javascript",
threads: 8,
threadPool: happyThreadPool__,
loaders: [{
loader: "babel-loader",
options: {
cacheDirectory: true,
plugins: ["@babel/plugin-transform-runtime"],
presets: [
[
"@babel/env",
{
"targets": {
"browsers": [ "last 2 versions" ]
}
}
],
"@babel/stage-3",
"@babel/react",
"@babel/preset-typescript"
],
env: {
production: {
//presets: ["react-optimize"] //Fails for react-bootstrap sub components
}
}
}
}]
})/*,
new HappyPack ({
id: "styles",
threads: 4,
threadPool: happyThreadPool__,
loaders: [
]
})
*/
/*
,
new StatsPlugin ("stats.json", {
chunkModules: true//,
// exclude: [/node_modules[\\\/]react/]
})
*/
]
if (__DEV__) {
debug("Enable plugins for live development (HMR, NoErrors).")
webpackConfig.plugins.push(
new webpack.HotModuleReplacementPlugin(),
new webpack.NoEmitOnErrorsPlugin ()
)
} else if (__PROD__) {
debug("Enable plugins for production (OccurenceOrder, Dedupe & UglifyJS).")
webpackConfig.plugins.push(
new webpack.optimize.OccurrenceOrderPlugin (true),
// new webpack.optimize.DedupePlugin(),
new webpack.optimize.UglifyJsPlugin({
compress: {
unused: true,
dead_code: true,
warnings: false
}
})
)
}
// Don"t split bundles during testing, since we only want import one bundle
if (!__TEST__) {
webpackConfig.plugins.push(
new webpack.optimize.CommonsChunkPlugin({
names: ["vendor"]
})
)
}
// ------------------------------------
// Pre-Loaders
// ------------------------------------
/*
[ NOTE ]
We no longer use eslint-loader due to it severely impacting build
times for larger projects. `npm run lint` still exists to aid in
deploy processes (such as with CI), and it"s recommended that you
use a linting plugin for your IDE in place of this loader.
If you do wish to continue using the loader, you can uncomment
the code below and run `npm i --save-dev eslint-loader`. This code
will be removed in a future release.
webpackConfig.module.preLoaders = [{
test: /\.(js|jsx)$/,
loader: "eslint",
exclude: /node_modules/
}]
webpackConfig.eslint = {
configFile: paths.base(".eslintrc"),
emitWarning: __DEV__
}
*/
// ------------------------------------
// Loaders
// ------------------------------------
// JavaScript / JSON
webpackConfig.module.rules = [{
test: /\.(js|jsx|ts|tsx)$/,
loader: "happypack/loader",
options: {
"id": "javascript"
},
exclude: /node_modules/
},
/*
{
test: /\.(js|jsx|ts|tsx)$/,
loader: "babel-loader",
options: {
cacheDirectory: true,
plugins: ["@babel/plugin-transform-runtime"],
presets: [
[
"@babel/env",
{
"targets": {
"browsers": [ "last 2 versions" ]
}
}
],
"@babel/stage-3",
"@babel/react",
"@babel/preset-typescript"
],
env: {
production: {
//presets: ["react-optimize"] //Fails for react-bootstrap sub components
}
}
},
exclude: /node_modules/
},
*/
{
test: /\.json$/,
loader: "json-loader"
}];
// ------------------------------------
// Style Loaders
// ------------------------------------
// We use cssnano with the postcss-loader loader, so we tell
// css-loader not to duplicate minimization.
/*
const BASE_CSS_LOADER = function (loadersNumber) {
return {
loader: "css-loader",
options: {
sourceMap: true,
"minimize": true,
"modules": !!loadersNumber,
"importLoaders": loadersNumber || 1
}
}
};
*/
const cssModulesLoader = function (loadersNumber: number, uselocalIdentName: boolean): webpack.NewLoader {
var result__: webpack.NewLoader = {
loader: "css-loader"
};
result__.options = {
sourceMap: true,
"minimize": true
};
if (typeof loadersNumber != "undefined" && loadersNumber > 0) {
result__.options ["modules"] = true;
result__.options ["importLoaders"] = loadersNumber;
}
if (uselocalIdentName) {
result__.options ["localIdentName"] = "[name]__[local]___[hash:base64:5]";
}
return result__;
};
const BASE_POST_CSS_LOADER = {
loader: "postcss-loader",
options: {
ident: "postcss",
plugins: (loader: any) => [
require ("cssnano") ({
autoprefixer: {
add: true,
remove: true,
browsers: ["last 2 versions"]
},
discardComments: {
removeAll: true
},
discardUnused: false,
mergeIdents: false,
reduceIdents: false,
safe: true,
sourcemap: true
})
]
}/*,
options: {
config: {
ctx: {
cssnano: {
autoprefixer: {
add: true,
remove: true,
browsers: ["last 2 versions"]
},
discardComments: {
removeAll: true
},
discardUnused: false,
mergeIdents: false,
reduceIdents: false,
safe: true,
sourcemap: true
}
}
}
}
*/
};
// Add any packge names here whose styles need to be treated as CSS modules.
// These paths will be combined into a single regex.
const PATHS_TO_TREAT_AS_CSS_MODULES = [
// "react-toolbox", (example)
]
// If config has CSS modules enabled, treat this project"s styles as CSS modules.
if (config.compiler_css_modules) {
PATHS_TO_TREAT_AS_CSS_MODULES.push(
paths.client().replace(/[\^\$\.\*\+\-\?\=\!\:\|\\\/\(\)\[\]\{\}\,]/g, "\\$&") // eslint-disable-line
)
}
const isUsingCSSModules = !!PATHS_TO_TREAT_AS_CSS_MODULES.length
const cssModulesRegex = new RegExp(`(${PATHS_TO_TREAT_AS_CSS_MODULES.join("|")})`)
// Loaders for styles that need to be treated as CSS modules.
if (isUsingCSSModules) {
/*
const cssModulesLoader = function (loadersNumber) {
return {
loader: "css-loader",
options: {
sourceMap: true,
"minimize": true,
"modules": !!loadersNumber,
"importLoaders": loadersNumber || 1,
"localIdentName": "[name]__[local]___[hash:base64:5]"
}
}
};
*/
webpackConfig.module.rules.push({
test: /\.scss$/,
include: cssModulesRegex,
exclude: /\/src\/styles\//,
use: [
{
loader: "style-loader"
},
cssModulesLoader (1, true),
// BASE_POST_CSS_LOADER,
{
loader: "sass-loader",
options: {
sourceMap: true,
includePaths: [paths.client("styles") ]
}
}
]
});
webpackConfig.module.rules.push({
test: /\.scss$/,
include: /\/src\/styles\//,
use: [{
loader: "style-loader"
},
cssModulesLoader (1, false),
// BASE_POST_CSS_LOADER,
{
loader: "sass-loader",
options: {
sourceMap: true,
includePaths: [ paths.client("styles") ]
}
}]
});
/*
webpackConfig.module.rules.push({
test: /\.css$/,
loader: "string-replace",
query: {
search: /body/g,
replace: "#application"
}
});*/
/*
webpackConfig.module.rules.push({
test: /.*\/main\.css$/,
loader: "string-replace",
query: {
search: /body/g,
replace: "#application"
}
});
*/
webpackConfig.module.rules.push({
test: /\.css$/,
include: cssModulesRegex,
use: [
{
loader: "style-loader"
},
cssModulesLoader (0, true)//,
// BASE_POST_CSS_LOADER
]
});
}
// Loaders for files that should not be treated as CSS modules.
const excludeCSSModules = isUsingCSSModules ? cssModulesRegex : false;
webpackConfig.module.rules.push ({
test: /\.scss$/,
exclude: excludeCSSModules,
use: [
{
loader: "style-loader"
},
cssModulesLoader (1, false),
// BASE_POST_CSS_LOADER,
{
loader: "sass-loader",
options: {
sourceMap: true,
includePaths: [ paths.client("styles") ]
}
}
]
} as any);
webpackConfig.module.rules.push ({
test: /\.css$/,
exclude: excludeCSSModules,
use: [
{
loader: "style-loader"
},
cssModulesLoader (0, false)//,
// BASE_POST_CSS_LOADER
]
} as any);
webpackConfig.module.rules.push({
test: /app\.min\.css$/,
loader: "string-replace-loader",
options: {
search: "body",
replace: "#application",
flags: "g"
}
});
// ------------------------------------
// Style Configuration
// ------------------------------------
// TODO: (Alexander) Do we really need this ???
/*
webpackConfig.postcss = [
cssnano({
autoprefixer: {
add: true,
remove: true,
browsers: ["last 2 versions"]
},
discardComments: {
removeAll: true
},
discardUnused: false,
mergeIdents: false,
reduceIdents: false,
safe: true,
sourcemap: true
})
]
*/
// File loaders
/* eslint-disable */
webpackConfig.module.rules.push(
{
test: /\.woff(\?.*)?$/,
loader: "url-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]",
limit: "10000",
mimetype: "application/font-woff"
}
},
{
test: /\.woff2(\?.*)?$/,
loader: "url-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]",
limit: "10000",
mimetype: "application/font-woff2"
}
},
{
test: /\.otf(\?.*)?$/,
loader: "file-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]",
limit: "10000",
mimetype: "font/opentype"
}
},
{
test: /\.ttf(\?.*)?$/,
loader: "url-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]",
limit: "10000",
mimetype: "application/octet-stream"
}
},
{
test: /\.eot(\?.*)?$/,
loader: "file-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]"
}
},
{
test: /\.svg(\?.*)?$/,
loader: "url-loader",
options: {
prefix: "fonts/",
name: "[path][name].[ext]",
limit: "10000",
mimetype: "image/svg+xml"
}
},
{
test: /\.(png|jpg)$/,
loader: "url-loader",
options: {
limit: "8192"
}
}
)
/* eslint-enable */
// ------------------------------------
// Finalize Configuration
// ------------------------------------
// when we don"t know the public path (we know it only when HMR is enabled [in development]) we
// need to use the extractTextPlugin to fix this issue:
// http://stackoverflow.com/questions/34133808/webpack-ots-parsing-error-loading-fonts/34133809#34133809
//TODO: (Alexander) Gonna make it later ...
/*
if (!__DEV__) {
debug("Apply ExtractTextPlugin to CSS loaders.")
webpackConfig.module.rules.filter((loader) =>
loader.loaders && loader.loaders.find((name) => /css/.test(name.split("?")[0]))
).forEach((loader) => {
const [first, ...rest] = loader.loaders
loader.loader = ExtractTextPlugin.extract(first, rest.join("!"))
Reflect.deleteProperty(loader, "loaders")
})
webpackConfig.plugins.push(
new ExtractTextPlugin("[name].[contenthash].css", {
allChunks: true
})
)
}
*/
export { webpackConfig };
+1
Node 9.5.0 (ubuntu) Webpack 3.5.5 Happypack 4.0.1 2,715 modules
Build time without Happypack: 25s Build time with Happypack: 27s
Looking at the activity monitor, only one node process seems to be doing anything during the build. For me, it does the same thing with or without cacheDirectory set for babel-loader
