rspack
rspack copied to clipboard
web-infra-dev
[Issue] rspack appears to run out of memory and fail silenty on local and CI bundle step (large bundle size of almost 100 MB)
Hello rspack team,
We seem to be running into an issue with rspack when attempting to bundle, both locally and in CI.
Some infra info:
- we are using Yarn
- we are using rspack like so: "bundle:client": "cross-env TS_NODE_PROJECT="tsconfig.webpack.json" rspack --config webpack.config.client.ts -- mode production && yarn check:bundle:duplicates"
- this particular change brings in many dependencies, but they should be either lazy loaded or in devDependencies
Both locally and in CI, this bundling step seems to just end, with logs truncated and no final message ("like rspack compiled x in y seconds")
We are operating in a pretty massive monorepo and ton of dependencies are being pulled in with this particular change that fails during bundling. The module with many deps in particular is lazy loaded, but for our relay build, we've had to include some deps in dev dependencies.
Thank you.
Some guesses and workarounds.
- Enlarge your CI environment memory size
- Try
NODE_OPTIONS="--max-old-space-size=8192" - Try disabling minimized
minimize: falsehttps://rspack.rs/config/optimization#optimizationminimize to see if it's minizer freezing
Or show a copy of webpack.config.client.ts
Okay, thank you for the incredibly speedy response @stormslowly . Let me try those out - for reference, here is a copy of the webpack.config.client.ts (I apologize for the formatting, it doesn't seem to want to play nice:
import { getAliasForProject, getDevServerConfig } from "@1js/webpack-common";
import path from "path";
import type { Configuration } from "webpack";
import type { Configuration as DevServerConfiguration } from "webpack-dev-server";
import { AzureSymbolServerPlugin } from "@1js/azure-symbol-server-webpack-plugin";
import { BundleStatsPlugin } from "@1js/webpack-bundle-stats-plugin";
import { getSwcLoaderRule } from "@1js/webpack-typescript-rules";
import rspack from "@rspack/core";
import { RspackManifestPlugin } from "rspack-manifest-plugin";
import { BundleAnalyzerPlugin } from "webpack-bundle-analyzer";
const prod: boolean = process.env["WEBPACK_MODE"] !== "dev";
const keyAssets = [
{
// vendors for main entry point and the layout (root) route
name: "main.vendors.[hash].js",
threshold: 10000,
},
{
// main entry point
name: "main.[hash].js",
threshold: 10000,
},
{
// localization strings
name: "loc_en-us.[hash].chunk.js",
threshold: 10000,
},
{
// vendors for chat route
name: "route-chat.vendors.[hash].chunk.js",
threshold: 10000,
},
{
// chat route chunk
name: "route-chat.[hash].chunk.js",
threshold: 10000,
},
];
function excludeLocalizationStringsFromStateExceptEnUS(stats: any): any {
const ignoredAssetPatterns = [
/chat_([a-f0-9]{16})\.svg$/i, // this asset is duplicated and is causing issues during web-stats diff process
/erroricon_([a-f0-9]{16})\.png$/i, // this asset is duplicated and is causing issues during web-stats diff process
/warning_([a-f0-9]{16})\.png$/i, // this asset is duplicated and is causing issues during web-stats diff process
/warningillustrationdark_([a-f0-9]{16})\.svg$/i, // this asset is duplicated and is causing issues during web-stats diff process
];
if (stats.assets) {
stats.assets = stats.assets.filter((asset) => {
// Keep en-us localization strings, exclude others
if (
["loc_c", "loc_", "aihublibrarymodule_"].some((v) =>
asset.name.includes(v)
)
) {
return (
asset.name.includes("_en-us.") || asset.name.includes("_en-us-json.")
);
}
if (ignoredAssetPatterns.some((p) => p.test(asset.name))) {
return false;
}
return true;
});
}
return stats;
}
const getConfig = async (): Promise<
Configuration & { devServer?: DevServerConfiguration }
> => {
const outputPath = path.join(
__dirname,
"dist",
prod ? "cdnresources" : "client"
);
const mode = prod ? "production" : "development";
const config: Configuration & { devServer?: DevServerConfiguration } = {
mode,
entry: prod ? "./lib/app/entry.client.js" : "./src/app/entry.client.tsx",
name: "app-name",
devtool: prod ? "hidden-source-map" : "eval-cheap-module-source-map",
output: {
path: outputPath,
sourceMapFilename: "../sourcemaps/[base].map[query]",
filename: (pathData) => {
if (!prod) return `[name].js`;
const name =
pathData.chunk?.name ?? String(pathData.chunk?.id ?? "chunk");
return `${name.toLowerCase()}.[contenthash].js`;
},
chunkFilename: (pathData) => {
if (!prod) return `[name].chunk.js`;
const name =
pathData.chunk?.name ?? String(pathData.chunk?.id ?? "chunk");
return `${name.toLowerCase()}.[contenthash].chunk.js`;
},
crossOriginLoading: "anonymous",
},
externals: {
// These Node.js core modules should not be bundled for the browser
fs: "commonjs fs",
path: "commonjs path",
},
resolve: {
extensions: [
".web.ts",
".ts",
".web.tsx",
".tsx",
".web.mjs",
".mjs",
".web.js",
".js",
".web.jsx",
".jsx",
".json",
],
alias: {
...getAliasForProject(),
// Map loadStrings imports to generated folder to avoid double write during build
// Using absolute path to be specific to this package only
...(prod
? {
[path.resolve(__dirname, "lib/strings/client")]: path.resolve(
__dirname,
"lib/strings_generated/client/index.js"
),
[path.resolve(__dirname, "lib/strings/server")]: path.resolve(
__dirname,
"lib/strings_generated/server/index.js"
),
[path.resolve(__dirname, "lib/app/routes/signinDev")]:
path.resolve(__dirname, "lib/app/components/noopClient.js"),
}
: {}),
// Fix DOMPurify import issue - use purify.js export that is defined in package.json exports
dompurify$: "dompurify/purify.js",
},
fallback: {
// Disable Node.js modules for browser bundles - packages like sort-css-media-queries
// should not be using these modules in browser context
fs: false,
path: false,
os: false,
crypto: false,
stream: false,
buffer: false,
util: false,
},
},
resolveLoader: {
modules: [path.resolve(__dirname, "src/rspack/loaders"), "node_modules"],
},
module: {
rules: [
getSwcLoaderRule({
isDevelopment: !prod,
enablePolyfills: false,
plugins: [
[
"@swc/plugin-relay",
{
language: "typescript",
rootDir: __dirname,
eagerEsModules: false,
artifactDirectory: "src/__generated__",
},
],
],
}),
{
// Allow importing .md and .txt files as strings.
test: /\.(md|txt)$/,
type: "asset/source",
},
{
test: /\.css$/i,
use: ["style-loader", "css-loader"],
},
{
test: /\.less$/,
use: ["style-loader", "css-loader", "less-loader"],
},
{
test: /\.scss$/,
use: ["style-loader", "css-loader", "sass-loader"],
},
{
test: /\.(jpg|png|woff|woff2|eot|ttf|svg|gif|ico|mp4|mp3|webp|avif|html)$/,
type: "asset/resource",
generator: {
filename: (pathData) => {
const full = pathData.filename || "";
const base = path
.basename(full, path.extname(full))
.toLowerCase();
const ext = path.extname(full);
return `${base}_[hash]${ext}`;
},
},
},
],
},
target: "web",
optimization: {
minimize: prod,
splitChunks: {
chunks: "all",
minSize: 50000,
cacheGroups: {
mainVendors: {
test: /[\\/]node_modules[\\/]/,
name: "main.vendors",
chunks: (chunk) => chunk.name === "main",
priority: 20,
enforce: true,
},
chatVendors: {
test: /[\\/]node_modules[\\/]/,
name: "route-chat.vendors",
chunks: (chunk) => chunk.name === "route-chat",
priority: 20,
enforce: true,
},
default: {
minChunks: 2,
priority: 10,
chunks: (chunk) => !chunk.name?.startsWith("route-"),
},
},
},
},
plugins: [
new rspack.ProgressPlugin({
prefix: "Build",
profile: true,
template:
"● {prefix:.bold} {bar:25.green/white.dim} ({percent}%) {wide_msg:.dim}",
}),
// Ignore Node.js modules in sort-css-media-queries for browser bundle
new rspack.IgnorePlugin({
checkResource(resource, context) {
// Ignore fs and path imports in sort-css-media-queries
if (context && context.includes("sort-css-media-queries")) {
return resource === "fs" || resource === "path";
}
return false;
},
}),
new rspack.DefinePlugin({
"process.env": JSON.stringify({ NODE_ENV: mode }),
__DEV__: JSON.stringify(mode === "development"),
IS_WEBPACK5: JSON.stringify(true),
__DISABLE_PSEUDO_LOC__: JSON.stringify(true),
}),
...(prod
? [
new AzureSymbolServerPlugin(),
new BundleStatsPlugin(
"app-name",
"../",
"client",
keyAssets,
excludeLocalizationStringsFromStateExceptEnUS
),
new BundleAnalyzerPlugin({
analyzerMode: "static",
reportFilename: `../azurestaticwebresources/app-name.client.stats.html`,
generateStatsFile: false,
openAnalyzer: false,
excludeAssets: [/\.worker\..*\.js$/i],
}),
new RspackManifestPlugin({
fileName: path.join(
"..",
"resourcecatalog",
"app-name.manifest.json"
),
}),
new rspack.IgnorePlugin({
resourceRegExp: /^@1js\/m365-chat-editor-client$/,
}),
]
: []),
],
watchOptions: {
ignored: /node_modules/,
poll: 1000,
},
};
// Add dev server configuration only in development mode
if (!prod) {
config.devServer = {
...(await getDevServerConfig({
port: 3001,
disableSsl: false,
})),
static: path.join(__dirname, "dist/client"),
hot: true,
historyApiFallback: true,
};
}
return config;
};
module.exports = getConfig;
@wongrichardalex can you provide more logs info(like screenshot or log.txt)
I believe the issue originates from the @1js/azure-symbol-server-webpack-plugin plugin. It subscribes to the processAssets hook, which copies source code strings and source map strings from Rust memory to the JavaScript side. For large projects, this may exceed node's default heap memory limit and cause errors.
https://github.com/microsoft/azure-devops-symbols/blob/main/packages/webpack-plugin/src/plugin.ts#L80
