moleculer
moleculer copied to clipboard
moleculerjs
hot-reload not working with moleculer runner in dev container
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [x] I am running the latest version
- [x] I checked the documentation and found no answer
- [x] I checked to make sure that this issue has not already been filed
- [x] I'm reporting the issue to the correct repository
Current Behavior
When moleculer project is opened in a dev container using vscode and docker, hot-reload of services when changed is not happening. When same project is opened in vscode without using a dev container hot-reload works. Furthermore in dev container command ./node_modules/moleculer/bin/moleculer-runner.js --hot --repl -E dev.env --config moleculer.config.ts backend/services/**/*.service.ts backend/services/**/**/*.service.ts is needed to find all services in dev container whereas ./node_modules/moleculer/bin/moleculer-runner.js --hot --repl -E dev.env --config moleculer.config.ts backend/services/**/*.service.ts finds all services when not using dev container.
Expected Behavior
Hot-reload to work and reload services when changed.
Steps to Reproduce
Please provide detailed steps for reproducing the issue.
- create moleculer project
- add devcontainer.json in .devcontainer folder at root of project (https://code.visualstudio.com/docs/remote/containers)
- open project in dev container
- change service code and save file.
Context
Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.
- Moleculer version: 4.11
- NodeJS version: 14.11
- Operating System: windows 10 WSL
cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" VERSION_CODENAME=stretch ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
Additional info
The issue is with fs.watch. It has issues working on linux. https://nodejs.org/api/fs.html#fs_caveats
I have created a temporary fix for now by pulling the hot reload middleware and modifying it. I set hotReload: false in moleculer.config and am not using the --hot or -hot in my npm run command. I imported the updated middleware and added it to middleware section of the config. Chokidar will need to be installed via npm, but I've plugged it into the .watch areas of the middleware. This is a TS file, but can quickly be changed to js. Hot reload now works in dev containers for me as expected.
temp middleware for hot reload:
HotReload.ts Middleware code (click to show)
/* eslint-disable @typescript-eslint/no-explicit-any */
/* eslint-disable complexity */
/*
* moleculer
* Copyright (c) 2019 MoleculerJS (https://github.com/moleculerjs/moleculer)
* MIT Licensed
*/
'use strict';
import fs from 'fs';
import chokidar from 'chokidar';
import kleur from 'kleur';
import path from 'path';
import _ from 'lodash';
const {
clearRequireCache,
makeDirs,
isFunction,
// eslint-disable-next-line @typescript-eslint/no-var-requires
} = require('../../node_modules/moleculer/src/utils');
/* istanbul ignore next */
module.exports = function HotReloadMiddleware(broker: any) {
const cache = new Map();
let projectFiles = new Map();
let prevProjectFiles = new Map();
let hotReloadModules: any[] = [];
function hotReloadService(service: any) {
const relPath = path.relative(process.cwd(), service.__filename);
broker.logger.info(`Hot reload '${service.name}' service...`, kleur.grey(relPath));
return broker.destroyService(service).then(() => {
if (fs.existsSync(service.__filename)) {
return broker.loadService(service.__filename);
}
});
}
/**
* Detect service dependency graph & watch all dependent files & services.
*
*/
function watchProjectFiles() {
if (!broker.started || !process.mainModule) return;
cache.clear();
prevProjectFiles = projectFiles;
projectFiles = new Map();
// Read the main module
const mainModule = process.mainModule;
// Process the whole module tree
processModule(mainModule, null, 0, null);
const needToReload = new Set();
// Debounced Service reloader function
const reloadServices = _.debounce(() => {
broker.logger.info(
kleur.bgMagenta().white().bold(`Reload ${needToReload.size} service(s)`),
);
needToReload.forEach((svc) => {
if (typeof svc == 'string') return broker.loadService(svc);
return hotReloadService(svc);
});
needToReload.clear();
}, 500);
// Close previous watchers
stopAllFileWatcher(prevProjectFiles);
// Watching project files
broker.logger.debug('');
broker.logger.debug(kleur.yellow().bold('Watching the following project files:'));
projectFiles.forEach((watchItem, fName) => {
const relPath = path.relative(process.cwd(), fName);
if (watchItem.brokerRestart)
broker.logger.debug(` ${relPath}:`, kleur.grey('restart broker.'));
else if (watchItem.allServices)
broker.logger.debug(` ${relPath}:`, kleur.grey('reload all services.'));
else if (watchItem.services.length > 0) {
broker.logger.debug(
` ${relPath}:`,
kleur.grey(
`reload ${watchItem.services.length} service(s) & ${watchItem.others.length} other(s).`,
) /*, watchItem.services, watchItem.others*/,
);
watchItem.services.forEach((svcFullname: any) =>
broker.logger.debug(kleur.grey(` ${svcFullname}`)),
);
watchItem.others.forEach((filename: any) =>
broker.logger.debug(
kleur.grey(` ${path.relative(process.cwd(), filename)}`),
),
);
}
// Create watcher
watchItem.watcher = chokidar
.watch(fName, { ignoreInitial: true, usePolling: true, interval: 1000 })
.on('all', (eventType, fName) => {
const relPath = path.relative(process.cwd(), fName);
broker.logger.info(
kleur
.magenta()
.bold(
`The '${relPath}' file is changed. (Event: ${eventType.toString()})`,
),
);
// Clear from require cache
clearRequireCache(fName);
if (watchItem.others.length > 0) {
watchItem.others.forEach((f: unknown) => clearRequireCache(f));
}
if (
watchItem.brokerRestart &&
broker.runner &&
isFunction(broker.runner.restartBroker)
) {
broker.logger.info(
kleur.bgMagenta().white().bold('Action: Restart broker...'),
);
stopAllFileWatcher(projectFiles);
// Clear the whole require cache
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
require.cache.length = 0;
return broker.runner.restartBroker();
} else if (watchItem.allServices) {
// Reload all services
broker.services.forEach((svc: Record<string, unknown>) => {
if (svc.__filename) needToReload.add(svc);
});
reloadServices();
} else if (watchItem.services.length > 0) {
// Reload certain services
broker.services.forEach((svc: Record<string, unknown>) => {
if (watchItem.services.indexOf(svc.fullName) !== -1)
needToReload.add(svc);
});
if (needToReload.size == 0) {
// It means, it's a crashed reloaded service, so we
// didn't find it in the loaded services because
// the previous hot-reload failed. We should load it
// broker.loadService
needToReload.add(relPath);
}
reloadServices();
}
});
});
if (projectFiles.size == 0) broker.logger.debug(kleur.grey(' No files.'));
}
const debouncedWatchProjectFiles = _.debounce(watchProjectFiles, 2000);
/**
* Stop all file watchers
*/
function stopAllFileWatcher(items: any) {
items.forEach((watchItem: any) => {
if (watchItem.watcher) {
watchItem.watcher.close();
watchItem.watcher = null;
}
});
}
/**
* Get a watch item
*
* @param {String} fName
* @returns {Object}
*/
function getWatchItem(fName: string) {
let watchItem = projectFiles.get(fName);
if (watchItem) return watchItem;
watchItem = {
services: [],
allServices: false,
brokerRestart: false,
others: [],
};
projectFiles.set(fName, watchItem);
return watchItem;
}
function isMoleculerConfig(fName: string) {
return (
fName.endsWith('moleculer.config.js') ||
fName.endsWith('moleculer.config.ts') ||
fName.endsWith('moleculer.config.json')
);
}
/**
* Process module children modules.
*
* @param {*} mod
* @param {*} service
* @param {Number} level
*/
function processModule(mod: any, service = null, level = 0, parents = null) {
const fName = mod.filename;
// Skip node_modules files, if there is parent project file
if ((service || parents) && fName.indexOf('node_modules') !== -1)
if (hotReloadModules.find((modulePath) => fName.indexOf(modulePath) !== -1) == null)
return;
// Avoid circular dependency in project files
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (parents && parents.indexOf(fName) !== -1) return;
// console.log(fName);
// Cache files to avoid cyclic dependencies in node_modules
if (fName.indexOf('node_modules') !== -1) {
if (cache.get(fName)) return;
cache.set(fName, mod);
}
if (!service) {
service = broker.services.find((svc: any) => svc.__filename == fName);
}
if (service) {
// It is a service dependency. We should reload this service if this file has changed.
const watchItem = getWatchItem(fName);
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
if (!watchItem.services.includes(service.fullName))
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
watchItem.services.push(service.fullName);
watchItem.others = _.uniq([].concat(watchItem.others, parents || []));
} else if (isMoleculerConfig(fName)) {
const watchItem = getWatchItem(fName);
watchItem.brokerRestart = true;
} else {
// It is not a service dependency, it is a global middleware. We should reload all services if this file has changed.
if (parents) {
const watchItem = getWatchItem(fName);
watchItem.allServices = true;
watchItem.others = _.uniq([].concat(watchItem.others, parents || []));
}
}
if (mod.children && mod.children.length > 0) {
if (service) {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
parents = parents ? parents.concat([fName]) : [fName];
} else if (isMoleculerConfig(fName)) {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
parents = [];
// const watchItem = getWatchItem(fName);
// watchItem.brokerRestart = true;
} else if (parents) {
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
parents.push(fName);
}
mod.children.forEach((m: any) =>
processModule(m, service, service ? level + 1 : 0, parents),
);
}
}
const folderWatchers: any[] = [];
function watchProjectFolders() {
// Debounced Service loader function
const needToLoad = new Set();
const loadServices = _.debounce(() => {
broker.logger.info(
kleur.bgMagenta().white().bold(`Load ${needToLoad.size} service(s)...`),
);
needToLoad.forEach((filename) => {
try {
broker.loadService(filename);
} catch (err) {
broker.logger.error(`Failed to load service '${filename}'`, err);
clearRequireCache(filename);
}
});
needToLoad.clear();
}, 500);
if (broker.runner && Array.isArray(broker.runner.folders)) {
const folders = broker.runner.folders;
if (folders.length > 0) {
folderWatchers.length = 0;
broker.logger.debug('');
broker.logger.debug(kleur.yellow().bold('Watching the following folder(s):'));
folders.forEach((folder: any) => {
makeDirs(folder);
broker.logger.debug(` ${path.relative(process.cwd(), folder)}/`);
folderWatchers.push({
path: folder,
watcher: chokidar
.watch(folder, {
ignoreInitial: true,
usePolling: true,
interval: 1000,
})
.on('all', (eventType, filename) => {
if (
filename.endsWith('.service.js') ||
filename.endsWith('.service.ts')
) {
broker.logger.debug(
`There is changes in '${folder}' folder: `,
kleur.bgMagenta().white(eventType),
filename,
);
const fullPath = path.join(folder, filename);
const isLoaded = broker.services.some(
(svc: Record<string, unknown>) =>
svc.__filename == fullPath,
);
if (eventType.toString() === 'rename' && !isLoaded) {
// This is a new file. We should wait for the file fully copied.
needToLoad.add(fullPath);
loadServices();
} else if (eventType.toString() === 'change' && !isLoaded) {
// This can be a file which is exist but not loaded correctly (e.g. schema error if the file is empty yet)
needToLoad.add(fullPath);
loadServices();
}
}
}),
});
});
}
}
}
function stopProjectFolderWatchers() {
broker.logger.debug('');
broker.logger.debug('Stop watching folders.');
folderWatchers.forEach((item) => item.watcher && item.watcher.close());
}
/**
* Expose middleware
*/
return {
name: 'HotReload',
// After broker started
started(broker: any) {
/* if (broker.options.hotReload == null) {
return;
} else if (typeof broker.options.hotReload === 'object') {
if (Array.isArray(broker.options.hotReload.modules)) {
hotReloadModules = broker.options.hotReload.modules.map(
(moduleName: any) => `/node_modules/${moduleName}/`,
);
}
} else if (broker.options.hotReload !== true) {
return;
} */
watchProjectFiles();
watchProjectFolders();
},
serviceStarted() {
// Re-watch new services if broker has already started and a new service started.
if (broker.started) {
debouncedWatchProjectFiles();
}
},
stopped() {
stopProjectFolderWatchers();
},
};
};
I'm wondering what the status of this is? With all the issues with fs.watch, is there a plan to switch to chokidar? I see @Karnith started the effort, though I suspect the debounce mechanism from the original code isn't still required. I notice recursive-watch is being used in the project dependencies, though I didn't chase down what the use was, and could probably be replaced as well. I believe recursive-watch also uses fs.watch. I'm happy to contribute a working solution if nobody else is pushing this forward.
Looking at this a bit more, I notice that we are using chokidar in poling mode, which is required for monitoring files on networked filesystems. If this is the case, we could simply shift to using fs.watchFile, which is all chokidar is using. This would be a trivial change, though obviously may not work well for very large projects. If we want good watch performance and support for networked filesystems, we may need to offer a configuration option, allowing users to choose between poling or non-poling file watching. I've modified hot-reload.js to use fs.watchFile locally and it is working much better for me. Previously on OSx just doing an ls in the services directory would trigger a reload.
