loadTasks as they are needed to speed up Grunt load time

[dylang]

The problem is that Grunt loads all tasks every time it runs. Even if the user just wants to run grunt jshint, it will still load grunt-uglify, grunt-sass, etc.

This can be slow if a task has a lot of dependencies that are require'ed before the task is run.

For example, this is a slow to load task:

module.exports = function(grunt) {

    // These `require` statements execute even if this task isn't run.
    var path = require('path'),
        semver = require('semver'),
        q = require('q'),
        utils = require('./utils/utils'),
        constants = require('./utils/constants'),
        local = require('./core/local')(grunt),
        styles = require('./core/styles')(grunt),
        debug = require('./utils/debug')(grunt),
        install = require('./core/install')(grunt);

    grunt.registerTask(
        'build',
        'Build project',
        function(){
           // Code here runs when the task runs.
           // Grunt would load faster if devs put their `require`
           // statements here because they would only run
           // when the task is run but it's much more common
           // in the Node world to put all `require`s at the top.
           // (code removed for this example)
        }
    );

I've made some changes to time-grunt so you can see how long it takes for tasks to load vs their actual run time. In that screenshot there are other tasks loading that aren't used but they still contribute to the load time.

[cowboy]

Maybe plugin authors should be told to defer requiring of libs until needed.

Also, task names can be totally different from plugin names. For example, if plugin "grunt-foo" had tasks "bar" and "baz" how could Grunt know which task files to run when the user ran grunt qux which is an alias task for "bar" followed by "baz"?

[dylang]

Also, task names can be totally different from plugin names. For example, if plugin "grunt-foo" had tasks "bar" and "baz" how could Grunt know which task files to run when the user ran grunt qux which is an alias task for "bar" followed by "baz"?

Maybe it's worth revisiting this feature for Grunt 0.5? If you want a task called foo then put your code in a file called /tasks/foo.js. If you also need a task called bar then create tasks/bar.js.

Or we could more extreme - the name of the node module is the only task that is registered. grunt-foo only can register a task called foo.

Maybe plugin authors should be told to defer requiring of libs until needed.

I think this is an awkward pattern for seasoned Node developers. We're changing our Grunt tasks to this pattern and it feels "dirty" to me. This is only my opinion, what do others think?

[cowboy]

If "require" was declarative and processed as part of a pre-compilation step—like requiring stdio.h in C—it would make sense to specify all libraries-to-be-required up-front. Unfortunately, require in Node.js doesn't behave this way. It's just a function call. As such, requiring a library in Node.js is subject to a run-time performance penalty, and should probably be deferred when necessary.

Lazily evaluating require calls might be considered an anti-pattern by those who maintain that require is declarative (which it is not) or by those who write tools that scan .js files for require calls in order to build library dependency graphs (which is very hacky), but it is a completely valid technique for solving this specific problem: deferring expensive operations until later.

module.exports = function(grunt) {
  // Just one way to solve this problem...
  var lib1, lib2, lib3;
  var init = function() {
    lib1 = require("lib1");
    lib2 = require("lib2");
    lib3 = require("lib3");
    init = function() {};
  };

  grunt.registerTask("foo", "do something.", function() {
    init();
    lib1(lib2, lib3).whatever();
  });
};

I'd imagine that with proxies, a "lazy" require will be able to be created to simplify this process.

[dylang]

I'm proposing taking advantage of lazy loading but doing it in Grunt instead of in the tasks. This removes the responsibility from the task developers, at the cost of backwards compatibility for tasks with source files that don't match the task names.

Currently grunt.tasks.loadTasks scans directories and does the require right away:

function loadTasks(tasksdir) {
  try {
     // Scan for available tasks
    var files = grunt.file.glob.sync('*.{js,coffee}', {cwd: tasksdir, maxDepth: 1});
    files.forEach(function(filename) {
      // "require" the task file
      loadTask(path.join(tasksdir, filename));
    });
  } catch(e) {
    grunt.log.verbose.error(e.stack).or.error(e);
  }
}

I'm proposing not calling loadTask for a task until the task needs to run.

BTW, I really appreciate that you are taking time from a well-deserved vacation (and Node Knockout?) to post replies to discussions like this one. I have no expectation of a quick reply and was impressed to see one.

[tmaslen]

I've wanted this feature for a while too, especially as I'm using a plugin that depends on imagemin which can take up to 30 seconds to spin up. This slows everything down and is especially annoying as the task that relies on imagemin is barely used.

I've recently found a way round this. I've moved the loadNpmTask into a custom task so its conditionally loaded only when its needed. This is how the gruntfile used to be structured...

grunt.loadNpmTasks('grunt-contrib-imagemin');
grunt.registerTask('images', ['copy:standardImages', 'responsive_images', 'imagemin']);

And this is how it is now...

grunt.registerTask('images', [], function () {
    grunt.loadNpmTasks('grunt-contrib-imagemin');
    grunt.task.run('copy:standardImages', 'responsive_images', 'imagemin');
});

Hope this helps and that I'm not doing something obviously wrong :-).

/t

[jacksleight]

@maslen Thanks for this, works great for me. Managed to reduce my compass compilation task by ~1.5 secs (was taking just over 3), which makes all the difference when you're using watch/livereload. For me grunt-contrib-imagemin is one of the worst offenders for startup lag, adds about a second.

[stephanebachelier]

Really helpful trick!

This seems to me the best solution :) Each task loads its own dependencies. But what happens if two modules load the same modules? Does grunt know what task are loaded ?

Stéphane Bachelier, Tél. 06 42 24 48 09 B8A5 2007 0004 CDE4 5210 2317 B58A 335B B5A4 BFC2

2013/11/22 Tom Maslen [email protected]

I've wanted this feature for a while too, especially as I'm using a plugin that depends on imagemin which can take up to 30 seconds to spin up. This slows everything down and is especially annoying as the task that relies on imagemin is barely used.

I've recently found a way round this. I've moved the loadNpmTask into a custom task so its conditionally loaded only when its needed. This is how the gruntfile used to be structured...

grunt.loadNpmTasks('grunt-contrib-imagemin'); grunt.registerTask('images', ['copy:standardImages', 'responsive_images', 'imagemin']);

And this is how it is now...

grunt.registerTask('images', [], function () { grunt.loadNpmTasks('grunt-contrib-imagemin'); grunt.task.run('copy:standardImages', 'responsive_images', 'imagemin'); });

Hope this helps and that I'm not doing something obviously wrong :-).

/t

— Reply to this email directly or view it on GitHubhttps://github.com/gruntjs/grunt/issues/975#issuecomment-29058707 .

[shaekuronen]

@maslen this is dope, shaved 1.9s off tasks loading in my preview task (which uses watch/livereload, so definitely matters)

if anyone wants to see an implementation of maslen's technique, just pushed it up to grunt-ejs-static-boilerplate

what's strange is it also shaved almost that much time off tasks loading for my optimize task, which uses all the modules I was previously loading indiscriminately for the preview task. Somehow just moving loadNpmTasks inside registerTask dramatically reduced the amount of time to load. Did not see increases in any other tasks, so looks like net gain.

thanks!

[wildeyes]

+1

[yuanyan]

@maslen Register a custom task for delay load really really nice trick.

[lmartins]

Some tasks can have a huge impact in running times, as I've just found out with the grunt-contrib-imagemin.

Im running a watcher to compile SASS, which takes about 24ms. When I add grunt-contrib-imagemin to the gruntfile, even when im not optimizing any image compilation times will immediately jump to more than 3 seconds. The time-grunt plugin shows me that 99% of that is spent on loading tasks.

You might say that 2-3 seconds is not a huge time, but when working with CSS where you are constantly saving for visualizing your changes this really degrades the workflow.

Would be really cool if this was something that could be improved in any way.

[vladikoff]

@lmartins imagemin is being updated in https://github.com/gruntjs/grunt-contrib-imagemin/pull/125

[kud]

:+1:

Just a sample of my watch:

Execution Time (2013-12-19 09:39:00 UTC)
loading tasks          3s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 62%
jsvalidate:compile  241ms  ▇▇▇ 6%
jshint:dev             1s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 28%
handlebars:compile   97ms  ▇▇ 2%
concat:compile       56ms  ▇ 1%

Loading tasks is ALWAYS the hugest task (except Sass for sure).

[lmartins]

Tried @maslen solution and it really helps. Shaves 2+ seconds on every save and now looks like this:

Execution Time (2013-12-19 09:55:38 UTC)
loading tasks  349ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 93%
sass:compile    23ms  ▇▇▇▇ 6%
Total 374ms

Thanks man, this makes it viable to me keep using grunt to compile sass.

[kud]

Indeed. Loading just packages you use is perfectly sane and efficient. Thanks @maslen !

[kud]

Execution Time (2013-12-19 11:45:37 UTC)
loading tasks       419ms  ▇▇ 3%
jsvalidate:compile  296ms  ▇▇ 2%
jshint:dev             1s  ▇▇▇▇▇▇ 11%
shell:fontcustom    970ms  ▇▇▇▇▇ 8%
concurrent:concat      9s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 74%
Total 12s

\o/

So much better, thanks!

[shootaroo]

I created a JIT(Just In Time) plugins loader for Grunt. https://github.com/shootaroo/jit-grunt

You can speed up while maintaining the simple Gruntfile. Try it.

[henrahmagix]

@shootaroo Your jit-grunt module worked fantastically well for me! Thanks!

[thSoft]

@shootaroo, thank you for your excellent module!

[bmds]

@shootaroo You just shaved nearly 4 seconds off my build process, fantastic module

[abarnwell]

@shootaroo, awesome plugin, thank you!

[Scotchester]

@shootaroo Thank you for the great module! It's working perfectly for me.

[gpluess]

@shootaroo Fantastic module, thank you!

[stevenvachon]

Indeed. Works great for me as well, and he quickly fixed issues with it too.

[eckdanny]

Sweeeeeeeeet! Thanks a ton @shootaroo!

[thom4parisot]

Thanks @shootaroo, you have the good approach I think.

Registering a plugin should not imply to load the task itself: grunt should just know about the task dependencies tree, then asynchronously loads the tasks (if not already loaded) and run them. So it goes in the same direction as @dylang proposes: keeping the same syntax and just changing how grunt handles their load under the hood.

[arthurpf]

@shootaroo Thank you! This is exactly what I was looking for!

[raphaeleidus]

similar to @shootaroo's module I created this for lazyloading plugins https://github.com/raphaeleidus/grunt-lazyload

[raphaeleidus]

@oncletom @shootaroo The grunt API allows one module to register multiple tasks, so unless you eager load the module or provide a full list of tasks registered this would be broken by lazy-loading. my module(https://github.com/raphaeleidus/grunt-lazyload) requires specifying the task names. I am not sure how many grunt plugins are actually taking advantage of this but unless the design pattern changes I could not find a way to lazy load tasks using the standard API.

[hirokith]

@shootaroo Thank you!