AMD Support

[alexo]

The Asynchronous Module Definition (AMD) API specifies a mechanism for defining modules such that the module and its dependencies can be asynchronously loaded. This is particularly well suited for the browser environment where synchronous loading of modules incurs performance, usability, debugging, and cross-domain access problems.

The purpose of this issue is to add AMD support to wro4j.

Useful links:

http://requirejs.org/docs/whyamd.html https://github.com/amdjs/amdjs-api/wiki/AMD

Google code issue page: http://code.google.com/p/wro4j/issues/detail?id=527

[ivarconr]

To support AMD is pretty straight forward as you only have to deliver javascript-files in a well defined folder-structure.

The real challenge is to find a way to define and combines related scripts together into build layers (and then perform basic optimalization, minification and caching).

This is challenging mainly because:

• Building the layers with rhino is time-consuming because it needs deep analysis of all the transitive dependencies. I think the main problem is that Rhino is single threaded, while node can do much of the work in parallell with multiple threads.
• You need to support a "common" way to define the layer. RequireJS already has a standard for this and it is not aligned with the wro4j-group model.
• You need a way to support "empty" layers in development, and built layers in production so that it is possible to debug the code during development.

Useful links on build layers: http://requirejs.org/docs/optimization.html http://ivarconr.wordpress.com/2012/01/07/custom-build-in-dojo-1-7/

[ivarconr]

Another complexity is to support dependencies to external modules. Should we require the user to provide this our should wro4j be able to retrieve it?

[alexo]

I think no matter what will be the final approach, the following should be implemented:

• An interface (ResourceModuleExtractor ?) responsible for analysis of a javascript resource and extracting the following info: moduleId, moduleDependencies
• Create a simple implementation which uses a non-expensive approach of retrieving those info (ex: regex or simple parsing).
• In future, it will be possible to create a rhino approach (more reliable but also time-consuming)...

I imagine the flow like this:

• user still has to define a model (similar to wro.xml). This model should be used as a set of all resources to be analyzed by ResourceModuleExtractor.
• To make it less invasive, there will be a dedicated filter (WroAmdFilter ?) which will intercept request for js resources and will handle the dependencies lookup, merge and processing.
• When user requests a resource, there will be a layer responsible for building a graph of dependencies computed with ResourceModuleExtractor from the model. This graph will allow to determine the order used to merge resources. As a result, will be served all resources required by requested resources (minimized and merged).
• The pre/post processors are applied the same way it is done in current approach.
• Once a resource is served, the result will be cached (similar with the current approach)

My questions are:

• Do you agree with requirement to have a model used to analyze dependencies from? Or are there other alternatives?
• What do you think about ResourceModuleExtractor? (name, implementation, etc?)
• What happens if a single resource defines more than one moduleId? What about none?
• What happens when a duplicated moduleId is found?
• What happens if an invalid moduleId is referenced as a module dependency?
• What other info should the ResourceModuleExtractor retrieve from a js resource?
• How would look like a request for an AMD aware resource? Should we map a new filter to .js for that? Or it should be mapped to a folder like /wro/ ?
• Other ideas or suggestions?

[ivarconr]

I will try to answer your questions in short:

• Do you agree with requirement to have a model used to analyze dependencies from? Or are there other alternatives?
  • Yes, but I think it can be beneficial to look at how requirejs and other amd implementers has defined these kind of models. They tend to call it a "build profile". The important part is to have somewhere to tell how to layer the modules and where to find external libs.
• What do you think about ResourceModuleExtractor? (name, implementation, etc?)
  • This one bears the responsibility of building and providing the layers based on the model/build profile.
• What happens if a single resource defines more than one moduleId? What about none?
  • The moduleId is optional in AMD and is implicitly determined by the filename and folder location of that module. One script can define multiple modules and it is up to the module loader to discover these modules. If it does not define one i guess the module loader can have multiple "path" configurations for a module and try somewhere else. If the required module is not found the application will fail. Same moduleId can even be defined multiple times, but then only the first discoverd is used by the module loader.
• What happens when a duplicated moduleId is found?
  • They are expected to be equal and the first one is used.
• What happens if an invalid moduleId is referenced as a module dependency?
  • We have a problem! The build will fail.
• What other info should the ResourceModuleExtractor retrieve from a js resource?
  • HTML templates to be built in.
• How would look like a request for an AMD aware resource? Should we map a new filter to .js for that? Or it should be mapped to a folder like /wro/ ?
  • my intial thought is to have a filter. The important part is to maintain the correct folder path and filename as this represent the moduleid. Other ideas or suggestions?

[ivarconr]

To me this issue is divided in two parts:

1. Deliver optimized AMD-resources
   • No layering, no analyzis of dependencies etc.
   • regular pre/post processors
   • this is the easy part!
2. Build and deliver optimized AMD-layers
   • bundle together dependencies based on a model
   • should not include a dependency more than one time in a layer
   • regular pre/post processors
   • should be possible to include/exclude modules from a layer
   • should be possible to exclude all dependencies from layer_1 from layer_2
   • I am not sure if this can be done efficiently enough to perform layer building runtime??

I believe the AMD format is significantly different from regular javascripts and could very well be a candidate to be implemented as a separate maven module. What do you think?

[alexo]

• Could you define what do you understand by layer? It is important to have a common understanding of basic concepts.
• How a "build profile" looks like? Are there any similarities with model concept used in wro4j?
• My initial thoughts about ResourceModuleExtractor is that it is not necessarily aware about the context it is used in. It just parse the js and extracts some details, like: moduleId, depedencies. It is very similar to Processor concept, except it produce a POJO instead of js or css resource. Do you see it differently?
• I'm not sure I understand what do you mean by "HTML templates to be built in." which should be retrieved from ResourceModelExtractor.
• When being used as a filter, the AMD feature should be capable to cache processed result, otherwise it will be too expensive to repeat the same logic for each request.

About the two parts you mentioned:

• Deliver optimized AMD-resources
  • If no layering or analyzis of dependencies is performed, then postProcessing doesn't make any sense. Since you can process only requested resources. What should be merged in that case?
• Build and deliver optimized AMD-layers
  • Agree with most of your points.
  • How is exclusion performed? (is this something available by default in AMD?). How exclusion is performed from layers?
  • Runtime building can be performant only when used with caching.

It does make sense to create a separate maven module, but I don't have a clear idea about the flow.

[ivarconr]

Could you define what do you understand by layer? It is important to have a common understanding of basic concepts. A "layer" is basically a set of related script bundled together. It can be compared to a wro-group.

How a "build profile" looks like? Are there any similarities with model concept used in wro4j? I might be to hung up with the dojo way of doing this. r.js has been the "norm" on how to combine and optimize amd-resources. They have an example-build profile: https://github.com/jrburke/r.js/blob/master/build/example.build.js. This does not say that we have to do it exacly the same way, but we should look at it and find out whether this is how we think it should be done. It might be that a lot of the configuration done there is redundant under the wro4j platform.

My initial thoughts about ResourceModuleExtractor is that it is not necessarily aware about the context it is used in. It just parse the js and extracts some details, like: moduleId, depedencies. It is very similar to Processor concept, except it produce a POJO instead of js or css resource. Do you see it differently? I totally agree, I just did not understand the first description.

I'm not sure I understand what do you mean by "HTML templates to be built in." which should be retrieved from ResourceModelExtractor. Amd modules can also depend upon HTML-templates and CSS resources (implemented as AMD-plugins and not directly part of the AMD spec). In dojo a widget which requires a HTML-template looks like:

define(["dojo/_base/declare","dijit/_WidgetBase", "dijit/_TemplatedMixin", "dojo/text!./AuthorWidget/templates/AuthorWidget.html"],
  function(declare, WidgetBase, TemplatedMixin, template){
    return declare([WidgetBase, TemplatedMixin], {
    });
});

The dojo-custom-builder allows for these "html" templates to be inserted in to the javascript as string-objects. This can be harder to support in a "general" AMD support, as it is not part of the official spec (correct me if I am worng).

AMD-layers: How is exclusion performed? This is how dojo does it for their AMD build tool. I don't think there is an actual spec for layer-configuration (correct me if I am wrong). I comes as very handy when the project consists of a large code base and you have some common-javascript files used a lot of places. These can be grouped together in a core-layer(s) and excluded from other layers, making the other layers smaller. To me this is an essential feature for a AMD-building system.

[ivarconr]

After thinking a bit about the build-profile a lot of what it defines is redundant for wor4j, as a lot of the information is unnecessary. But some details are needed such as:

• where to find extenal libs (example jquery, dojo, ..) (namespace to folder mappings)
• should we only look at "define" blocks, or should we also look for require blocks inside the script?
• which files to exclude, some projects (e.g dojo) keeps the tests together with the source files. These files you really don't want to process.

[alexo]

I've pushed a branch called issue527 (should I rename it amd for simplicity?). This is where the amd support implementation will go.

I'm thinking about adding new module: wro4j-amd which would depend on wro4j-core only. It would be nice if we could start with defining interfaces (skeleton). Probably I'll have a better understanding once we progress on development.

[ivarconr]

cool. I would prefer amd for simplicity.

[ivarconr]

Circular dependencies is also something we need to find a solution for

[alexo]

Circular dependencies is something which should be resolved with DirectedGraph. I mentioned earlier that the dependencies discovered by ResourceModuleExtractor should produce a graph used to identify the resources which should be bundled.

Btw, branch amd created. It contains the wro4j-amd module.

[ivarconr]

Thanks for reminding me.

Btw I do believe the order of which the dependencies are merged in a layer is insignificant.

[alexo]

I think keeping the order defined by user is the safest approach. At least if something goes wrong, it will be possible to change the order.

Regarding the "namespace to folder mappings". I would name it "namespace to uri mappings", since uri can be anything (external url, servlet context relative resource, classpath resource, etc...). Probably this mapping should be also configurable. Therefore defining a factory for building those mappings and providing a default implementation (property file) should work at this stage.

[ivarconr]

uri is indeed more appropriate

[AGresvig]

Any new developments on this? We are using AMD components with RequireJS on our project. Does anyone have any examples using wro4j with AMD components, or is it not yet possible?

[alexo]

There was no progress on integrating AMD with wro4j. Nevertheless, I would be interested in your use-case and how would you like to use both...

[AGresvig]

Thanks for the quick reply.

Well, the use-case is really just that we are using AMD modules for our BackboneJS-based front-end (SPA) and are looking for a way for maven to handle resource optimization (minifying & optimizing JS and CSS files). So far the only option we've found is http://github.com/mcheely/requirejs-maven-plugin but it doesn't completely fit our needs.

I like the concepts wro4j is based on, and the fact that it takes care of both JS and CSS, but unfortnunately we cannot use the JS-part without AMD support. So it looks like we'll be using two plugins for resource optimization..

Our POMs are getting heavy..

[alexo]

Integrating AMD with wro4j isn't trivial, simply because both are doing quite similar job. wro4j approach is declare all resources and its dependencies inside the model, while require.js declare dependencies inside resources and builds dependency tree based using the requested resource as starting point.

The solution I was thinking of, is to create a filter (AmdFilter) which would build WroModel dynamically adding dependencies on the fly, the same way require.js does. But in order to do that, there should be a mechanism to parse the AMD aware js and extract correctly the dependencies and map them to valid resources. This parsing mechanism is the most challenging task and if it would be implemented using Rhino, it would be to slow to worth the effort... I'm open to discuss these aspects with anybody interested in share ideas or contribution.

Thanks, Alex

[AGresvig]

I understand it is a complex task. Thanks for the insight.

[priand]

Alex, you might want to look at https://github.com/OpenNTF/JavascriptAggregator, developed by my fellow colleagues @ibm. It is very complete (handles has(), i18n, template inlining, ...). The dependencies are managed by dynamically injecting the whole list of dependencies (direct and indirect) into every JS file. Then, a custom client side loader is able to ask for multiple files in one single request. For example, if C requires B that requires A, then a statement like require(['C'],...) is in a JS file, is transformed by the server into require(['A','B','C'],...). The loader then asks for A+B+C in one single request. If 'A', for example, was already loaded, then it asks for B & C only. My team is currently working at making this aggregator easier to use with less dependencies, as it requires OSGi right now. I'm wondering if it would be a good idea to work together and get the interesting pieces of JAGGR right into wro4j.

[alexo]

@priand I'm more than interested to integrate JAGGR into wro4j. I'll take a look into JAGGR implementation and I'm open to discuss any suggestion you have.

Thanks, Alex

[ivarconr]

it would be fantastic if wor4j would be able to support AMD with JAGGR.

[priand]

Excellent. Let me come back to guys in a few. I need to better understand how wro4j works internally so we can think about an integration plan.

[nistvan86]

Any news on this guys?