Compiler.exclude + compiler cache bug

[ncannasse]

When using --macro exclude('package') with compiler cache, this will cause problems with inline constructor. First compilation works, but then next one (if the file using the inline constructor changes) will error with Extern constructor could not be inlined

Here's a reproducible example:

class Main {	
	static function main() {
		new h2d.col.Point();
	}	
}

Compile with:

haxe --connect 6666 -js main.js -main Main -lib heaps --macro exclude('h2d')
touch Main.hx
haxe --connect 6666 -js main.js -main Main -lib heaps --macro exclude('h2d')

[Simn]

Wow that's dumb...

I guess if we want to use cl_extern for this we have to delay it until after save_class_state?

[ncannasse]

What I don't understand is that it still works with first compilation... So most likely something is not correctly restored for constructors

[Simn]

The problem is that we run onGenerate before saving the class state:

	let t = filter_timer detail_times ["callbacks"] in
	List.iter (fun f -> f()) (List.rev com.callbacks.before_dce); (* macros onGenerate etc. *)
	t();
	let t = filter_timer detail_times ["save state"] in
	List.iter (save_class_state tctx) new_types;
	t();

Basically, the main issue here is that "onGenerate" has a bad name. We still run a whole bunch of stuff after this point, which is why internally I renamed it to before_dce.

Here's an overview of our compilation process. I never documented this because we kept shifting things around, but we should add a manual page for this when we release Haxe 4:

Typing

• Parse CLI arguments
• Run initialization macros
• Callback: after_init_macros
• Type
• Callback: after_typing
• Finalize (sort types and determine main-function)

Expression filters part 1

• VarLazifier (deals with something extractor-specific, could perhaps be moved to the pattern matcher)
• Local variable initialization checks
• Abstract casts (handles calls on abstracts as well as some @:multiType-specific stuff)
• Constructor inlining
• Expression reduction (the general expression optimization pass)
• Variable capturing for closures
• (Java/C#/Js) Try/catch wrapping

Pre-analyzer type filters

• (C#) Event handler (deals with @:event metadata)
• (Java/C#) Default argument Null<T> wrapping

Analyzer

• Run everything through the transformer, optimize if -D analyzer-optimize is set

Expression filters part 2

• Sanitize code
• Add final return (on platforms that need it)
• Rename local variables
• Switch-break marker in loops

Before DCE

• Callback: before_dce
• Save class state
• Remove @:generic base classes
• Remove extern fields
• Update dependency for compilation cache
• Remove implements for interfaces that have @:remove

DCE

After DCE

• Check private type paths (checks if there are no naming conflicts with the _ renaming)
• Apply @:native to types and fields
• Add RTTI
• (not Java/C#) Move member field initializations to the constructor
• (not HL) Generate __meta__ field where required
• Check for Void-typed fields
• (C++) Optimize interface relationships
• Commit features of all remaining modules
• Check reserved type paths for some targets
• (C#) Native interface property handling
• (Js) Call-stack injection

Generation (if --no-output is not set)

• (Flash/C++/HL) Override fixing (rewrites variance overrides to base-types + casts)
• Generate dump files if -D dump is set
• Generate dependency dump files if -D dump-dependencies is set
• Create output directory if necessary
• Run target generator
• Callback: after_generation

---

Some things to infer from that:

• The last point at which we can allow additional typing is the after_typing callback. Anything later would not go through the full filter process.
• The before_dce callback is run before the class state is saved and its effects thus persist on the compilation server. I'm thinking that we could add a boolean argument to onGenerate which is interpreted as beforeSave = true, and something like exclude would use false here. The main problem is that this is impossible to document for people who don't develop compilers.
• A true onGenerate would execute just before we run the target generators. However, at this point we have to think about what a callback like this is supposed to do. It could still mutate some generator-specific state, but on the other hand this might lead to annoying inconsistencies because filters have already run with the previous state...

[ncannasse]

I'm thinking that we could add a boolean argument to onGenerate which is interpreted as beforeSave = true,

I agree, let's do that for now and name it persistent, and we can document it this way: persistent : Bool changes made within this callback are persistent wrt compilation server, see @:persistent.

[Simn]

Done

I took the opportunity to clean up our callback structure a bit. Keeping this open so I don't forget to put this documentation somewhere.

[kLabz]

I took the opportunity to clean up our callback structure a bit. Keeping this open so I don't forget to put this documentation somewhere.

Guess this will need an update and be added somewhere for Haxe 5? :sweat_smile:

[Simn]

We can make a wiki page or something.