Asar 2.0

[Horrowind]

trafficstars

You mentioned in #58 the possibility of a rewrite ("if that ever happens"). We don't need to do this right now, since I think there can be many improvements without a rewrite, but since you brought it up, maybe we could at least start thinking about, what we would want in this iteration, what we would want from the rewrite and when we would start with this.

A code rewrite could offer mostly architectural benefits. There are many things which are deeply integrated into the code so that we can not change them now:

• Restructuring into Lexing - Preprocessing - Parsing - Execution (- maybe Optimization) - Code Generation. This is not trivial to do while trying to achieve backwards compatibility, but I think it is possible. This would clean up the use of string functions all over the place, would allow math operations to contain spaces and removes much of the interconnection of the code in all places.
• Removal global state. Asar has too much of it.
• Clean platform abstraction.
• Code quality in general.
• Rethink freespaces / mappings / optimizations / bank crossing. Most of these seem to be adhoc and solve the usual case, but fail in unexpected ways when doing things differently.
• It is more and more apparent (to me), that having only a double type is insufficient. We really want to have at least a string type. Maybe we also need more?
• I proposed an ASM builder in #23. This would essentially be the librification of the code generation part.
• (Speed. This is something only I am interested in, but hey).

What would we still might want to do in this iteration:

• Bug fixes and general maintenance
• Memory files #12
• Include guards #14
• Better Warnings? #13 (should be easier in a rewrite)
• UTF-8 support? #31 #73 (should be most certainly easier in a rewrite)
• Module support?
• Slowly refactor the code until it is more manageable?
• Many things I forgot or we did not thought about yet (#76 - #99 :) )

It looks to me from this, that we should at least discuss, if / when we would want to do this. Some thoughts:

• We are three developers at the time, which is rare for a SMW tool project.
• We can leverage some work I did for nuts so we could get done with lexing and parsing faster. My code is written in C, so we might need to convert it to C++ if that is what we want the recreation to be written in.
• It is still an undertaking.
• We could start with this right now in a separate branch, after 1.60, later, or never. (It is totally fine if we decide to not do this, but we should at least talk about it).
• We will need some planning on freespaces / mappings / optimizations / bank crossing and similar features. They are tricky to get right.
• How much we want to be compatible in general?

[RPGHacker]

You basically summed up most of the things I would want to achieve with a rewrite, if one was to happen. Basically, my main goals for a rewrite would be to make Asar cleaner, easier to maintain, more consistent and more reliable. I would also try to accomplish this via better structuring. There are a lot of features in Asar that could (and should) be implemented via a preprocessor, such as macros, defines and some other minor things. I would also like to have some kind of an abstract syntax tree of some sort to have a more consistent (and easily extendible) syntax. This alone should already improve the code considerably. And yeah, I would also like to get rid of global state as much as possible, possibly by wrapping everything into a class/object.

A nice benefit of a new Asar with a syntax-tree would be that if we exposed this syntax via the lib API, it would be easie to write tools with proper syntax highlighting for Asar. Currently, it's all rather nasty. All of the syntax highlighting stuff I did for Asar so far has some major shortcomings.

As for compatibility, I think it would be important to support all features that are already in Asar or are going to be in Asar (aside from deprecated features). At the very least, a new assembler should be compatible with most to all resources hosted on SMWC. I don't think we would literally have to add everything from current Asar to a new Asar, though. I think in the simplest case, we could get away by just distributing the new Asar with the old Asar and having the new one forward patches to the old one as needed (it could probably use @asar 2.00 or something like that as an indicator for which patches should run through the new Asar). Anyways, I think compatibility is important, because without it, people probably won't use a rewritten tool, thus making it rather pointless.

As for whether this can/should happen, from my side that's of couse quite problematic, because while I would be interested, I simply lack the time for a full rewrite (and this isn't likely going to change in the foreseeable future), I mean I barely manage to keep up with the current Asar version and don't get to do much other stuff besides it. So basically, at this point in time, the only way I could be involved would be if someone else did it and I helped out a bit here and there.

If we were to do it, yeah, either a new branch or a new repository alltogether should be used.

Module support?

I guess current Asar already kinda supports this via stdincludes.txt and stddefines.txt. It might not be what most people think about when hearing "module support", but considering the majority of Asar's user base, it seems like a reasonably simple way of achieving something very similar to modules. Basically, those text files allow you to just unpack the contents of a ZIP file into the Asar directory to have it automatically add some include paths/defines to all patches.

[trillllian]

Another thing i noticed (or rather, that Alcaro pointed out to me) is that Asar stores the current freespace number in the highest byte of the current snes position. This enforces a theoretical limit of 255 freespaces. In practice the number is 125 though since Alcaro thought that would be enough for every use case. Also Alcaro suggested the name "Quasar" for a asar rewrite (before i even told him about this thread). Maybe we could make it Asar 2.0, codename "Quasar".

[RPGHacker]

As a full rewrite, it could even get a new name entirely (just like how Asar was originally based on xkas, but got an entirely new name). So there are many options, really. Me being me, I would probably propose a name like A.R.S.E. (obviously an acronym for "Asar's Rather Suboptimal, Eh?"). However, a name is, of course, something for way down the line. If this project was to happen, I wouldn't mind just going with asar2 as a repository name and only finalizing the name once the project actually got far enough.

[ExE-Boss]

I’ve begun work on Rsar, which I intend to be the successor to Asar.

(The name Rsar was made by combining the .rs extension with Asar’s name)

[RPGHacker]

I haven't worked with Rust yet, but I've always been interested, so maybe it's a good opporunity to take a look.

Do you have documented your goals/directions for Rsar somewhere? Might be a good idea to be on a common demoninator (or at least know what we're aiming for) before anyone starts doing their thing.

[ExE-Boss]

I’d like to add the following to the above plan:

• Allowing assert to be used with labels (e.g. assert (Label1_end-Label1) == 256), this could be used for cases like in excharactertilemap.asm from the 32×32 Player Tilemap Patch.
• Adding inccfg (name subject to change), which would be evaluated only once per file (i.e. if another incsrc file or inccfg calls inccfg on a file which has already been included, then inccfg won’t get evaluated again), and would only be permitted to have defines, functions and macros.
• Drop xkas support (#6), as no-one is really using it now because new xkas based patches aren’t accepted on SMW Central anymore.

Also, Rust has some libraries for implementing parsers and assemblers (like nom , which someone used to make a 6502 assembler for their NES emulator).

[RPGHacker]

The second one should already be covered by inlcudeonce which was recently added, I think. In this case, you would basically just put it in the included file and then to incsrc on it.

Label math would be nice. I don't know if it's accomplishable. I think Asar forbids it because the position of labels can change between passes, or maybe it's because the position of labels isn't known until the final pass. In any case it would be kinda tricky to make it work. I think p4plus2's modified Asar version allows it, but it may lead to some problems.

Anyways, I haven't mentioned this previously because I kinda forgot about it, but one thing I'd like for a new assembler would be an object-oriented design. The assembler object should clean up its own mess (to avoid the reset problematic from the current Asar), and it should be possible to create assembler objects on different threads and assemble multiple files at the same time (not too useful right now, especially not for just patches, but should we ever implement a linker it could be quite useful for homebrew development and speed up assembly drastically). The object would also serve as the assembler's C++ interface (you would just have to include the header of this one object in a client application). Well, I suppose if we do use Rust, we will need a binding, anyways. Does Rust support object orientation?

[Alcaro]

Asar 2.0 already exists. Please don't recycle version numbers.

I'm fine with Asar 3.0, Asar 2.1 (if it has the tagline 'Now tested'), Rsar, ARSE, Quasar, and whatever else you can think of.

[RPGHacker]

We will just call it Asar' 2.0. People will totally notice the subtle difference and not confuse the two versions!

[ExE-Boss]

Does Rust support object orientation?

Essentially, yes.

[RPGHacker]

I'm assuming it's also easy to make dynamic libs from Rust code with bindings for other languages, like C?

[ExE-Boss]

Yes, that was one of the design goals when Rust was made (since its original goal was to re-implement parts of Mozilla Firefox, which is mainly implemented in C/C++), Chapter 19.1 of The Rust Programming Language Book, Second Edition explains how to do this.

In fact, it’s possible to use Rust on iOS, Android and even on the web.

[RPGHacker]

Another thing to put on my wishlist just crossed my mind: I don't quite like how the current version of Asar handles user interaction. It's certainly important to have for the people who don't like dealing with command lines, but how it's implemented is kinda hacky and people who use it also can't make use of any extended command line parameters since the interface is too simple for that.

Proposal: we should remove this user interaction from the assembler entirely and move it into a separate application to be distributed with the assembler. This second application would then serve as the interface for users who don't want to use the command line. It wouldn't have to be very complex. In the simplest case, it would be just another console application routing some parameters to the assembler, though ideally it would be a GUI application. I know that GUI applications don't have the best standing in the SMW Hacking community, but that's mainly because they're rarely maintained and are often somewhat complex. I don't expect an Asar GUI application to ever become very complex, though, since it would still just route parameters to the assembler and, at most, save some settings to an INI file, but nothing more complex than that. However, it would be more intuitive and practical than even the current way of user interaction with Asar, since we could make use of stuff like file navigators etc., so no more typing or copy-pasting paths.

I think my ideal solution here would be very similar to the GUI application of CMake: a simple list of properties the user could configure. It could have file navigators for input file and output file (or anything else that asks for file paths), check boxes for stuff like --no-title-check and -werror and drop-down-menus for enum-based parameters like --symbols and --pause-mode. Of course drag & drop should also still work. A solution like this should, essentially, make it quite easy to maintain the application, and compatibility would be preserved as long as command line parameters of the assembler itself weren't renamed or removed. I would probably go with Qt for this, but technically everything goes (not sure if Rust currently has anything for GUI applications).

[RPGHacker]

Another item for my Asar 2.0 wishlist: getting rid of the DLL interface entirely. The more I work with Asar, the more apparent it becomes that it's little more than a minor convenience for tool coders, while being a huge inconvenience to Asar coders (since it has to be actively maintained, along with tests and bindings, not to mention the DLL reset problematic). Tools that use Asar should just be running the executable in a new process and parse its output. That's what all toolchains that I'm aware of do. It might be slightly slower than using DLLs, but I think for all of our intents and purposes, it doesn't really matter. And when it does matter, there are likely better ways of optimizing performance, anyways.

EDIT: Plus, if a new assembler is written properly and cleanly with a nice interface, it should be easily usable as a static lib without needing any particular bindings. (At least for tools writtein in the same language)

[ExE-Boss]

Also, I’d like to improve the command line interface by adding some Unix-like options and the ability to compile a patch from stdin, thus making the following possible:

patch-emitting-command | rsar --file rom.sfc

I’d also like to remove the internal stderr to stdout redirection when the file is renamed to xkas and if anyone relies on this for whatever reason, they can always do:

rsar patch.asm rom.sfc 2>&1

Also, add the --help command line option, aliased to -h and -?, which would print the command usage to stdout.

[RPGHacker]

Yeah, we already marked the latter as deprecated, anyways, so it's going to be removed at some point.