[Tiled] Content exporter

[drakbar]

So I stumbled across this article , and I ended that journey here at this repository. Awesome article, blog, tutorial or whatever that piece is classified as. It was very well written and it also had excellent pictures.

After skimming through the code for the Tiled sections, I noticed some methods like AddTileset(TiledMapTileset tileset) AddLayer(TiledMapLayer layer)

I assume these are for the content loader to add the assembled tilesets and layers into the TiledMap. However, I maybe wrong. Anyways, I would like to build a tool that creates the tmx file from the TiledMap object, and before people say why not use Tiled, I would like to explain.

I am working on a system that uses Procedural Generation. The levels will be built using a pretty bare tmx files, mostly just a TileSet and Map properties(i.e. MaxRoomNumbers, MinRoomNumbers, MaxRoomSize, etc..). I will use these values to populate the map. Finally, I would like to create a "content exporter" that takes the map and saves it out to the disk as a tmx file.

The main reason for saving the map, is so you can play that level again if you happen to get interrupted for any reason, without having to play parts of a level over etc. The classes you have is more than enough to create the Tmx file without a hitch. The problem I foresee is building the .xnb file after the tmx file has been saved out. Is there a way to build those from inside a project, via programmatically? Because if there is, I figure as long as you save it to the right directory and use a wild card inside project file, it can load up using the good 'ole Content Pipeline.

I am asking here because ya'll are pretty savvy and I feel like I can get legit feedback. Awesome job on the extended framework!

[craftworkgames]

Interesting idea. I think it could work.

As you say, there might be some challenges dealing with the Content Pipeline but I believe it'd be possible without too many headaches. The way I understand it, the Content Pipeline tool takes each content file (in this case it's a .TMX file) and runs it through a pipeline (surprise!) of classes to transform it into a binary XNB file.

Essentially the process works like this:

• The MGCB.exe takes each bit of content and looks for the importer specified in the Content.mgcb file.
• The importer is a C# class that reads the TMX file and prepares it for the processor.
• The processor takes the output of the importer and gets it ready for the writer.
• The writer is the thing that takes the processed data and outputs it to the XNB file.
• When you call Content.Load<T> in your game it looks for a matching reader to load the XNB file.
• The reader is the thing that reads in the XNB file and gets it ready to use in your game.

Note: One thing to keep in mind with this process is that the importer, processor and writer are typically not multi-platform code. They will usually only work on desktop machines like Windows or Linux.

However, with that in mind, it might be possible to bypass the MGCB.exe and process the TMX file through the same set of classes described above. In theory, what you'd end up with is an XNB file just like the one output by the Content Pipeline. Depending on what else happens in this process?

Even if you can't do it that way, there still might be a sensible way to get the data processed into some format readable by the reader. Or maybe you could just hijack MGCB.exe and process it through that as part of your tool.

Now, so far, we've only been talking about the TMX file. I assume that's all you'd need to worry about but I probably should at least mention that you might need to consider how textures are dealt with. The way they are processed through the Pipeline is outside the realm of Extended. It's probably best (at least at first) to limit your tool to textures that are already processed. Maybe.

Anyways, I hope that helps. I'm interested to see where this goes.

[lithiumtoast]

I am working on a system that uses Procedural Generation. The levels will be built using a pretty bare tmx files, mostly just a TileSet and Map properties(i.e. MaxRoomNumbers, MinRoomNumbers, MaxRoomSize, etc..). I will use these values to populate the map. Finally, I would like to create a "content exporter" that takes the map and saves it out to the disk as a tmx file.

Note that the current Tiled code was written with immutability in mind. So changing some values such as the render order of tiles, or the image index of tiles itself, or even deleting or adding new tiles would of required changing the underlying models (geometry) for rendering at runtime. By preventing these changes, games that do not change the map at runtime get the best performance by having the maps processed once at build time and written in a format that is optimized for loading at runtime.

So do you want your game to change the geometry and the meta of the map at runtime? If so, what kind of changes? Re-sizing the map? Adding / removing tiles from layers? Adding / removing tilesets? The requirements matter. We can move forward on how to change the current code once we have the exact requirements.

The main reason for saving the map, is so you can play that level again if you happen to get interrupted for any reason, without having to play parts of a level over etc.

This can be resolved by having a meta-data file serialized and deserialized at runtime serving as the level progress. When a map is loaded in as a clean slate, things can be restored to a certain state by the contents of deserialized meta-data file.

The problem I foresee is building the .xnb file after the tmx file has been saved out. Is there a way to build those from inside a project, via programmatically?

Only where the content pipeline is available; Windows, Mac, and Linux. And even then it would be hacky.

[drakbar]

@craftworkgames

Note: One thing to keep in mind with this process is that the importer, processor and writer are typically not multi-platform code. They will usually only work on desktop machines like Windows or Linux.

This should not be a issue for me since I am not developing for mobile or console, but I see, from a general portability perspective, that this is important fact.

However, with that in mind, it might be possible to bypass the MGCB.exe and process the TMX file through the same set of classes described above. In theory, what you'd end up with is an XNB file just like the one output by the Content Pipeline.

I seemed to have exposed my ignorance of what is really happening under the hood. I thought that these classes were for reading, processing, and writing from the .xnb, and not the .tmx (facepalm). I swear that I read the article, but I was tired at the time, and now it is all soaking in.

It's probably best (at least at first) to limit your tool to textures that are already processed.

I envisioned that this would be the case, in fact I cannot imagine what would be the utility of something different. Custom assets, modding?

---

@LithiumToast

When I first conceptualized how all the parts would work together I wasn't aware of all of the features that accompanied the Extended.Tiled (I am still not aware of everything). A prime example, the fact that there is a map renderer. I was originally under the assumption that the TiledMap was just a handle to the data and it was up to me to take it from there.

However, to answer your questions let me present to you the working idea the best that I can. This is as much for you as it is for me. So, if I reiterate stuff, it's for me to hash out what is rattling around in the old noggin.

---

Here is a diagram of the original idea.

It starts with loading the "template" file. It contains the information that is common to all maps of this particular level. This file contains information that will never change( e.g. total area in tiles such as 25x25, tileset(s), etc. ). I suppose that it will eventually include things such as number of tile, object, collision, and image layers. What is not contained in this file is data such as tile id =" 0". This will be generated in the procedural generation algorithm.

Once the the TiledMap object is returned from Content.Load<T> the map properties are used to create level. This is where the procedural generation algorithm will take hold and create the level. Note: the point from which the map properties are handed to the PG algorithm is the DMZ between MonoGame.Extended.Tiled and the game.

Once the level is created the game will enter the PlayState.

Once the player decides to save the game, the level data handed to the "Content Exporter" part of the program. This should output a .tmx file to the disk, which is then input into the faux MGCB.exe. This should have an .xnb file as an output which will be saved to the disk.

This design doesn't take into account that the extended library is capable of doing some of the heavy lifting.

---

Here is a diagram of the revised idea. It moves the .tmx and .xnb file creation to before the game actually hits the PlayState so that the MonoGame.Graphics classes can be used to render.

So do you want your game to change the geometry and the meta of the map at runtime? If so, what kind of changes? Re-sizing the map? Adding / removing tiles from layers? Adding / removing tilesets?

Well the only thing I can think of, (which is possible that it already does this), is setting values like if tiles are visible, turning collision on/off. PlayState logic stuff, but as of right now I am just trying to assign a tile set id to the tiles in the tile layers.

My plan way to generate a .tmx file by writing it(I may be wrong but I thought all a .tmx file has is strings?) and then feed it back into the content pipeline.

However, after looking at the MonoGame.Extended.Content.Pipeline.Tiled classes I figure I should be able to create the content object and hand it to the TiledMapWriter.

This can be resolved by having a meta-data file serialized and deserialized at runtime serving as the level progress. When a map is loaded in as a clean slate, things can be restored to a certain state by the contents of deserialized meta-data file.

Ok, that does make sense, it means that the diagram should look like so?

And this would result in three files per level.

At this point this post is very lengthy and so I will end it with that. Thank you guys for the feedback!

[lithiumtoast]

Kudos on the images. I'm a big fan of images to express communication in design phase; walls of text are daunting.

A prime example, the fact that there is a map renderer. I was originally under the assumption that the TiledMap was just a handle to the data and it was up to me to take it from there.

Getting the .tmx loaded and ready to render is on us. It's up to you to parse the loaded .xnb version of the .tmx file to create the objects / entities in the scene of your game. Some people like to use Tiled object groups to attach that data to the map and then parse it form there; others just keep the Tiled logic specific to tiles and come up with a different way to handle loading and parsing objects / entities.

It starts with loading the "template" file. It contains the information that is common to all maps of this particular level. This file contains information that will never change( e.g. total area in tiles such as 25x25, tileset(s), etc. ). I suppose that it will eventually include things such as number of tile, object, collision, and image layers. What is not contained in this file is data such as tile id =" 0". This will be generated in the procedural generation algorithm.

So the only requirement is that the "mapped" image for a tile (from an immutable tileset) can change at runtime? Everything else can stay the same? If so, what is the context of these changes to tiles at runtime? How often does it need to happen? For example do these changes happen every frame? Or do the changes to tiles happen less frequently? Also is it acceptable to have a small delay between applying the changes to the tiles, perhaps requiring a short loading screen for the player for example? (I'm not suggesting there would be a delay, just that if there is one it would be handled correctly.)

By the way, if you want simply tile collision like classic Game Boy Pokemon or Final Fantasy it's simply enough just to have a 2D multi-dimensional array, a.k.a. matrix, that has enough cells for each tile location, (x, y). Each value for a cell in the matrix would represent the state of that tile such as if it's impassable, free, or hinders movement, etc. Then when the player requests to move using some form of peripheral (keyboard, mouse, etc) it's easy enough just to check the state of the next tile the player is about to move into by checking the matrix at the correct cell. If the tile is free apply some basic physics or simply just move the player position's directly so the player occupies the tile. By keeping the matrix update-to-date as the player and other movable things move around such as NPCs, basic tile collision detection can be achieved.

[drakbar]

UPDATE: So, I hit a major breakthrough today. I have figured out the most important part of the equation (outside the procedural generation).

The problem I foresee is building the .xnb file after the tmx file has been saved out. Is there a way to build those from inside a project, via programmatically?

Apparently, someone over at the MonoGame camp thought that building .xnb files at run time was a good feature to have(this is all conjecture except for the fact that it's possible!). After reading this thread over at the monogame forums, I found out that compiling .xnb files from inside that game can be achieved by using the PipelineManager object.

The class details can be found here.

The code implementation is very easy, however, I would like to know what is the best way to feed these arguments to these functions.

PipelineManager pipelineManager = new PipelineManager(projectDir, outputDir, intermediateDir); pipelineManager.AddAssembly(projectDir+"/MonoGame.Extended.Content.Pipeline.Tiled.dll"); pipelineManager.BuildContent(projectDir+ // path to file to compile, projectDir+ // output file location);

I have tested these functions including a Content.Load<TiledMap> call and everything works as one would like it too. The only thing now is it to architect an elegant and "correct" way of calling these.

[craftworkgames]

Very interesting.

The only thing now is it to architect an elegant and "correct" way of calling these.

I don't think there is a "correct" way. This kind of thing is likely going to be quite specific to your game. It's also only going to work on desktop platforms since the Pipeline DLL's are not portable. So I'm pleased you got it working but I'm not sure if it fits into MonoGame.Extended.

That said, if you do come up with some kind of generic solution that should be shared with others I'm sure there's a place for it somewhere. I just don't think it belongs in the core library.

[drakbar]

I completely agree with you this is very specific, and doesn't fit criteria of the MonoGame.Extended.

It's also only going to work on desktop platforms since the Pipeline DLL's are not portable.

Yeah with the other platforms there would have to be some other solution. Probably a server / client situation where you generate the maps at the server and then push them out. It would also be a good idea to push a batch of maps instead of generating just one map for each push. Or you could just ship it with X number of map variations.

[drakbar]

Just like libGDX, this will allow modding easily, i don't think puting everything in xnb is helpfull during development, it slow down iteration, maybe for mobiles it make sense for loading, but not for deving imo

I am a student working on a project, and this isn't something that going to be shipped. So I don't consider myself a developer. Having said that, I do recognize the point you made about the time lost during each iteration, and I have taken that into consideration in the design.

The time lost compiling is probably exacerbated by orders of magnitude when coupled with procedural generation. This problem is what pushed me towards a data driven design. Instead of having to recompile the program each time I needed to tweak one of the rules that govern the procedural generation, I could just update the data file and restart. the program. The data file in this case is a .tmx file because it's free, it's well documented, and there is plenty of support for it (MonoGame.Extended being one of them 😃). It may not be the most appropriate choice, but storage space is cheap.

However, it doesn't just stop there I want to take it one step further. I am also using ImGui to create some pretty nice debug tools. Eventually, I will create a tool that will accept changes either through text fields or perhaps sliders (all depends on how fancy I wish to make it), and then a submit button which will regenerate the data file, compile it to its binary form, and reload from the content manager. All of this without having to leave the game. This of course does nothing to save time if the PG algorithm needs to be be changed. If only you could modularize that portion of it.....

[lithiumtoast]

This problem will be solved by not using the content pipeline for Tiled.

[lithiumtoast]

There is a rough idea in my head to use ImGui to create an editor with feature parity to Tiled for an optimized developer workflow as described in the above comments. The idea here is not to use the content pipeline. @prime31 is experimenting with this idea in his own code base; I'll ask him more about it as he goes down this path.