Atom ECS - ultra fast

[sschmid]

Hi guys! I just wanted to share my latest experiments. This post is just to share my thoughts and have a general conversation about ECS. I might or might not continue working on this.

Last night I wrote a new ECS proof of concept

Meet Atom ECS

But why? 🤦‍♂️

I always had these thoughts:

For ECS we only need data and functions.

Data

For example:

Health : int

We don't need extra components wrapping those data types.

Result:

Actual generated code from Atom ECS

health = new int[entityCount];

--> Raw data arrays without any indirection, contiguous memory. No component layer in between

Functions

For example:

Move()

We don't need extra classes or structs wrapping those functions.

Result:

Actual code from Atom ECS

[EcsSystem(allOf: new[] {Position, Velocity})]
void Move(int entity, ref Vector3 position, ref Vector3 velocity)
{
    SetPosition(entity, position + velocity);
}

--> Raw functions. I mean, look at this function. Can it get any more beautiful? No clutter, no noise, pure information. EcsSystem will generate additional code that calls this function and will take care of iterating efficiently over all affected entities. The pure data will be read and passed in directly from the internal component arrays. SetPosition will directly write data to the array. No GC and very fast 👍

Reactive ECS

New reactive model will drastically increase performance for "reactive systems"

Actual code from Atom ECS

readonly Group _healthGroup = new Group(new[] {Health});

void OnHealth()
{
    foreach (var entity in _healthGroup.Changed)
        if (GetHealth(entity) == 0)
            DestroyEntity(entity);
}

Bonus, I already implemented a highly requested feature ;) You can now choose between added or Changed (aka Updated)

group.Added
group.Changed
group.Remove

Automatic order of execution (planned)

Goal is that AtomECS figures out which functions to call in the correct order, so you don't have to think about that + it could potentially automatically call these functions on different threads in parallel, leveraging the full power of the target device.

This could be achieved by annotation the functions. Everything else will generated for you.

[EcsSystem(allOf: new[] {Health})]
void UpdateHealth(int entity, ref int health)

Tiny

It's still early stages, but this framework is ridiculously tiny and has almost no complexity making it easy for anyone to get started with ECS. Let's see if and how it will evolve.

Fast, fast, fast

Contiguous memory, no component wrappers, pure functions, new reactive backend. This all benefits performance

Left: Entitas (50.000 entities, 1 exe system, 1 reactive system, single thread) Right: Atom ECS (50.000 entities, 1 exe system, 1 reactive system, single thread)

[JesseTG]

How does this compare to Unity's ECS?

[sschmid]

That's what I'm most interested as well :) I will find out soon

[FNGgames]

This sounds really cool. How will you handle array resizing when entity count changes?

[sschmid]

@FNGgames I need to do some performance testing first. Atm you specify the initial size (estimated max entities) but there's also a Resize() method, which you probably can use with different strategies

[optimisez]

Hi @sschmid, how about memory consumption? Seems like it will consume less memory than Entitas a lot.

[Leopotam]

@sschmid

int entity

What about protection from reusing removed entities? Some sort of generations for entities?

Fast, fast, fast

Can you make performance test with this ecs to compare (this repo for reactive support), please?

[SirMetathyst]

@sschmid This looks cool. Will you allow multiple instances of the simulation. EDIT: Also, Will you write a tiny code generator to go with it. i.e. "Atomic" code gen ;)

[SirMetathyst]

In this code

[EcsSystem(allOf: new[] {Position, Velocity})]
void Move(int entity, ref Vector3 position, ref Vector3 velocity)
{
    SetPosition(entity, position + velocity);
}

rather than specifiying Position and Velocity in the attribute could you not read the params of Move and know that you want "position" and "velocity" which then becomes shorter?

[EcsSystem]
void Move(int entity, ref Vector3 position, ref Vector3 velocity)
{
    SetPosition(entity, position + velocity);
}

[sschmid]

@SirMetathyst

Will you allow multiple instances of the simulation

yes!

Will you write a tiny code generator

Hell yes!!! :D

read the params of Move

hattrick :D

I was thinking about this too, because the info in the attribute is redundant. Reading the method signature will be enough I guess

[sschmid]

@Leopotam If I continue I will probably also test it against other ECS. I like the challenge ;)

In the first proof of concept active entity ids are stored in a hashset. I will research other solutions too. I'm happy for any kind of input on that!

[sgf]

you idea is good.

1. I've thinking about how to Fastest Filter the Entitys on recently.

The result is use compressed Bitmap algorithm. As far as I know the fastest implementation -- Roaring Bitmap(https://github.com/RoaringBitmap/RoaringBitmap) ,but its java. https://github.com/Tornhoof/RoaringBitmap .net port,but no update long time.

2. about contiguous memory(Component),i have no good idea now.(It may be impossible to perfect solve)

one way:(It is not fully expect very good, very clear the way.i need more time to think about it) aslo use Roaring Bitmap to store datas:

ALLEID:1,2,3,4,5
EID&DataIndex:[2,1][4,2] --saves by Bitmap
Data:[fixsize-Block][fixsize-Block]
// for get compontant

EID&DataIndex].EID xor ALLEID

[delaneyj]

It the POC available anywhere @sschmid?

[sschmid]

@delaneyj no, but I will keep you updated here

[laughsuggestion]

@sschmid I'm definitely interested in the new smaller syntax and performance improvements. I have a few concerns:

1. I want to make sure I can still specify structs in C# - not use a separate scripting language. Absolutely no interest in using two languages.
2. You mentioned automatic scheduling and multi-threading of systems. I definitely have concerns here; even in current Entitas I have to manage how the systems are ordered to achieve correctness. I'm not sure how you're going to do this for me. Annotations that give me control are OK; although the current way of managing "Features" and system hierarchy's works really well.
3. Not having classes for each system may muddy things up; classes often have references to performance things. For instance, I keep List<SomeComponentNameHere> around when using GetComponentsInChildren so that I can reuse a list instead of allocating new ones. As it stands now, each file has just the thing it needs: the system; its collector and filter; its setup; its cleanup; its initialization; its performance utility objects. Entitas currently has a very clear way of structuring code and projects. Will Atom also have this clarity?

Really looking forward to where this goes; I've been hoping for more from Entitas.

[ghost]

I have a suggestion for Atom. Wouldn't it be a good to have a message system as well for stuff like commands? Where we don't want to store commands/messages as components. Something like this perhaps...

[EcsMessage]
void NewBuildingMessageReceiver(ref NewBuildingMessage msg)
{
    int entity = CreateEntity();
    SetPosition(entity, msg.position);

    var newMsg = new SomeOtherMessage(10,10);
    PublishSomeOtherMessage(ref newMsg);
}

[ghost]

Also, Another suggestion. For unique components you could specify it as "unique" with param attribute. (Also, I assume that since component classes no longer exist, unique components would be as simple as int counter = 0) Something like this perhaps? (Although, now that I think about it, is that unique attribute really needed?):

[EcsSystem]
void IncreaseCounter([Unique] ref int counter)
{
    SetCounter(counter + 1);
}

[laughsuggestion]

I have a suggestion for Atom. Wouldn't it be a good to have a message system as well for stuff like commands? Where we don't want to store commands/messages as components. Something like this perhaps... ...

@T2RKUS what’s the difference between a message and a component in a Messages context? If you create a MessageEntity with a given component you can leverage all of Entitas’/Atom’s features without having a separate messaging system attached.

_{Sent with GitHawk}

[ghost]

@laughsuggestion I Imagine Messages being part of a queue of some sort instead of being a component which would get called somewhere in the game loop (depending on your other systems order). "Messages" shouldn't be a component because it would mean if I have 1000 max entities I'd have an array of 1000 of these new building messages which would be apart of entities when really I don't want them to be. I dont want to keep them around I just want to fire them and repond to it.

It might look like

void ProcessNewBuildingMessages()
{
            for(int c=0; c <= _newBuildingMessageQueue.Count; c++)
            {
                var msg = _newBuildingMessageQueue.Dequeue();
                NewBuildingMessageReceiver(ref msg);
            }
}

[laughsuggestion]

Also, Another suggestion. For unique components you could specify it as "unique" with param attribute. (Also, I assume that since component classes no longer exist, unique components would be as simple as int counter = 0) Something like this perhaps? (Although, now that I think about it, is that unique attribute really needed?): ...

@T2RKUS is it the case that component classes are going away? Or do you mean system classes? The first case would be odd, because if the component is unique in one system but not another what happens?In the latter case you could just use a global variable.

_{Sent with GitHawk}

[ghost]

Re-read the first comment, it states that Component classes would be replaced with code like health = new int[entityCount]; so I imagine unique components would simply become int something and their methods SetXXX(value); instead of SetXX(int entityId, xxx...);

How I understand it is when you type

[EcsSystem]
void UpdateHealth(int entity, ref int health)

the code gen would detect that you want an int health and so generate health = new int[entityCount]; in the entity manager/context so why wouldn't that also generate for unique components as well? If you have multiple functions which use that unique component (and has the same name) it shouldn't generate more than once...

you could infer that it is "unique" because it makes no sense that it would have an entityId as only one would exist... like

[EcsSystem]
void UpdateCounter(ref int counter)

Now I don't know what Atom is like but if you wanted to use a unique component in a system like updating position you could use the generated method GetXXX (however in order to generate that method and component you'd have to specify that somewhere like maybe having [Unique] ref int someXXX or something at the end of that method idk)

[EcsSystem]
void Move(int entity, ref Vector3 position, ref Vector3 velocity)
{
    SetPosition(entity, position + velocity + GetSomeXXX());
}

[laughsuggestion]

The health example is a trivial case but in reality components are generally more complicated that one integer. Even health is usually accompanied by a max value. You could separate all the fields of every component but that would probably be a frustrating way of working. I think we need more concrete clarification here.

_{Sent with GitHawk}

[ghost]

Ok... then use a struct. The point is that a "Component" is the data and before it was wrapped in a class implementing IComponent inside a class entity but now it's an array of some type that you access by it's index (which is the entity) It doesn't need to apart of some a class now.

[laughsuggestion]

@laughsuggestion ...

@T2RKUS your suggestion of having a queue will be backed by an array. This array will resize when it grows. In the case of Unity you don’t want to GC so it wouldn’t ever shrink. You’d be holding on to this memory. Also, I don’t think there’s a max entities size; at least there shouldn’t be. That would be a very brittle way of writing the code. I know he put in an entityCount into the array, but I’m hoping this is pseudocode for something smarter underneath.

_{Sent with GitHawk}

[ghost]

I think you're missing my point about the messages. If I wanted to have a "BuildNewBuildingMessage" I'd have to create a new entity, add that data and then it would get processed by a system but this data is not meaningful to the game/sim. I only want to send a message. The actual implementation doesn't matter right now. I just think that it's better to say create some message struct, and call a method like PublishNewBuildingMessage(msg) it gets queued up somewhere, in someway and processed and this separate from the entity data. If it were a component for entity then I'd have the generated methods for that

Result: Actual generated code from Atom ECS health = new int[entityCount];

Atm you specify the initial size (estimated max entities) but there's also a Resize() method, which you probably can use with different strategies

[laughsuggestion]

Ok... then use a struct. The point is that a "Component" is the data and before it was wrapped in a class implementing IComponent inside a class entity but now it's an array of some type that you access by it's index (which is the entity) It doesn't need to apart of some a class now.

My only point about this is that if we just always use a struct, we continue to have a easy place to put [Unique].

[laughsuggestion]

I'll address these separately:

I think you're missing my point about the messages. If I wanted to have a "BuildNewBuildingMessage" I'd have to create a new entity, add that data and then it would get processed by a system but this data is not meaningful to the game/sim. I only want to send a message. The actual implementation doesn't matter right now. I just think that it's better to say create some message struct, and call a method like PublishNewBuildingMessage(msg) it gets queued up somewhere, in someway and processed and this separate from the entity data. If it were a component for entity then I'd have the generated methods for that

Would this not be the same as an auto-cleaned up component used like:

// Message declared like:
[Cleanup(CleanupModes.Destroy)]
class BuildNewBuildingMsg : IComponent { ... }

// Emission declared like:
context.CreateEntity().AddBuildNewBuildingMessage(...)

// Consumption like
[EcsSystem]
void BuildNewBuildingHandler(ref BuildNewBuildingMsg msg) { ... }

When you say:

I'd have to create a new entity, add that data and then it would get processed by a system but this data is not meaningful to the game/sim. I only want to send a message.

I think I'm getting tripped up here. You'd have to create an entity and attach a component, but this doesn't feel very different from calling "new" with constructor parameters. It's one line of code. Also why are you creating messages that are not meaningful to the game/sim? What would be the point?

Result: Actual generated code from Atom ECS health = new int[entityCount];

Atm you specify the initial size (estimated max entities) but there's also a Resize() method, which you probably can use with different strategies

I hope this changes, because as I said I think it is brittle.

[ghost]

What I mean by "not meaningful to the game/sim" I see the data attached to entities as the world/sim data which would be meaningful but I don't want messages/commands as components on entities because I see them as different, separate things... but yeah... you could add messages/commands to entities and process them that way... It's just that if you have a fixed amount of entities (that entityCount) then your messages take up an entity (when they don't need to be)... And I personally like the fact that you specifiy a max entities count and can Resize it later rather than it be continously increasing tbh

[laughsuggestion]

What I mean by "not meaningful to the game/sim" I see the data attached to entities as the world/sim data which would be meaningful but I don't want messages/commands as components on entities because I see them as different, separate things... but yeah... you could add messages/commands to entities and process them that way... It's just that if you have a fixed amount of entities (that entityCount) then your messages take up an entity (when they don't need to be)... And I personally like the fact that you specifiy a max entities count and can Resize it later rather than it be continously increasing tbh

@T2RKUS so I think we might just disagree on meaning and allowable purpose of entities. Although I’ll concede that if you have maximum I get why you’d want to save your entities. I think a better way to store entity arrays is with fixed sizes in chunks. Then store chunks in a growable list. This would be completely behind the scenes to the user and would probably not be much less performant.

_{Sent with GitHawk}

[akoolenbourke]

What I mean by "not meaningful to the game/sim" I see the data attached to entities as the world/sim data which would be meaningful but I don't want messages/commands as components on entities because I see them as different, separate things... but yeah... you could add messages/commands to entities and process them that way... It's just that if you have a fixed amount of entities (that entityCount) then your messages take up an entity (when they don't need to be)... And I personally like the fact that you specifiy a max entities count and can Resize it later rather than it be continously increasing tbh

Is there a reason Atom wouldn't allow you to use another context for the messages? Wouldn't that work, although that wouldn't supply an ordered queue if that's what you want.

I'd also like to add that one of the biggest problems we have found with Entitas is the 'frame delay' issue. Sure I can order my systems to some degree but sometimes you can't satisfy everyone and you also have problems when you need to sequence. For instance when I was looking at a behaviour tree implementation I would set an ActionState component to "running". A system for the action could recognize that execute, fine, but if that action finished immediately and set it's own ActionState to "success" the tree wouldn't pick that up until next time and so each successive action was at least a frame after the first, which was bad. In the end I had to ride off the events for adding components to entities which felt yuk.

How Atom could solve this I am not sure but maybe someone to specify the system like this

void Move(int entity, ref Vector3 position, ref Vector3 velocity)
{
    SetPosition(entity, position + velocity);
}

Then instead of collecting everything until one single system processes it it would call my system with a single entity that changed. I realise the example above wouldn't be used as you can save the position updates to one entity but what are effectively message handlers might benefit from this.

[laughsuggestion]

I'd also like to add that one of the biggest problems we have found with Entitas is the 'frame delay' issue. Sure I can order my systems to some degree but sometimes you can't satisfy everyone and you also have problems when you need to sequence. For instance when I was looking at a behaviour tree implementation I would set an ActionState component to "running". A system for the action could recognize that execute, fine, but if that action finished immediately and set it's own ActionState to "success" the tree wouldn't pick that up until next time and so each successive action was at least a frame after the first, which was bad. In the end I had to ride off the events for adding components to entities which felt yuk.```

So I've come to the conclusion that anytime I need multi-pass resolution of state, I just need to work with the appropriate data structure for the problem and not force it into ECS. In your case, I suspect that I'd probably use a system(s) that internally maintain and executes against the tree. The system can take entities and their components as inputs to the tree, and then output entities or component changes as results.

ECS could solve this problem if you had a notion of running a Feature/set of systems until there are no unresolved entities... then continue to the next frame. I've not tried this though.

[akoolenbourke]

Yeah I had experimented initially with a completely ECS based behaviour tree. It involved running systems multiple times and in the end I felt it was just utter overkill for what I wanted to I implemented a classic style OOP tree with entities for action state. I'm still not utterly happy and to be honest I could get completely rid of ECS for the actual action state.

[T2RKUS]

@sschmid What's the verdict on this? Do you think there is hope? I feel like this is an improvement over the Entitas model.

[ianeinman]

I wanted to weigh in on this. I would rather see effort put into extending Entitas rather than designing something completely new. I think it would be tough to maintain both and would worry about Entitas falling by the wayside. I'd also be more interested in one system that had multiple ways of doing things, over having to choose between different systems.

I recently switched from using Unity ECS over to Entitas. It was a lot of work to switch but I'm glad I did. Considering Unity's ECS / DOTS is the shiny new object, why did I do this? Well, what it comes down is that raw performance is only part of an equation. Flexibility is also really important. I found the Unity ECS to be really lacking when used in a hybrid model, and it is not even close to ready for what they call a "pure" ECS model (no GameObjects at all). Maybe someday, but not until years down the road.

I would be wary of great performance results from a contrived example. The example of Atom ECS above is a single integer field. Components and the systems that use them can be a lot more complex and the time saved by making access faster is only a small part of the whole equation. This is why I found Unity ECS to be a false hope, you could do simple things fast but it had so many limitations that I needed to work around (which slowed things down), the overall process was it took longer to get stuff done and the performance wasn't anything like their demos.

I think it would be great to have a simpler way of working in the ultra-simple case. Right now, if I have a component that has a single field, like this:

[Game] public struct InterstellarPosition : IComponent { public double3 Position; }

I have to access it entity.interstellarPosition.Position. I wish there was an attribute that I could attach to the class which would allow me to access it as just entity.interstellarPosition and not have to give a name to the member. But it should be an attribute, because the component might have multiple things.

You also talk about the benefits of "Raw data arrays without any indirection, contiguous memory". My personal experience is that most of my components are sparse, only being attached to a small percentage of the entities. I haven't studied how Entitas deals with memory, but the Atom approach sounds like it might be wasteful depending on what the component looks like. So I'd rather see it used only when you add a certain attribute to a component, rather than having a separate ECS system which does everything a separate way.

Lastly, I really like the concept you have here with defining an attribute that allows a method to get called for every entity. Where it might be limiting is when there's also some calculation or other entity you need to get data from for all entities in the loop. For example, I might have a system that needs to calculate the distance between the entity, and the player's current position. I would not want to read the player's current position again for each entity. So you need some way of getting a method called at the start, and also at the end, of the system's entity loop which could act on class member variables.

[davoodkharmanzar]

and where is this Atom ECS repo?

[SirMetathyst]

@davoodkharmanzar It's still private as far as I know. I just ended up writing my own for a project although written in golang, you could do the same too if like the idea. ZincECS is kinda similar to Entitas and Atom. Please look at https://github.com/SirMetathyst/zinc for inspiration or to use :)