working data location question

[dfoxvt]

We are currently creating a leader following setup in azure K8s and have some questions about tile38 working data location. The data folder has 3 files: config queue.db appendonly.aof

Could you tell me what these files are used for and if those form part of the working set. At present the appendonly is growing huge > 1GB and since these are stored on a storage account share in azure and the tile38 is the docker image running from a container would this have an impact and what happens if the container is rebuilt?

Looking at the core go file it mentions the below params so it does sound like these form the working set / config

--appendonly yes/no : AOF persistence (default: yes) --appendfilename path : AOF path (default: data/appendonly.aof) --queuefilename path : Event queue path (default:data/queue.db)

[iwpnd]

Tile38 is somewhat of a snowflake in kubernetes from my experience.

To your questions: The aof file has two purposes.

1. every command send to the leader is appended there, so when the leader goes down, and comes back up, it is used to restore state
2. every follower of the leader gets its state from the file.

When the container is rebuild and the aof stays with your volume, the leader will read it again to get back to the state before rebuilding. The aof file increases in size constantly. We run an cronjob executing an aofshrink on a regular basis to keep it manageable.

The config can be used to configure your leader or your followers on startup. Say you want to start a pod as a follower of a particular leader, you can do this in a config. An example can be found here for docker compose.

The queue.db is an embedded buntdb that is used to queue hooks if I'm not mistaken.

[dfoxvt]

Thanks Ben, That will help and brings me onto a another question: What happens if the leader is down for a while.... our idea is to sit tile38 behind an azure load balancer so that every leader / following gets a hit if this is an option assuming a follower can respond to activity? Can you have more than one leader ? Active/passive cluster

[iwpnd]

What happens if the leader is down for a while....

If the leader goes down, your follower can stay up and serve stale data until the leader comes back up.

Can you have more than one leader ?

You can have more than one leader, but you have to take care of how you keep all of them synced on your own - Tile38 does not support leader replication unfortunately.

• You can shard the world, and let every leader take care of one part of the world/country.
• You can have Redis sentinel elect a new leader if the old one goes down.
• You can have Apache Kafka in between with a sink that receives all the updates for the leader. If the leader goes down, requests land in the a topic, once the leader is back up, it receives the events it has missed. Lots of possibilities here, none that have been properly documented anywhere unfortunately.

Also be aware of how you deploy Tile38. If you deploy it along your application, all your instances are down for the period they catch up from the aoffile and usually the aoffile is stale by the time the Tile38 instances are back up. Told you, it's a snowflake ❄️ 😄

our idea is to sit tile38 behind an azure load balancer so that every leader / following gets a hit if this is an option assuming a follower can respond to activity?

Put your followers behind a load balancer. Your leader take the writes, then let your followers scale horizontally and take all the reads.

[dfoxvt]

Thanks Ben,

That makes sense. :)

[mamurev]

Is there any repo/example showing the sharding the world into parts and assigning them to the leaders?