Problems with the Data Books for Plugins

[MoroseCorpse]

I have a question about the books in plugin programming. Are there still things being added or are they already complete?

For example, if I use the FullClient.Book.Server.OptionalData.ClientCount then this error occurs:

2020-06-01 19:22:47.0847|FATAL||Core.UnhandledExceptionHandler Critical program failure!
System.NullReferenceException: Object reference not set to an instance of an object.
   at WelcomeBot.WelcomeBot.Initialize() in D:\Users\Moritz\Desktop\Development\WelcomeBot\WelcomeBot\WelcomeBot\WelcomeBot.cs:line 71
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

This is also the same with querycount or similar The data from FullClient.Book.Channels does not really work either

Furthermore, I would like to see isAway added to the FullClient.Book.Clients

[Flakebi]

The bookkeeping has most information but it is not complete. Expanding it should be quite easy, it is declared in Book.toml and MessagesToBook.toml in this repo, so feel free to make pull-requests: https://github.com/ReSpeak/tsdeclarations/ OptionalData, like the name suggests, is optional, so you should not rely on it being there. In this case you have to explicitly request it via a command (so the data can be outdated). I think nobody tried this so far, so this is probably not yet working in an easy way.

The channel information should work (unless you are referring to Channel.OptionalData). For the away state, Client.AwayMessage is either null (= not away) or a string (allowed to be empty), this should work too.

[Bluscream]

Can't you just add some kind of wrapper around these properties so they're automatically requested the first time they're accessed and null and after that return the cached one?

[Flakebi]

When would the cached value be updated? E.g. the ClientCount can quickly get outdated and fetching an outdated value can be surprising for someone who uses that api. I hope that making this explicit reduces the confusion. In addition, getting these data can take an arbitrary amount of time and even fail if the connection times out.

[Bluscream]

Well, afaik book does update whenever the bot or any plugin recieves updated values (either through events or through issuing acommand that returns a value that can be written to the book, right? So that would answer your first question. In the case of clientcount, that should be able to get calculated through other visible values, i once wrote a pytson plugin for that. About the time, yes that can take time, but it does not differ from checking is the prop is null and if yes requesting it yourself, duration-wise

[Bluscream]

if you think the concept would cause too much confusion anyway, you could make a explicit extension method like item.property.CachedOrRequest() similar to .FirstOrDefault()

[Flakebi]

if you think the concept would cause too much confusion anyway, you could make a explicit extension method like item.property.CachedOrRequest() similar to .FirstOrDefault()

That looks like a nice and obvious way to do it!

Well, afaik book does update whenever the bot or any plugin recieves updated values (either through events or through issuing a command that returns a value that can be written to the book, right?

For everything that receives updates, yes, that is already working. Everything else goes into OptionalData exactly for the reason that there are no updates.

In the case of clientcount, that should be able to get calculated through other visible values

You can do that for all visible clients, yes. But I think you can’t compute this value if the client is not subscribed to all channels (or not able to because of missing permissions).

[Bluscream]

https://github.com/Bluscream/pyTSon_plugins/blob/3edfaacc74577b52b3b3509e6367fdcf1511a034/scripts/dontHide/init.py#L51 is what i was refering too, but yeah it requires commands sadly

[MoroseCorpse]

Ok so I understand correctly that they generally advise me against the books because they don't update themselves?

[Flakebi]

The book is completely fine, it is meant to be a convenient interface for everything related to the server state. But there is a difference between things like channels and clients, which are always up-to-date, and OptionalData, which is not requested or updated from the server by default.

1. So, if some information is available in the book and is not in OptionalData, definitely use the book, it is a lot faster, more efficient and less error-prone because no network communication is needed.
2. In other cases, you are currently better of with sending and receiving commands, although of course we appreciate contributions to make the book more usable, as it is planned as the main high-level interface.

[MoroseCorpse]

I think I found another way to get the current client count, but I don't really understand the difference between FullClient.GetServerVariables().Result.Value.ClientsOnline and ClientConnections

[Bluscream]

ClientConnections is the total count of connections for a server, so for example ClientConnections=1000 means 1000 times some client connected to the server since creation. ClientsOnline is just the currently online clients

@Flakebi would FullClient.GetServerVariables() also update the server book when executed through a plugin?

[MoroseCorpse]

Ok thanks for the information

[MoroseCorpse]

I just noticed an error, because apparently the books are not thread-safe, because they only use normal dictionaries. The following error just came up:

2020-06-02 20:10:59.2513|FATAL||Core.UnhandledExceptionHandler Critical program failure!
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
   at SupportBot.SupportBot.MySQLUpdate(String BotID, TsFullClient fullClient) in D:\Users\Moritz\Desktop\Development\SupportBot - 0.12 Version\ClassLibrary5\ClassLibrary5\SupportBot.cs:line 1639
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

At this point in my code is only this:

var clis = fullClient.Book.Clients.Values;

foreach (var clon in clis)
{
...
}

[Splamy]

Your code accessing the book collection is running from the threadpool

at System.Threading.ThreadPoolWorkQueue.Dispatch()

the system is built to have one thread per bot scheduled by the dedicatetaskscheduler, so your code should run on that one too. If you need a timer use scheduler.CreateTimer instead of other timers or threads. In all other cases like bot commands etc are already scheduled on the correct thread.

[Flakebi]

would FullClient.GetServerVariables() also update the server book when executed through a plugin?

Looking into MessagesToBook.toml, there is currently nothing that updates the optional data, so currently no. If this is implemented at some point, it would be a yes. Every incoming command gets matched against the commands in this file.

[MoroseCorpse]

Your code accessing the book collection is running from the threadpool

at System.Threading.ThreadPoolWorkQueue.Dispatch()

the system is built to have one thread per bot scheduled by the dedicatetaskscheduler, so your code should run on that one too. If you need a timer use scheduler.CreateTimer instead of other timers or threads. In all other cases like bot commands etc are already scheduled on the correct thread.

I have now switched to your timer, but the problem still occurs. My timer code:

public DedicatedTaskScheduler scheduler { get; set; }
public void Initialize()
{
        TickWorker tickWorker = scheduler.CreateTimer(OnTimedEvent, TimeSpan.FromSeconds(10), true);
}
public async void OnTimedEvent()
{
...
}

And here is the code at the point where the error occurs:

var commandInfo = (await FullClient.ChannelInfo(channelid)).Value;
foreach (var chinfo in commandInfo)
{
        cname = chinfo.Name;
        cdesc = chinfo.Description;
        ctopic = chinfo.Topic;
        cmaxclients = chinfo.MaxClients;
}

[Splamy]

The second code part you posted doesn't even access the book, so it seems like something else is wrong

[MoroseCorpse]

But the error output refers exactly to this position in the code