Significant delay in sending large arrays (2.3.1 vs 2.4.0+)

[Gmjjr]

Large arrays that take seconds to send in 2.3.1 take 10+ minutes in 2.4.0+.

I have 2 servers, 1 acts as a master handling user data and the other as a game server. The game server connects to the master and is sent a large chunk of data upon startup; this process takes 2-3 seconds in 2.3.1 and 10+ minutes in 2.4.0+ occasionally resulting in a crash or failure to send.

For Issues

• 2 standalone servers connected via DarkriftClient.dll, both hosted on different machines.
• Send a large array to the client and wait, my example uses an array containing 100k items.

[Gmjjr]

This also happens server to client, even if on the same machine. I wrote a quick server script and used the chat demo alongside it, the results locally were: 2.10.1: 2 minutes 2.3.1: 0.35 seconds

using DarkRift;
using DarkRift.Server;
using System;
using System.Collections.Generic;

namespace DarkriftIssue
{
    public class Class1 : Plugin
    {
        public override bool ThreadSafe => true;
        public override Version Version => new Version(0, 0, 0);

        private List<string> words = new List<string>();
        public Class1(PluginLoadData pluginLoadData) : base(pluginLoadData)
        {
            ClientManager.ClientConnected += ClientConnected;
        }
        protected override void Loaded(LoadedEventArgs args)
        {
            base.Loaded(args);
            for(int i = 0; i < 100000; i++)
            {
                words.Add(i.ToString());
            }
        }

        void ClientConnected(object sender, ClientConnectedEventArgs e)
        {
             e.Client.MessageReceived += Client_PlayerEvent;
        }
        public void Client_PlayerEvent(object sender, DarkRift.Server.MessageReceivedEventArgs e)
        {
            using (DarkRiftWriter writer = DarkRiftWriter.Create())
            {
                WriteEvent($"Preparing to send {words.Count} words at {DateTime.Now}", LogType.Info);
                writer.Write(words.ToArray());
                using (Message message = Message.Create(0, writer))
                {
                    e.Client.SendMessage(message, SendMode.Reliable);
                }
                WriteEvent($"Sent at {DateTime.Now}", LogType.Info);
            }
        }
    }
}

[xAL95]

Hi,

I have one solution (tested)

In DarkRiftWriter.cs change function public void Write(string[] value)

to

public void Write(string[] value)
        {
            buffer.EnsureLength(Position + 4);          //Encodings suck, just do this manually
            EndianHelper.WriteBytes(buffer.Buffer, Position, value.Length);
            Position += 4;
            buffer.Count = Math.Max(Length, Position);

            int length = 0;
            foreach(string b in value)
                length += Encoding.GetByteCount(b);

            buffer.EnsureLength(Position + (value.Length * 4) + length);

            foreach (string b in value)
            {
                int bLen = Encoding.GetByteCount(b);
                EndianHelper.WriteBytes(buffer.Buffer, Position, bLen);
                Encoding.GetBytes(b, 0, b.Length, buffer.Buffer, Position + 4);
                Position += 4 + bLen;
            }

            buffer.Count = Math.Max(Length, Position);
        }

Tested resullt: 4ms

[JamJar00]

Just to confirm what I think you've done in this code: You've moved the EnsureLength call out so that it's not called once per string (ignore the array length bit) but only once per the whole array. I can see that sorting the performance problem to be honest.

I think when I first saw this I assumed it was a problem with the sending code rather than the allocation code. A simple test to see if that is actually the case would be to pass in an appropriatly large initial length when calling DarkRiftWriter.Create((int length) since if it doesn't have to rearrange arrays constantly it should be back at 2.3.1 levels of performance. (Side note, you should really be doing this anyway rather than forcing DR to resize you arrays)

Good find!

[4real]

My attempt at solving this in a general manner: https://github.com/DarkRiftNetworking/DarkRift/pull/164

To be fair, I was surprised it wasn't implemented like this already. Of course, hypothetically 50% memory waste a big concern, but it's a trade I'm willing to make at any time to avoid this kind of memory buffer thrashing.