Inconsistent select ordering

[bheni]

Related to an issue that was previously fixed:

https://github.com/dolthub/dolt/issues/984

Now we are seeing that every ~20 minutes or so the ordering changes. Running the query:

What's weird is that the ordering is flipping back and forth between two different result sets. E.g. run it and the first result is label X. Wait 20 minutes then the first result is label Y. Wait another 20 minutes and the result will be label X again. Wait again label Y...etc etc

[fulghum]

I set up some repro scripts to try and catch this happening, but haven't caught it yet.

I populated a table with ~1M records, then started a script that slowly adds a few more records to the table every couple of seconds, and then another script continuously polls the table with a limit of 1 and stops if it detects a different value than it's expecting. I ran this set up for well over an hour, but didn't see anything. However... when I was initially writing the scripts, I think I did see a different result than what I'm getting now.

I'll leave this running and see if we can repro this and find some more clues. I have a few other ideas for tweaks to the repro set up, too. After that I can add some more debug output to a local build and see if that helps turn anything up.