Peter Bourgon
Peter Bourgon
OK Log is reasonably well described as a system of queues. I’d really like to model the system, in the style of Adrian Cockcroft’s [microservice response time distribution analysis](http://www.slideshare.net/adriancockcroft/microxchg-analyzing-response-time-distributions-for-microservices). I've...
At the moment, if a producer produces records faster than the forwarder can forward them, the producer will experience backpressure. I think this is the correct default choice, but it...
Ingesters can gossip some concept of load, e.g. connected clients, total ingest throughput, etc. With that knowledge shared, they can do some kind of connection balancing, in order of increasing...
Right now the stores just choose random peers during replication. If they gossip their e.g. data set size, they can instead bias toward those nodes that have a lower overall...
The ingester can take an entire segment in a single write or series of writes, somehow. It can block until that segment has been successfully replicated to store nodes, and...
Right now the stores are polling every 100ms or so to consume segments from ingesters. This is of course inefficient. It would be better if they attached some kind of...
Queries can perform read repair, by counting each unique ULID (and thus log record) encountered during the deduplicate phase. Those which have lower representation than expected can be bundled together...
Right now, during queries, we spawn a new segment reader (and filterer) per matching segment. That'll hit our fdlimit real real fast, and at some point we get diminishing returns...
Let's sit down and have a think about what protecting the system at its borders would look like. - Exhaustive input validation on user-facing store APIs, e.g. dates in range...
Proposed interface ``` go type Logger interface { Logf(format string, args ...interface{}) } ```