Should the logspout container be collecting it's own logs?

[treeder]

I noticed while trying to create a module that the adapter is getting the logs from the "logspout" container itself. So if I have any log output in the Stream method, it's recursive. For instance:

func (a *MyAdapter) Stream(logstream chan *router.Message) {
    for message := range logstream {
        // buf := new(bytes.Buffer)
        atomic.AddUint64(&a.counter, 1)
        fmt.Println(atomic.LoadUint64(&a.counter), "mdata:", message.Data)
    }
}

I will see log lines that grow recursively like this:

May 31 08:55:40 063f7159f635 logspout: 738 mdata: 736 mdata: 734 mdata: 732 mdata: 730 mdata: 728 mdata: 726 mdata: 724 mdata: 722 mdata: 720 mdata: 718 mdata: 716 mdata: 714 mdata: 712 mdata: 710 mdata: 708 mdata: 706 mdata: 704 mdata: 702 mdata: 700 mdata: 698 mdata: 696 mdata: 694 mdata: 692 mdata: 690 mdata: 688 mdata: 686 mdata: 684 mdata: 682 mdata: 680 mdata: 678 mdata: 676 mdata: 674 mdata: 672 mdata: 670 mdata: 668 mdata: 666 mdata: 664 mdata: 662 mdata: 660 mdata: 658 mdata: 656 mdata: 654 mdata: 652 mdata: 650 mdata: 648 mdata: 646 mdata: 644 mdata: 642 mdata: 640 mdata: 638 mdata: 636 mdata: 634 mdata: 632 mdata: 630 mdata: 628 mdata: 626 mdata: 624 mdata: 622 mdata: 620 mdata: 618 mdata: 616 mdata: 614 mdata: 612 mdata: 610 mdata: 608 mdata: 606 mdata: 604 mdata: 602 mdata: 600 mdata: 598 mdata: 596 mdata: 594 mdata: 592 mdata: 590 mdata: 588 mdata: 586 mdata: 584 mdata: 582 mdata: 580 mdata: 578 mdata: 576 mdata: 574 mdata: 572 mdata: 570 mdata: 568 mdata: 566 mdata: 564 mdata: 562 mdata: 560 mdata: 558 mdata: 556 mdata: 554 mdata: 552 mdata: 550 mdata: 548 mdata: 546 mdata: 544 mdata: 542 mdata: 540 mdata: 538 mdata: 536 mdata: 534 mdata: 532 mdata: 530 mdata: 528 mdata: 526 mdata: 524 mdata: 522 mdata: 520 mdata: 518 mdata: 516 mdata: 514 mdata: 512 mdata: 510 mdata: 508 mdata: 506 mdata: 504 mdata: 502 mdata: 500 mdata: 498 mdata: 496 mdata: 494 mdata: 492 mdata: 490 mdata: 488 mdata: 486 mdata: 484 mdata: 482 mdata: 480 mdata: 478 mdata: 476 mdata: 474 mdata: 472 mdata: 470 mdata: 468 mdata: 466 mdata: 464 mdata: 462 mdata: 460 mdata: 458 mdata: 456 mdata: 454 mdata: 452 mdata: 450 mdata: 448 mdata: 446 mdata: 444 mdata: 442 mdata: 440 mdata: 438 mdata: 436 mdata: 434 mdata: 432 mdata: 430 mdata: 428 mdata: 426 mdata: 424 mdata: 422 mdata: 420 mdata: 418 mdata: 416 mdata: 414 mdata: 412 mdata: 410 mdata: 408 mdata: 406 mdata: 404 mdata: 402 mdata: 400 mdata: 398 mdata: 396 mdata: 394 mdata: 392 mdata: 390 mdata: 388 mdata: 386 mdata: 384 mdata: 382 mdata: 380 mdata: 378 mdata: 376 mdata: 374 mdata: 372 mdata: 370 mdata: 368 mdata: 366 mdata: 364 mdata: 362 mdata: 360 mdata: 358 mdata: 356 mdata: 354 mdata: 352 mdata: 350 mdata: 348 mdata: 346 mdata: 344 mdata: 342 mdata: 340 mdata: 338 mdata: 336 mdata: 334 mdata: 332 mdata: 330 mdata: 328 mdata: 326 mdata: 324 mdata: 322 mdata: 320 mdata: 318 mdata: 316 mdata: 314 mdata: 312 mdata: 310 mdata: 308 mdata: 306 mdata: 304 mdata: 302 mdata: 300 mdata: 298 mdata: 296 mdata: 294 mdata: 292 mdata: 290 mdata: 288 mdata: 286 mdata: 284 mdata: 282 mdata: 280 mdata: 278 mdata: 276 mdata: 274 mdata: 272 mdata: 270 mdata: 268 mdata: 266 mdata: 264 mdata: 262 mdata: 260 mdata: 258 mdata: 256 mdata: 254 mdata: 252 mdata: 250 mdata: 248 mdata: 246 mdata: 244 mdata: 242 mdata: 240 mdata: 238 mdata: 236 mdata: 234 mdata: 232 mdata: 230 mdata: 228 mdata: 226 mdata: 224 mdata: 222 mdata: 220 mdata: 218 mdata: 216 mdata: 214 mdata: 212 mdata: 210 mdata: 208 mdata: 206 mdata: 204 mdata: 202 mdata: 200 mdata: 198 mdata: 196 mdata: 194 mdata: 192 mdata: 190 mdata: 188 mdata: 186 mdata: 184 mdata: 182 mdata: 180 mdata: 178 mdata: 176 mdata: 174 mdata: 172 mdata: 170 mdata: 168 mdata: 166 mdata: 164 mdata: 162 mdata: 160 mdata: 158 mdata: 156 mdata: 154 mdata: 152 mdata: 150 mdata: 148 mdata: 146 mdata: 144 mdata: 142 mdata: 140 mdata: 138 mdata: 136 mdata: 134 mdata: 132 mdata: 130 mdata: 128 mdata: 126 mdata: 124 mdata: 122 mdata: 120 mdata: 118 mdata: 116 mdata: 114 mdata: 112 mdata: 110 mdata: 108 mdata: 106 mdata: 104 mdata: 102 mdata: 100 mdata: 98 mdata: 96 mdata: 94 mdata: 92 mdata: 90 mdata: 88 mdata: 86 mdata: 84 mdata: 82 mdata: 80 mdata: 78 mdata: 76 mdata: 74 mdata: 72 mdata: 70 mdata: 68 mdata: 66 mdata: 64 mdata: 62 mdata: 60 mdata: 58 mdata: 56 mdata: 54 mdata: 52 mdata: 50 mdata: 48 mdata: 46 mdata: 44 mdata: 42 mdata: 40 mdata: 38 mdata: 36 mdata: 34 mdata: 32 mdata: 30 mdata: 28 mdata: 26 mdata: 24 mdata: 22 mdata: 20 mdata: 18 mdata: 16 mdata: 14 mdata: 12 mdata: 10 mdata: 8 mdata: 6 mdata: 4 mdata: 2 mdata: mylogmessage

And those get bigger and bigger, very quickly.

Is this intended behavior?

[josegonzalez]

Thats how log collection works! You ever try turning on verbose logging in elasticsearch and then collect logs via logstash? It's great fun.

How do you think we should handle this case?

[treeder]

Sounds fun! ;)

One way could be to check running containers, if image name is gliderlabs/logspout, then ignore those lines. I've added this to my adapter, but feels like it should be something built in that could do this and not pass them to the logstream channel.

for message := range logstream {
   if message.Container.Name == "/logspout" {
      continue
   }
   ....

[josegonzalez]

PRs welcome!

[treeder]

Do you think this is a common thing that most people would want, or is it just me?

[josegonzalez]

We should at least gate it. I can see someone naming a container funny and this breaking things. It seems like it should be a default though.

Maybe we can do IGNORED_CONTAINERS as a comma-separated env var (if that isn't already a thing)?

[michaelshobbs]

We also have this method for ignoring containers

[josegonzalez]

Though that is a bit more invasive to a user's infrastructure.

[ebr]

@treeder are you sure this is the default behaviour and not something specific to the module you're working on? I actually expected to mitigate against this sort of recursion when i first started using logspout, but was pleased to see it wasn't collecting its own logs. Also, happened to double check this yesterday while working on a repro case for #187, with the same results (no recursion)...

[treeder]

@ebr I'm not doing anything fancy with the module, in fact when generating that, the only thing the module did at that point was print the message data field so I could see what I was getting. Think it was just something like this:

func (a *StatsdAdapter) Stream(logstream chan *router.Message) {
    for message := range logstream {
        atomic.AddUint64(&a.counter, 1)
        log.Println(atomic.LoadUint64(&a.counter), "mdata:", message.Data)
     }
}