Add BufferChannel for fsm incoming/outgoing channels.

[taitov]

Added the ability to select a type of fsm incoming/outgoing channels:

1. InfiniteChannel (old behavior, default)
2. BufferChannel

---

BufferChannel allows to accumulate and combine paths in a separate thread, in the case when the reading side (for example, BgpServer.Serve) is busy.

A map with NLRI key (bufferChannel.pathIdxs) is also used to combine multiple changes of one path into a single update.

BufferChannel prevents infinite memory consumption because its maximum size is comparable to the size of the adj-in (or adj-out for the fsm outgoing channel).

[taitov]

cc #2862

[bayrinat]

@fujita could you please take a look at it?

[fujita]

I don't think it's a good idea for GoBGP's most core code to have two different implementations. What's the maximum buffer size of Go's buffered channel? If it's large, then we could use buffered channel instead of infinite channel like #2862 The buffered size can be specified by a command line option or such.

[fujita]

btw, what kind of use case do you have that puts such a heavy load on GoBGP?

[taitov]

I don't think it's a good idea for GoBGP's most core code to have two different implementations. What's the maximum buffer size of Go's buffered channel? If it's large, then we could use buffered channel instead of infinite channel like #2862 The buffered size can be specified by a command line option or such.

A regular channel with a limited size will not work, as it can still be reached, which may lead to disconnection with peer.

BufferChannel can combine multiple updates into one, discarding outdated path's attributes. The most effectiveness of this channel is shown when there is a storm of attribute updates for one path from peer.

[taitov]

Сonfiguration example:

peer-groups:
  - config:
      peer-group-name: group1
      peer-as: 12345
      local-as: 54321
      incoming-channel-timeout: 1
      incoming-channel:
        channel-type: buffer
        size: 64
      outgoing-channel:
        channel-type: buffer
        size: 64

neighbors:
  - config:
      neighbor-address: 10.0.0.1
      peer-group: group1

or

neighbors:
  - config:
      neighbor-address: 10.0.0.1
      incoming-channel:
        channel-type: buffer
        size: 64
      outgoing-channel:
        channel-type: buffer
        size: 64
  - config:
      neighbor-address: 10.0.0.2
      incoming-channel:
        channel-type: buffer
        size: 256

[taitov]

btw, what kind of use case do you have that puts such a heavy load on GoBGP?

Paths are being cyclically updated from peers due to attribute updates. And main thread BgpServer.Serve() a busy (ex, processes the mgmtOperation or propagateUpdate paths when peer went disconnect). In this scenario, InfiniteChannel begins to fill up quickly.

[taitov]

@fujita gentle ping

[taitov]

@fujita gentle ping

[fujita]

Why just not removing incoming queues?

[taitov]

Why just not removing incoming queues?

• Then peers can start flap (hold timer expired).
• OOM on remote peer (overflow outgoing fsm channel).

[taitov]

Why just not removing incoming queues?

• Then peers can start flap (hold timer expired).
• OOM on remote peer (overflow outgoing fsm channel).

But with this PR - сan add another type of fsm incoming/outgoing channels, which will disable queues.

[fujita]

I meant that why not removing incoming channels?

[taitov]

I meant that why not removing incoming channels?

I understand you :) And my answer: if we remove incoming channels, we will get possible problems: peers flap and OOM on remote gobgp server.

[fujita]

Hmm, why? goroutine, reads from a socket, parses a BGP message, and updates the table with a lock and PrefixLimit config. I don't see the difference about OOM.

[mpeim]

Hey! Do you know where we are regarding this PR? Couldn't we do it in two steps, first solve inbound channels and afterward the others?

[taitov]

Test result with 10k updates per second:

Green line: memory usage. Blue line: CPU usage.

InfiniteChannel (old behavior)

BufferChannel

InfiniteChannel used 4GB memory vs. BufferChannel 16MB on same BGP updates pattern.

[fujita]

Nice graphs! Really clear and informative.

Is there a way to reproduce a similar load in my environment (like MRT or something)?

[taitov]

Is there a way to reproduce a similar load in my environment (like MRT or something)?

I think the easiest way would be to create this scheme:

       Peer1                               Peer2
┌─────────────────┐                 ┌─────────────────┐
│      GOBGP      │   100k routes   │  ModifiedGOBGP  │
│ Infinite/Buffer │◄────────────────┤                 │
│     Channel     │   10M updates   │                 │
└─────────────────┘                 └────────▲────────┘
                                             │         
                                             │         
                                      Add 100k routes  
                                       to global RIB   

Where ModifiedGOBGP is https://github.com/osrg/gobgp/commit/f42a6fec4346b5015c75872fe05028fb45ec595a

[bayrinat]

@fujita hey, is there any details that you need? This PR fixes unnecessary OOMs, it would be nice to have it in upstream.

[fujita]

https://github.com/osrg/gobgp/pull/2993 I plan to remove incoming channels completely.

[fujita]

Incoming channel was removed completely. It would be great if you could also check the memory usage with the same test.