UCT/IB/DC: bugfix - dci_pool_index have only 3 bits in ep->flags (it's max value is 15)

[roiedanino]

What

Adding dci_pool_index attribute to uct_dc_mlx5_ep_t instead of using the first 3 bits of ep->flags.

Why ?

UCT_DC_MLX5_IFACE_MAX_DCI_POOLS is defined to be 16, meaning 3 bits aren't enough, this patch fixes the issue by adding a separate field for dci_pool_index.

[gleon99]

@roiedanino given that we already mess with the flags enum there, maybe expand the dci pool index to 4 bits instead of adding a separate field?

[roiedanino]

@roiedanino given that we already mess with the flags enum there, maybe expand the dci pool index to 4 bits instead of adding a separate field?

The cons are:

1. Encountering the same issue next time MAX_DCI_POOLS is increased.
2. Extra opcode (bitwise AND) each time the pool index is being accessed.
3. Index is not semantically a flag - less intuitive.

I guess the benefit is having a smaller ep struct.

[roiedanino]

I guess the benefit is having a smaller ep struct.

Although now we can decrease flags to use uint8_t instead of uint16_t

[brminich]

I guess the benefit is having a smaller ep struct.

Although now we can decrease flags to use uint8_t instead of uint16_t

imo, better to keep flags as uint16_t and use 4 (or 5) bits for dci pool index. We do not need 8 bits for dci pool index and this way we can keep the rest bits for flags.

[yosefe]

@roiedanino can you pls squash?