allow to explicitly request a pending transfer source re-evaluation

[rcarpa]

In some cases, when a rule is added towards an RSE, it is desired to re-evaluate the sources of a pending transfer; cancel the pending transfer; and re-start the transfer with a new source.

The operator will have to explicitly ask for this behavior by adding a flag on the rule.

This may be impossible. Because the pending transfer can be part of a bigger FTS bulk job. So cancelling that job may be impossible. Evaluate the feasibility of this behavior and, if it's feasible, implement it.

[rcarpa]

I have a feeling that the better solution is to just never submit those transfers to FTS and keep them in waiting state on rucio side. Maybe implement a way on FTS side to slow down submission for a particular pair. Something like the TCP flow control. This way they will be correctly submitted with the best source at the time of submission

[rcarpa]

Discussed this ticket off-line with @mpatrascoiu. There is no consensus about it. FTS prefers rucio to submit all transfers. So that they can better handle their queues. This means that we must implement a logic to cancel pending transfers in FTS and re-submit them from a different source. It's borderline impossible to do it for bulk transfers.

I still strongly remain on the opinion that we should keep transfers in a WAITING state in rucio rather than submitting everything.