Click Processing Pipeline Rewrite

[mitsuhiko]

Click evolved unfortunately really badly the last two releases because the internal logic just grew too complex and it's totally not sound. This needs to be cleaned up but I'm not yet sure how.

Click needs to somehow become a pipeline that is clearly separated to be able to work better. Starting with parsing, handling of value callbacks and then invoking the callbacks. Maybe we need to split this pipeline into more steps where we can have args that are not yet processed by callbacks somehow?

Some effects of the current broken handling:

• https://github.com/pallets/flask/issues/1124 (auto help does not invoke callbacks)
• chain is broken. There is a failing testcase in the repo

[mitsuhiko]

I'm not entirely sure how we ended up on this point but it was probably a bad design from the start and it got more and more eroded over time. I will need to write down the exact effects this causes but essentially we badly mix parsing with invoking so if something needs to "inverse" the order things get really out of hand. For instance --help internally is a huge hack right now. This also shows up in a few situations now outside as people have discovered.

• As an example chain commands broke so bad in 6.x that they cannot accept arguments any more.
• Flask's --app option is not processed for the default help rendering (removed now in Flask)
• You cannot reliably implement --config flags that can override some settings that might affect the execution of Click because there is no good hook point.
• Because we invoke parent commands before the subcommand but the separation is not clear there are some really odd effects where arguments might or might not be available to them on the context.