Allow the guide to consume the eos_token

[RobinPicard]

Addresses https://github.com/dottxt-ai/outlines-core/issues/227

This PR aims at fixing a contradictory behavior in the current implementation of the Index class: the eos_token is listed in the output of the function allowed_tokens, but it cannot be consumed in the next_state function. As a result, in the Guide, the eos_token is returned by the get_tokens function but causes an error if used when calling the advance function.

The solution proposed here is to allow the Guide to consume the eos_token such that calling the next_state function with the eos_token as an argument leads to the current state.

Downside: if some users expect the eos_token to be rejected, by the Guide.accepts_tokens function for instance, it could lead to unexpected behavior for them. An exemple would be this implementation in vllm:

    def accept_tokens(self, request_id: str, tokens: list[int]) -> bool:
        """Accepts a list of tokens and advances the FSM.

        Returns True if the FSM was advanced successfully.
        Returns False if the FSM failed to advance.
        """
        if self.guide.accepts_tokens(tokens):
            # Advance cannot fail because we checked Guide.accepts_tokens()
            for t in tokens:
                self.guide.advance(t)
                self.num_processed_tokens += 1
            return True
        return False