Add `TextTape::from(Vec<TextToken>, bool)` for Tape construction after token manipulation.
Currently, there is no way to reconstruct a Tape after performing raw token manipulation. A use case (which is what I have in a project I'm working on) is parsing scripted_variables files and replacing their reference tokens on, say, technologies with their actual values.
Looks good, let me think about if it makes sense to supply the bom marker in the constructor, as that should only surface from initial parsing 🤔
A thought ocurred to me, is your use case covered by something like:
// pseudo-code
impl TextTape {
pub fn tokens_mut(&mut self) -> &mut Vec<TextToken>
...
}
}
A benefit is that you don't need to clone the tokens (that's how I'm assuming you are able to pass in a Vec<TextToken> to the constructor).
If that does work, it would be nice to include a rustdoc example and give the caveat that any modification needs to respect expected invariants or could otherwise risk object / array index corruption.