Steven Bird comments

Results 76 comments of


                                            Steven Bird

[fix] Fixes missing probabilities and unreachable rules in PCFG CNF conversion

@ruoxining, it's good to have this contribution, however I'm sure you're not the first person to come up with an algorithm for this task. So it would be good if...

[fix] Fixes missing probabilities and unreachable rules in PCFG CNF conversion

@ruoxining, would you please open a new PR when you're ready? This will be a welcome contribution.

Neither `word_tokenize` nor `TreebankWordTokenizer` matchs the original Penn Word Tokenizer

Thanks @oxinabox. @alvations: I'm not clear what you mean when you say: "a new Tokenizer object and get whichever regex hack we need from any existing or outside tokenizer"

Move the pickles to a special collection

@hteeyeoh this is not a time-critical issue, so no promises. I suggest you use whichever punkt package you need.

Let tokenizers be installed via Pip

I for one welcome an overhaul here. It would be good to have broad consensus about a good way forward.

Avoid KeyError in langnames.py

@ekaf would you like to wrap this up? It is still in draft.