Ben Kurtovic

Implement missing node types

22

- ~~`Comment`~~ - `ParserFunction` / `MagicWord` / `BehaviorSwitch` - ~~`Link`~~ - ~~`Table`~~ - `Redirect`

The tokenizer incorrectly handles some difficult tag-related markup

14

1. Bold and italics that cross contexts are handled incorrectly, because the tree structure does not support overlapping nodes (for example, `''foo'''bar''baz'''`, or `''foo{{bar|baz''}}`). Fixing this will probably be very...

C tokenizer: emit tokens simpler than the expensive PyObject* kind

3

Allocating and filling the slots of `PyObject*`s every time we create a token (even if it is later discarded) is a large overhead; ideally, we use custom structs for each...

The Python C API docs recommend `goto` (gasp) for error handling in C functions, where there's a block at the end of each function that calls `Py_XDECREF()` on every Python...

get_sections() should handle headings nested inside other nodes

1

Another avenue to address part 4 of #40. Not sure how to solve this, though. Might be impossible.

There can be a lot of external links in a page, causing a check to go on for far too long (several minutes). We should enqueue external links with lower...

https://phabricator.wikimedia.org/T110124

Sometimes things like namespaces are added to sites, so periodically, the bot should update this info.

Unit tests. I desperately need to write some, mainly in wiki, but IRC commands could use them as well.

Ruby parser bugs

1

Got this: ``` 14-07-06 01:47:15 DEBUG bitshift.crawler.indexer.GitIndexer Indexing file: ruby/ruby: test/ruby/test_primitive.rb 14-07-06 01:47:15 ERROR bitshift.crawler.indexer.GitIndexer Exception raised while parsing: Traceback (most recent call last): File "bitshift/crawler/indexer.py", line 169, in _insert_repository_codelets...