bkb2135

Currently, we generate references in the task, as well as in the async reference generation process

Create token_limit argument

1

Create token_limit, which when generating a reference answer, overrides the otherwise universal max_tokens argument defined by the llm object.

Stop Validators from Sending Empty Challenges

2

For wiki-based tasks, validators send empty challenges ~1.2% of the time.

Automatic Detection of Dates in Date Scoring

1

Currently we are using a regex function to capture dates because datetime objects cannot contain all of the years found in the wikipedia date sections. This requires some specificity in...

Occurs pretty infrequently but should be looked at

The priority function defined in the base miner does not behave as intended because of async keyword.

Add Layout Matching

1

Adds layout matching task and criteria to answer step in validation.

Some math tasks have the following problems. Numbers too big (numbers greater than the float limit) Numbers too small and inconsistently rounded (sometimes it's 0, other times it's 2e-12) 11^{7^{7}}...

Some math challenges are low quality or rely on unreasonably large computations.