Support huge JSON schemas

[wjn0]

Is your feature request related to a problem? Please describe. Huge JSON schemas fail when generating the template.

Describe the solution you'd like Ideally, the whole schema should not be processed at once. Instead, process it on the fly as generation proceeds.

Describe alternatives you've considered Provide some kind of progress indicator to see if the current implementation is in fact hopeless, or I'm just impatient/under-resourced hardware-wise :)

Additional context

Minimum reproducible example (note: downloads a 3.5MB public-facing JSON Schema from a well-known org, probably cache this if you're gonna run it a bunch):

from typing import List
import requests

from transformers import AutoModelForCausalLM, AutoTokenizer
import guidance
from pydantic import BaseModel


print("guidance version: ", guidance.__version__)


model_name = "gpt2"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
lm = guidance.models.Transformers(model=model, tokenizer=tokenizer)

# download latest FHIR schema build
schema = requests.get("https://build.fhir.org/fhir.schema.json").json()


messages = [
    {"role": "system",
     "content": "Construct a FHIR Bundle resource from the provided patient information. Provide your output in valid FHIR JSON."},
    {"role": "user",
     "content": "John Smith was diagnosed with diabetes on 2021-01-01. He is prescribed metformin 500mg daily. He is allergic to penicillin."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

lm += prompt

print(lm)

# lm += guidance.gen(max_tokens=100)
lm += guidance.json(schema=schema)

print(lm)

I ran it against wjn0/guidance@improve-json-schema-support in order to have hackish fixes of #887 and #888 (which hopefully are not the reason for the hang...)