[Feature Request]: show_progress=True to return an iterator with progress?

[chigkim]

Feature Description

When running ...Index.from_documents(documents, show_progress=True) prints the progress into the console. Can we have a feature to return an iterator with progress result, so we can use the info elsewhere like GUI?

Reason

N/A

Value of Feature

You can redirect the progress to GUI.

[logan-markewich]

That would be pretty hard to implement

Probably better off generating embeddings and parsing outside of llama-index

nodes = node_parser(documents)

node_texts = [n.get_content(metadata_mode="embed") for n in nodes]

# can do this using util function
embeddings = embed_model.get_text_embeddings_batch(node_texts)

# or iterate yourself one by one, or something in between)
# embeddings = [embed_model.get_text_embedding(text) for text in node_texts]

for node, embedding in zip(nodes, embeddings):
  node.embedding = embedding

index = VectorStoreIndex(nodes=nodes, ...)

[chigkim]

Thanks, that's extremely helpful! What about DocumentSummaryIndex? How would I make my own loop and create the index?

response_synthesizer = get_response_synthesizer(response_mode="tree_summarize", use_async=True) index = DocumentSummaryIndex.from_documents(documents, response_synthesizer=response_synthesizer)

Thanks so much!