polymath

polymath copied to clipboard

Add a way to pass request info back in libraries

2

We have a few use cases that need an extra place to stash information--mainly for results from Library.query(). for example, we need a way to pass back: - Number of...

jkomoros

Rationalize the server POST parameters and return type

9

To switch https://github.com/dglazkov/wanderer to rely on polymath, it needs a way to query a random set of text chunks, to reproduce this functionality here: https://github.com/dglazkov/wanderer/blob/main/ask_embeddings.py#L119

dglazkov

Add schemas in a format where VSCode will naturally validate the various json files and help with autocomplete and catching errors. This is done in https://github.com/jkomoros/CASsim

jkomoros

A lot of chunks have an identical `info` (since the info is related to the article they come from, and many articles have multiple chunks). Now that we've done #15,...

jkomoros

https://help.readwise.io/article/121-how-can-i-customize-the-markdown-export

jkomoros

Since server already uses Brotli to compress responses, we should probably take advantage of it and teach `urllib3` requests Brotli: https://urllib3.readthedocs.io/en/stable/advanced-usage.html#brotli-encoding

dglazkov

Create a tool to generate embeddings given text

1

To start, it should produce a `.pkl` file according to the format described in https://github.com/dglazkov/polymath/blob/main/format.md It should accept a JSON document with a structure like: ``` { chunks: [ text:...

jkomoros

I wonder if we could pull off using the same format for query and for response? Like, basically, it's the fill-in-the-blanks type of request?

dglazkov

Add a tool to generate chunks given substack

1

Basically, port https://github.com/dglazkov/ask-my-substack/blob/main/prepare.py here.

dglazkov

Consider having a namespace prefix on `embedding_model`

1

Currently `embedding_model`, in library and in server, is the string `text-embedding-ada-002`. But in the future there might be multiple models in production. Should we prepend a namespace in front of...

jkomoros