kdb 2024-07-15 - KDB Project Call Meeting Minutes

Date

Monday, July 15, 2024 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Karl Moll	FINOS	@karlmoll
Peter Gyorok	KX Systems	@gyorokpeter
Ross Duffy	Morgan Stanley	@rdffy
Rian Ó Cuinneagáin	KX Systems	@rianoc
Gary Davies	Data Intellect	@drgdavies
Aaron Davies	Morgan Stanley	@adavies42
Jude Reilly	Data Intellect
Alvin Shih	Morgan Stanley	@alvin-c-shih

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.
All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.
FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.
FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
(35 min) Preview of Pythonic query API for PyKX and feedback.
- https://github.com/finos/kdb/issues/99 has QF fragment for comparison.
- Current PyKX throws them straight into q.
- Hard to pass in variables.
- New PyKX uses & as alternative to list syntax.
  - Builds a QueryPhrase object.
  - What about & operator in q?
    - kx.q._and - Can't use a Python keyword as a method name.
    - kx.q('&') also works.
  - Users will be able to register functions.
- Also a kx.ParseTree class.
- Questions around syntax for common string operations.
  - ssr each entry of a column?
    - iterator=kx.q.each
  - String match also commonly needs an iterator:
    - c3~\:"foo"
  - b-strings already taken to mean q strings in PyKX
    - can't be used as a shorthand for kcol wrapper
- Had to pause the conversation to leave time for next presentation.
(10 min) Demo of KDB.AI for lung images.
- https://dataintellect.com/blog/kdb-ai-a-breath-of-fresh-air/
- KDB.AI allows inserting chunks up to 10 MB.
- Classifying lung images using k Nearest Neighbors with k=9.
- Embeddings from RESNET 50.
- Nearest neighbors shown to user for transparency / explainability.
- PoC accuracy of 79.9%.
  - Might be nice to show a confusion matrix to see if the false positives / negatives skew a certain way.
- Flat index is exhaustive search.
  - Higher accuracy.
- HNSW (Hierarchical Navigable Small Worlds) would be better for larger data set.
(5 min) AOB, Q&A, Adjourn.