2024-02-12 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, February 12, 2024 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Alvin Shih	Morgan Stanley	@alvin-c-shih
Francisco Garau
Claire Poland	Data Intellect	@claire-sloan
Aaron Davies	Morgan Stanley	@adavies42
Karl Moll	FINOS	@karlmoll
Peter Gyorok	KX Systems	@gyorokpeter
Rian Ó Cuinneagáin	KX Systems	@rianoc
Ross Duffy	Morgan Stanley	@rdffy
Connor Gervin	KX Systems	@Connor-Gervin
Gary Davies	Data Intellect	@drgdavies

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

• (5 min) Convene, roll call, welcome new people.
• [ ] Display FINOS Antitrust Policy summary slide
• [ ] Review Meeting Notices (see above)
• (5 min) Approve previous meeting minutes.
  • Approved by acclamation.
• (10 min) PyKX
  • Docs for using it from the q side:
    • https://code.kx.com/pykx/2.3/pykx-under-q/intro.html
    • https://code.kx.com/pykx/2.3/pykx-under-q/upgrade.html
    • https://code.kx.com/pykx/2.3/getting-started/PyKX%20Introduction%20Notebook.html
  • PyKX vs arrowkdb for loading Parquet into q side:
    • Currently, arrowkdb is more efficient for loading.
      • https://github.com/KxSystems/arrowkdb
    • PyKX would incur more conversion overhead.
      • Focus has been on efficient interoperability with numpy.
    • From the Python side, Pandas 2.0 has better integration with pyarrow-backed DataFrames.
  • Question about potential for Pola.rs support.
    • For enterprise customers, onboarding an addtional language has overhead in terms of:
      • SMEs to curate the ecosystem, packaging tools, scanning tools, deployment tools, hygiene remediation, etc.
      • Rust not a first-class language at many of the biggest KX customers, and therefore a lower priority for KX.
• (10 min) kdb+ 4.1
  • Excitement about dictionary decompose, multiple assignment, and function signature enhancements to q.
    • Explicit data types should make code easier to test.
    • Pattern matching is structural, not regex.
  • qclone incompatibility due to use of file descriptor 3 for epoll
    • Will need a PR for that.
• (10 min) qclone
  • https://github.com/finos/kdb/tree/main/q/qclone
  • More flexible than peach / .z.pd:
    • Launch any kind of task off the main thread.
    • Useful for running blocked operations like HTTP requests that might not return for some time.
    • Results sent back to main process using async IPC.
  • Doesn't work on Windows because it doesn't have fork(2).
  • Potentially can increase horizontal scaling on AWS FinSpace.
• (5 min) KDB.AI
  • https://dataintellect.com/blog/exploring-anomaly-detection-with-kdb-ai/
    • Use of vector similarity seems more like supervised learning.
    • Anomaly detection is more about "unlike what I'm used to seeing".
    • Would be good to see an example along those lines.
  • DataIntellect has an internal TorQBot built on KDB.AI.
• (5 min)qbinparse
  • Updated to support big endian unsigned integers.
  • Will look into applying it to LSEG PCAP file.
• (5 min) AOB, Q&A, Adjourn.

Join Zoom Meeting

• https://zoom.us/j/94523415326?pwd=NkJDaUtUUlZTcHVwcUJHdG0yRFJUdz09
• Meeting ID: 945 2341 5326
• Passcode: 361990
• Find your local number: https://zoom.us/u/abTtYRZ0B6

[adavies42]

Aaron Davies/Morgan Stanley

[karlmoll]

Karl Moll / FINOS

[gyorokpeter]

Péter Györök/KX

[rianoc-kx]

Rian Ó Cuinneagáin / KX

[claire-sloan]

Claire Poland Data Intellect

[rdffy]

Ross Duffy/ Morgan Stanley

[rianoc-kx]

• https://code.kx.com/pykx/2.3/getting-started/PyKX%20Introduction%20Notebook.html
• https://code.kx.com/pykx/2.3/pykx-under-q/intro.html
  • https://code.kx.com/pykx/2.3/pykx-under-q/upgrade.html

[karlmoll]

https://github.com/finos/community/issues/303

[alvin-c-shih]

Alvin Shih / Morgan Stanley

[alvin-c-shih]

I've updated the Agenda / Minutes to link to the qclone area of the repo.

Read qclone.md first, then test_qclone.q, then the rest.