DatAasee (0.5)

DatAasee centralizes and interlinks distributed library/research metadata into an API‑first union catalog.

A Metadata-Lake for Libraries

Repository: github.com/ulbmuenster/dataasee (nb sources backup)

Maintainer: Christian Himpe (at University and State Library of Münster)

Licenses: MIT (add. CC-BY for openapi.yaml)

Function: Metadata-Lake, Metadata Catalog, Metadata Aggregator, Union Catalog

Audience: University Libraries, Research Libraries, Academic Libraries, Scientific Libraries

Documentation

• Dependencies Overview
• Software Documentation
• Architecture Documentation
• Database Schema
• OpenAPI Schema (Swagger UI)
• DatAasee: A Metadata-Lake as Metadata Catalog for a Virtual Data-Lake (Companion Paper, Open Access)

Getting Started (Deployment)

Quick Start (Prepare a dedicated directory, inside run:)

$ wget https://raw.githubusercontent.com/ulbmuenster/dataasee/0.5/compose.yaml
$ mkdir -p -m 766 backup
$  DL_PASS=password1 DB_PASS=password2 docker compose up

Web: http://localhost:8000 (API: http://localhost:8343/api/v1/ )

• Depends on docker compose (and compatible to docker and podman)
• To deploy, no need to clone, just use the compose.yaml file.
• See the Deploy Documentation for details.

Tech Stack Canvas

• Setting: Many distributed data and metadata sources
• Goals:
  • Centralize metadata
  • Interlinked metadata catalog
  • Super-index for bibliographic and research data
• Features:
  • Interact through HTTP-API (JSON)
  • Search by filter, full-text, source, doi
  • Custom query via: SQL, Gremlin, Cypher, MQL, GraphQL
• Frontend: Lowdefy (Optional)
• Backend: Connect (fmr. Benthos)
• Data Storage: ArcadeDB (Graph Database)
• Infrastructure: Compose (via Docker or Podman)
• Deployment: via Harbor (at Uni Münster)
• Monitoring: Container Logs (local logging driver)
• Integrations:
  • Protocols: OAI-PMH (HTTP), S3 (HTTP), GET (HTTP), DatAasee (HTTP)
  • Encodings: XML (Plain-Text)
  • Formats: DataCite (XML), DC (XML), LIDO (XML), MARC (XML), MODS (XML)
• Exports: DataCite (JSON), BibJSON (JSON)
• Security: Privileged endpoints (CQRS)
• Testing: check-jsonschema
• Development: Github

Default Ports

• 8343 DatAasee API
• 8000 Web Frontend
• 2480 Database API (Development Container Images Only)
• 9999 Database JMX (Development Container Images Only)

API Cheat Sheet

• GET api/v1/api Returns API specification and schemas.
• GET api/v1/ready Returns service readiness.
• GET api/v1/metadata Returns queried metadata records.
• GET api/v1/sources Returns ingested metadata sources.
• GET api/v1/schema Returns database schema.
• GET api/v1/enums Returns enumerated attributes.
• GET api/v1/stats Returns metadata record statistics.
• POST api/v1/backup Triggers database backup.
• POST api/v1/ingest Triggers async ingest of metadata.
• POST api/v1/insert Inserts single metadata record.
• POST api/v1/health Probes and returns service liveness.

Repository Contents

• api/ API definition and message schemas
• assets/ Logos and style definition
• backend/ Processor pipeline and component definitions
• container/ Dockerfiles
• database/ Database initialization, schemas and enumerated data
• docs/ Documentation of software, data and architecture
• frontend/ Prototype frontend definition
• tests/ Test definitions and data

Getting Started (Development)

• Available make targets:
  • make setup Build server images (builds development images)
  • make start Start servers
  • make stop Stop servers
  • make reset Stop and start servers
  • make build Build release images (pass REGISTRY= to set container image registry)
  • make empty Delete database backups
  • make logs Show logs (requires grep)
  • make peak Report peak database memory usage (requires grep)
  • make test Run tests (requires check-jsonschema, busybox, wget)
  • make tidy List violations of StrictYAML (requires yamllint)
  • make todo List inline TODOs in repo (requires grep)
• Custom make variable: COMPOSE (set Compose implementation)

Contributors

• See here

tl;dr

DatAasee is centralized Metasearch for distributed Metadata.