DataFed icon indicating copy to clipboard operation
DataFed copied to clipboard

Epic - Support for Metadata Only Allocation

Open JoshuaSBrown opened this issue 9 months ago • 4 comments

Description

There is a need to allow creation of metadata only allocations, meaning storage for raw data files would only be supported externally. There is also interest in supporting other types of allocations on different storage mediums other than POSIX filesystems. Any rework of the repository and allocation types would need to be flexible enough to accomodate future changes along the same lines.

JoshuaSBrown avatar Apr 21 '25 11:04 JoshuaSBrown

As admin, when you manage a repo Manage Data Repo > New > Globus collection

A repositoryless one > Manage Data Repo > New > Want to use provenance Everything associated with a metadata will be avail (+ python API) Can't upload download get Can change allocations Move data from metadata record only repo to x

4 tickets Manage Data Repo > New > Non-Globus-Repository Disabling upload, download, get Python CLI Docs

AronPerez avatar May 05 '25 21:05 AronPerez

This is how we have doing this directly in the database.

db._useDatabase("sdms");


db.repo.save({
    "_key" : "empty",
    "_id" : "repo/empty",
    "capacity" : 80000000000000,
    "pub_key" : "dummy_repo_key",
    "address" : "tcp://somedomain:9000",
    "endpoint" : "<endpoint UUID>",
    "path" : "/path/to/content/empty/",
    "title" : "My dummy repo",
    "desc" : "",
    "domain" : "localhost"
});

db.alloc.save({
    "_key" : "6956825",
    "_id" : "alloc/6956825",
    "_from" : "u/<someone>_username",
    "_to" : "repo/cades-test",
    "_rev" : "_iAbn6Eq--t",
    "path" : "/data/cades-test/user/<someone>_username/",
    "rec_count" : 0,
    "rec_limit" : 1000,
    "data_size" : 0,
    "data_limit" : 104857600
  })

JoshuaSBrown avatar May 12 '25 20:05 JoshuaSBrown

The Repo and User documents are nodes in the database, the allocation connects the two.

JoshuaSBrown avatar May 12 '25 20:05 JoshuaSBrown

Image

AronPerez avatar May 12 '25 20:05 AronPerez