celestiaorg
Namespace pinning
We should support the ability to allow nodes to "pin" namespaces, so that they automatically download all the data in that namespace, and also advertise themselves on the network that they have pinned that namespace, so other nodes that are querying for data by namespace can download that data.
This would allow rollups to be responsible for the storage of their data while piggy-backing on celestia-node's peer-to-peer network, rather than needing to use their own peer-to-peer network.
Should they do pinning of historical data as well or only within recency? Although, we can make this configurable by the same archival flag.
Pinning should include historical data - the main point of it is to allow rollups to be responsible for their historical data storage.
Yes, that's the plan for partial nodes @musalbas
FWIW, my intuitive opinion is that namespaced / pinned nodes should act as full nodes within the sampling window (3 weeks + 1 week buffer = ~1 month) and then have a specific implementation of pruning that will go through blocks and delete all other data outside of their pinned namespace.
This would 1). make the sampling window more robust by not complicating the roles/responsibilities within that range so to not complicate serve-ability for that period, and 2). reduce code complexity.
We will also need partial nodes that only download their namespace's data within the sampling window, though, otherwise they are not really partial nodes but custom pruned full nodes.
I think namespace pinning should be something that can be enabled for both full nodes and partial nodes (when implemented)
Partial nodes could be light nodes that have a service to automatically "subscribe" or download the namespaced data for every block?
What is the point of a partial node within the sampling window?
Partial nodes could be light nodes that have a service to automatically "subscribe" or download the namespaced data for every block?
This would be the equivalent of just calling GetAll on every block height within sampling window + storage for those shares + proofs.
What is the point of a partial node within the sampling window?
Rollup nodes that only care about downloading their rollups' data, and not other people's
I'm more concerned from a networking standpoint about whether partial nodes have an important role? If not, I don't see implementing partial nodes within the sampling window to be that high of a priority.
You mean whether we need to specifically implement service discovery for partial nodes within a sampling window? Probably not, though it could be high priority if we expect that certain rollups have a lot more rollup nodes downloading their data by namespace than others. For namespace pinning though, we just need to discover nodes that have historical data for certain namespaces to start with at least, though.
This is becoming increasingly called for by users (private DA related), I would much rather suggest running a "partial archival" node that is a net benefit to the network over a paid RPC and/or bespoke "selfish" services that don't advertise or gossip to Celestia.
Most users don't want to maintain a full network archive (heavy on storage and gossip bandwidth) but I bet would consider running a much lighter weight only pinned data one (pruned of most other data) for stuff they care about.
We have empirical results that users (i.e. rollup providers) don't even want to run light nodes, and have by and large requested more reliable ways of submitting and retrieving blobs with lower devops overhead. There's no reason to believe that those same users would be interested in running even more expensive partial nodes any time soon. So IMO this is a longer-term feature rather than something that should be prioritized.
As an application developer, I would love to avoid implementing my own storage and syncing systems, at least initially, when I am bootstrapping and experimenting with an application idea. Later on, when demand increases and storage access becomes a bottleneck, I would invest in reimplementing and optimizing it. However, until then, piggybacking on an existing general solution would be a huge time-saver, especially for base rollupors.
I don't mean that we should prioritize namespace pinning or caching immediately. Comparing it to other priorities, it does seem less critical. (However, I can find a few arguments on why we would want to prioritize optimizing developer onboarding over, e.g., light node bandwidth atm)
Additionally, I began to notice a bias towards hosted providers, such as Twinkle/QuickNode, in our prioritization. Personally, I would prefer to get a DO VPS, run my application with a namespace node on the server, and not rely on other third parties except the VPS provider.
Celestia offers numerous options for building with it, and it would be beneficial for us to define and align on the architecture of the apps we expect people to make in the short term, and then prioritize around it. Currently, everyone has their own idea of the application architecture, and this misalignment leads to different parties pulling things in opposite directions.