feat(gnovm): gnomod.toml add new field `private`
fix third task of https://github.com/gnolang/gno/issues/4370
depends on: https://github.com/gnolang/gno/pull/4413
still in progress
private should make realm replaceable meaning it should not harm or causse issue with other realms:
- it cannot be import by other pkg (or replace could break other realms)
- others realms should not be able to store things from this realm (struct & pointer)
π PR Checks Summary
All Automated Checks passed. β
Manual Checks (for Reviewers):
- [ ] IGNORE the bot requirements for this PR (force green CI check)
Read More
π€ This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers.
β Automated Checks (for Contributors):
π’ Maintainers must be able to edit this pull request (more info) π’ Changes to 'docs' folder must be reviewed/authored by at least one devrel and one tech-staff π’ Pending initial approval by a review team member, or review from tech-staff
βοΈ Contributor Actions:
- Fix any issues flagged by automated checks.
- Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include
BREAKING CHANGEnotes. - Link related issues/PRs, where applicable.
βοΈ Reviewer Actions:
- Complete manual checks for the PR, including the guidelines and additional checks if applicable.
π Resources:
Debug
Automated Checks
Maintainers must be able to edit this pull request (more info)
If
π’ Condition met βββ π’ And βββ π’ The base branch matches this pattern: ^master$ βββ π’ The pull request was created from a fork (head branch repo: MikaelVallenet/gno)Then
π’ Requirement satisfied βββ π’ Maintainer can modify this pull requestChanges to 'docs' folder must be reviewed/authored by at least one devrel and one tech-staff
If
π’ Condition met βββ π’ And βββ π’ The base branch matches this pattern: ^master$ βββ π’ A changed file matches this pattern: ^docs/ (filename: docs/resources/configuring-gno-projects.md)Then
π’ Requirement satisfied βββ π’ And βββ π’ Or β βββ π΄ Pull request author is a member of the team: tech-staff β βββ π’ At least 1 user(s) of the team tech-staff reviewed pull request(with state "APPROVED") βββ π’ Or βββ π΄ Pull request author is a member of the team: devrels βββ π’ At least 1 user(s) of the team devrels reviewed pull request(with state "APPROVED")Pending initial approval by a review team member, or review from tech-staff
If
π’ Condition met βββ π’ And βββ π’ The base branch matches this pattern: ^master$ βββ π’ Not (π΄ Pull request author is a member of the team: tech-staff)Then
π’ Requirement satisfied βββ π’ If βββ π’ Condition β βββ π’ Or β βββ π’ User leohhhn already reviewed PR 4422 with state APPROVED β βββ π’ At least 1 user(s) of the team tech-staff reviewed pull request β βββ π΄ This pull request is a draft βββ π’ Then βββ π’ Not (π΄ This label is applied to pull request: review/triage-pending)Manual Checks
**IGNORE** the bot requirements for this PR (force green CI check)
If
π’ Condition met βββ π’ On every pull requestCan be checked by
- Any user with comment edit permission
Codecov Report
:x: Patch coverage is 61.53846% with 75 lines in your changes missing coverage. Please review.
:loudspeaker: Thoughts on this report? Let us know!
got smth working will clean & set PR ready2review tomorrow
hello guys @thehowl, @ltzmaxwell, @mvertes do you mind to take a look. i'm not sure it's the good way to do, right now i'm working on understanding better the ownership & more globally running process of a tx to improve the PR. but it's mostly working, except in some case the gas cost increase at no-limit (or did not find the limit yet) leading to out of gas error always.
I'm pretty sure, it's not the good way or implemented the good way so i'm exploring more in depth how it works but if you have some advice, based on what i already did feel free to take a look
PR #4435 is adjusting the "gno maketx run" by marking certain packages as "ephemeral" in the Gno machine. Ephemeral packages will behave like a private realm, meaning they will panic if we try to save local objects or types remotely.
At some point, we could consider merging this effort. We might reuse the concept of IsEphemeral for a function that checks whether the package is ephemeral or private for specific validations.
In my opinion, #4435 is likely straightforward to review and will simplify the review of #4422 after the merge.
related docs PR: https://github.com/gnolang/gno/pull/4409
Finally, when the PR is done, please update the docs added in #4409 and ping me for a review π
start reading
Next topics i'll explore:
- Optimize OID, include private in the oid marshalling only if field is true
- How works HeapItemValue in details and why all my values are wrap in it & is it a way to bypass my checks
- Why i got refValues with oid zero & needed to add a check to skip them
@omarsy fixed your issue, it was not about created but more about checking for value that are not heapItemValue: 796948a
@MikaelVallenet thank you for working on this!
I have two questions here
- I'm curious about the rationale behind these rulesβspecifically, how they relate to
enabling private realm swapping without impacting other realms
-
For variables of private types: they can be modified within the public realm but cannot be stored by it.
-
For private primitive-type variables: they can be stored by the public realm.
-
private slices of strings in the private realm cannot be stored by the public realm. https://github.com/gnolang/gno/pull/4422/files#diff-3ea6c65f7953c81129f0c549e7c5a16ff31262a60485c1e1fe36dc875a7e90a2R32
-
private array of string in the private realm can be stored by the public realm
-
A private realm is callable by gnokey. How is that different from a public realm that are never be discovered or imported by other realms? Why we would treat them differently in the above rules?
It looks like these assertObjectIsPublic() and assertTypeIsPublic() are called at runtime, with validation for all objects recursively and repeatedly, regardless of whether a private realm is involved or not for AddPkg, Call and Run https://github.com/gnolang/gno/pull/4422/files#diff-7041eca77b078f22ee334239995c403673f040b3726bcffa5a8a053b68a951a7R805
I'm trying to understand the purpose and trade-offs behind these constraints. Any insights would be appreciated.
@MikaelVallenet thank you for working on this!
I have two questions here
- I'm curious about the rationale behind these rulesβspecifically, how they relate to
enabling private realm swapping without impacting other realms
- For variables of private types: they can be modified within the public realm but cannot be stored by it.
- For private primitive-type variables: they can be stored by the public realm.
- private slices of strings in the private realm cannot be stored by the public realm. https://github.com/gnolang/gno/pull/4422/files#diff-3ea6c65f7953c81129f0c549e7c5a16ff31262a60485c1e1fe36dc875a7e90a2R32
- private array of string in the private realm can be stored by the public realm
- A private realm is callable by gnokey. How is that different from a public realm that are never be discovered or imported by other realms? Why we would treat them differently in the above rules? It looks like these assertObjectIsPublic() and assertTypeIsPublic() are called at runtime, with validation for all objects recursively and repeatedly, regardless of whether a private realm is involved or not for AddPkg, Call and Run https://github.com/gnolang/gno/pull/4422/files#diff-7041eca77b078f22ee334239995c403673f040b3726bcffa5a8a053b68a951a7R805
I'm trying to understand the purpose and trade-offs behind these constraints. Any insights would be appreciated.
Hi,
-
"For variables of private types: they can be modified within the public realm but cannot be stored by it."
Since a public realm cannot import private types by design, Iβm not sure how this case would arise except through passing interfaces and i don't think it would cause problem. Do you have an example? Iβm not entirely certain I understand the scenario youβre describing here but we just don't want the public realm to have references to the private realm so if we remove/update the private realm it does not affect the public one. -
"For private primitive-type variables: they can be stored by the public realm."
What exactly is meant by a βprivate primitive typeβ? If itβs just a primitive value passed through a function, it will be copied. If you pass a pointer, the public realm still canβt store it if it's owned by the private one. -
"Private array of string in the private realm can be stored by the public realm."
Thatβs not the case; thereβs a test showing calledSaveYourSliceToPublicRealmthat is a []string{} stored in private and that show it fails -
"A private realm is callable by gnokey. How is that different from a public realm that are never be discovered or imported by other realms? Why we would treat them differently in the above rules?"
You canβt guarantee that a public realm will remain undiscovered or unimported forever. With private realms, the rules can be enforced to ensure isolation and safe swapping/removal later. -
"It looks like these assertObjectIsPublic() and assertTypeIsPublic() are called at runtime, with validation for all objects recursively and repeatedly, regardless of whether a private realm is involved or not for AddPkg, Call and Run."
Good point β if the private realm isnβt the one being called/deployed, and since it cannot be imported i don't think we need to run the validation, it should only be on deploy/call a private realm or a msg run since a run will be a private realm i think. I did not think of that before, i think it can be done in another PR or later if this approach is accepted
A proposal to improve complexity:
Actually, we only ever persist:
- the current object (oo), and
- the unsaved children returned by getUnsavedChildObjects(oo).
saveUnsavedObjectRecursively already walks that exact set depthβfirst:
- For oo, it calls getUnsavedChildObjects(oo).
- For each unsaved child that isnβt escaped, it recurses into saveUnsavedObjectRecursively(child).
- After children are handled, it persists oo itself.
Because of this recursion, every object that will be persisted is inevitably revisited as the βcurrent ooβ in its own recursive call. Thatβs the ideal moment to validate it. We donβt need I think an early deep scan over the whole graph or all nested types: the traversal guarantees coverage. If an object appears multiple times in the graph, flags (IsNewReal/IsDirty/IsEscaped) and the guards in the recursion prevent double persistence and thus avoid duplicate validation. Escaped/newβescaped nodes are intentionally skipped from preemptive saving (and thus from validation here), since theyβre not persisted via this path.
Practically, this means:
- Perform objectβlevel checks (private FuncValue PkgPath, private foreign ObjectID) right before persisting the current oo.
- Perform shallow, slot-by-slot type checks for the current ooβs owned fields (Array items, Struct fields, Func captures, BoundMethod func/receiver, Map key/value, HeapItem payload). We validate βwhat we ownβ now; deeper objects will be validated when they become the current oo in their own recursive call.
Somethinh like this I think
--- a/gnovm/pkg/gnolang/realm.go
+++ b/gnovm/pkg/gnolang/realm.go
@@ -801,9 +801,6 @@ func (rlm *Realm) saveUnsavedObjectRecursively(store Store, oo Object) {
}
}
- // assert object have no private dependencies.
- rlm.assertObjectIsPublic(oo, store)
-
// first, save unsaved children.
unsaved := getUnsavedChildObjects(oo)
for _, uch := range unsaved {
@@ -852,6 +849,19 @@ func (rlm *Realm) saveObject(store Store, oo Object) {
oo.SetIsEscaped(true)
// XXX anything else to do?
}
+
+ if fv, ok := oo.(*FuncValue); ok {
+ if fv.PkgPath != rlm.Path && IsPkgPrivateFromPkgPath(fv.PkgPath) {
+ panic("cannot persist function or method from private realm")
+ }
+ }
+
+ if oo.GetObjectID().PkgID != rlm.ID && IsPkgPrivateFromPkgID(oo.GetObjectID().PkgID) {
+ panic("cannot persist reference of object from private realm")
+ }
+
+ rlm.assertTypeCanBeOwned(oo)
+
// set object to store.
// NOTE: also sets the hash to object.
rlm.sumDiff += store.SetObject(oo)
@@ -904,143 +914,6 @@ func (rlm *Realm) clearMarks() {
rlm.escaped = nil
}
-// assertObjectIsPublic ensures that the object is public and does not have any private dependencies.
-// it check recursively the types of the object
-// it does not recursively check the values because
-// child objects are validated separately during the save traversal (saveUnsavedObjectRecursively)
-func (rlm *Realm) assertObjectIsPublic(obj Object, store Store) {
- objID := obj.GetObjectID()
- if objID.PkgID != rlm.ID && IsPkgPrivateFromPkgID(objID.PkgID) {
- panic("cannot persist reference of object from private realm")
- }
-
- // NOTE: should i set the visited tids map at the higher level so it's set one time.
- // it could help to reduce the number of checks for the same type.
- tids := make(map[TypeID]struct{})
- switch v := obj.(type) {
- case *HeapItemValue:
- if v.Value.T != nil {
- rlm.assertTypeIsPublic(store, v.Value.T, tids)
- }
- case *ArrayValue:
- for _, av := range v.List {
- if av.T != nil {
- rlm.assertTypeIsPublic(store, av.T, tids)
- }
- }
- case *StructValue:
- for _, sv := range v.Fields {
- if sv.T != nil {
- rlm.assertTypeIsPublic(store, sv.T, tids)
- }
- }
- case *MapValue:
- for head := v.List.Head; head != nil; head = head.Next {
- if head.Key.T != nil {
- rlm.assertTypeIsPublic(store, head.Key.T, tids)
- }
- if head.Value.T != nil {
- rlm.assertTypeIsPublic(store, head.Value.T, tids)
- }
- }
- case *FuncValue:
- if v.PkgPath != rlm.Path && IsPkgPrivateFromPkgPath(v.PkgPath) {
- panic("cannot persist function or method from private realm")
- }
- if v.Type != nil {
- rlm.assertTypeIsPublic(store, v.Type, tids)
- }
- for _, capture := range v.Captures {
- if capture.T != nil {
- rlm.assertTypeIsPublic(store, capture.T, tids)
- }
- }
- case *BoundMethodValue:
- if v.Func.PkgPath != rlm.Path && IsPkgPrivateFromPkgPath(v.Func.PkgPath) {
- panic("cannot persist bound method from private realm")
- }
- if v.Receiver.T != nil {
- rlm.assertTypeIsPublic(store, v.Receiver.T, tids)
- }
- case *Block:
- for _, bv := range v.Values {
- if bv.T != nil {
- rlm.assertTypeIsPublic(store, bv.T, tids)
- }
- }
- if v.Blank.T != nil {
- rlm.assertTypeIsPublic(store, v.Blank.T, tids)
- }
- case *PackageValue:
- if v.PkgPath != rlm.Path && IsPkgPrivateFromPkgPath(v.PkgPath) {
- panic("cannot persist package from private realm")
- }
- default:
- panic(fmt.Sprintf("assertNoPrivateDeps: unhandled object type %T", v))
- }
-}
-
-// assertTypeIsPublic ensure that the type t is not defined in a private realm.
-// it do it recursively for all types in t and have recursive guard to avoid infinite recursion on declared types.
-func (rlm *Realm) assertTypeIsPublic(store Store, t Type, visited map[TypeID]struct{}) {
- pkgPath := ""
- switch tt := t.(type) {
- case *FuncType:
- for _, param := range tt.Params {
- rlm.assertTypeIsPublic(store, param, visited)
- }
- for _, result := range tt.Results {
- rlm.assertTypeIsPublic(store, result, visited)
- }
- case FieldType:
- rlm.assertTypeIsPublic(store, tt.Type, visited)
- case *SliceType, *ArrayType, *ChanType, *PointerType:
- rlm.assertTypeIsPublic(store, tt.Elem(), visited)
- case *tupleType:
- for _, et := range tt.Elts {
- rlm.assertTypeIsPublic(store, et, visited)
- }
- case *MapType:
- rlm.assertTypeIsPublic(store, tt.Key, visited)
- rlm.assertTypeIsPublic(store, tt.Elem(), visited)
- case *InterfaceType:
- pkgPath = tt.GetPkgPath()
- for _, method := range tt.Methods {
- rlm.assertTypeIsPublic(store, method, visited)
- }
- case *StructType:
- pkgPath = tt.GetPkgPath()
- for _, field := range tt.Fields {
- rlm.assertTypeIsPublic(store, field, visited)
- }
- case *DeclaredType:
- tid := tt.TypeID()
- if _, exists := visited[tid]; !exists {
- visited[tid] = struct{}{}
- rlm.assertTypeIsPublic(store, tt.Base, visited)
- for _, method := range tt.Methods {
- rlm.assertTypeIsPublic(store, method.T, visited)
- if mv, ok := method.V.(*FuncValue); ok {
- rlm.assertTypeIsPublic(store, mv.Type, visited)
- }
- }
- }
- pkgPath = tt.GetPkgPath()
- case *RefType:
- t2 := store.GetType(tt.TypeID())
- rlm.assertTypeIsPublic(store, t2, visited)
- case PrimitiveType, *TypeType, *PackageType, blockType, heapItemType:
- // these types do not have a package path.
- // NOTE: PackageType have a TypeID, should i loat it from store and check it?
- return
- default:
- panic(fmt.Sprintf("assertTypeIsPublic: unhandled type %T", tt))
- }
- if pkgPath != "" && pkgPath != rlm.Path && IsPkgPrivateFromPkgPath(pkgPath) {
- panic("cannot persist object of type defined in a private realm")
- }
-}
-
//----------------------------------------
// getSelfOrChildObjects
@@ -1567,6 +1440,75 @@ func fillTypesTV(store Store, tv *TypedValue) {
tv.V = fillTypesOfValue(store, tv.V)
}
+func (rlm *Realm) assertTypeCanBeOwned(oo Object) {
+ switch cv := oo.(type) {
+ case nil:
+ case *ArrayValue:
+ for _, ctv := range cv.List {
+ rlm.assertTypeCanBeOwned2(ctv.T)
+ }
+ case *StructValue:
+ for _, ctv := range cv.Fields {
+ rlm.assertTypeCanBeOwned2(ctv.T)
+ }
+ case *FuncValue:
+ for _, c := range cv.Captures {
+ rlm.assertTypeCanBeOwned2(c.T)
+ }
+ case *BoundMethodValue:
+ rlm.assertTypeCanBeOwned2(cv.Func.Type)
+ rlm.assertTypeCanBeOwned2(cv.Receiver.T)
+ case *MapValue:
+ for cur := cv.List.Head; cur != nil; cur = cur.Next {
+ rlm.assertTypeCanBeOwned2(cur.Key.T)
+ rlm.assertTypeCanBeOwned2(cur.Value.T)
+ }
+ case *PackageValue:
+ case *Block:
+ case *HeapItemValue:
+ rlm.assertTypeCanBeOwned2(cv.Value.T)
+ default:
+ panic(fmt.Sprintf(
+ "unexpected type %v",
+ reflect.TypeOf(oo)))
+ }
+
+}
+
+func (rlm *Realm) assertTypeCanBeOwned2(ty Type) {
+ var pkgPaths []string
+ switch tt := ty.(type) {
+ case nil:
+ case *FuncType: // maybe impossible
+ case FieldType:
+ fmt.Printf("assertTypeCanBeOwned2: %T\n", ty)
+ case *SliceType, *ArrayType, *ChanType, *PointerType:
+ pkgPaths = append(pkgPaths, tt.Elem().GetPkgPath())
+ case *tupleType: // maybe impossible
+ case *MapType:
+ pkgPaths = append(pkgPaths, tt.Key.GetPkgPath())
+ pkgPaths = append(pkgPaths, tt.Elem().GetPkgPath())
+ case *InterfaceType: // maybe impossible
+ fmt.Printf("assertTypeCanBeOwned2: %T\n", ty)
+ pkgPaths = append(pkgPaths, tt.GetPkgPath())
+ case *StructType:
+ pkgPaths = append(pkgPaths, tt.GetPkgPath())
+ case *DeclaredType:
+ pkgPaths = append(pkgPaths, tt.GetPkgPath())
+ case *RefType:
+ pkgPaths = append(pkgPaths, tt.GetPkgPath())
+ case PrimitiveType, *TypeType, *PackageType, blockType, heapItemType:
+ default:
+ panic(fmt.Sprintf("assertTypeIsPublic: unhandled type %T", tt))
+ }
+
+ for _, pkgPath := range pkgPaths {
+ if pkgPath != "" && pkgPath != rlm.Path && IsPkgPrivateFromPkgPath(pkgPath) {
+ panic("cannot persist object of type defined in a private realm")
+ }
+ }
+}
+
// Partially fills loaded objects shallowly, similarly to
// getUnsavedTypes. Replaces all RefTypes with corresponding types.
func fillTypesOfValue(store Store, val Value) Value {
WDYT ?
+func (rlm *Realm) assertTypeCanBeOwned2(ty Type) { + var pkgPaths []string + switch tt := ty.(type) { + case nil: + case *FuncType: // maybe impossible + case FieldType: + fmt.Printf("assertTypeCanBeOwned2: %T\n", ty) + case *SliceType, *ArrayType, *ChanType, *PointerType: + pkgPaths = append(pkgPaths, tt.Elem().GetPkgPath()) + case *tupleType: // maybe impossible + case *MapType: + pkgPaths = append(pkgPaths, tt.Key.GetPkgPath()) + pkgPaths = append(pkgPaths, tt.Elem().GetPkgPath()) + case *InterfaceType: // maybe impossible + fmt.Printf("assertTypeCanBeOwned2: %T\n", ty) + pkgPaths = append(pkgPaths, tt.GetPkgPath()) + case *StructType: + pkgPaths = append(pkgPaths, tt.GetPkgPath()) + case *DeclaredType: + pkgPaths = append(pkgPaths, tt.GetPkgPath()) + case *RefType: + pkgPaths = append(pkgPaths, tt.GetPkgPath()) + case PrimitiveType, *TypeType, *PackageType, blockType, heapItemType: + default: + panic(fmt.Sprintf("assertTypeIsPublic: unhandled type %T", tt)) + } + + for _, pkgPath := range pkgPaths { + if pkgPath != "" && pkgPath != rlm.Path && IsPkgPrivateFromPkgPath(pkgPath) { + panic("cannot persist object of type defined in a private realm") + } + } +}
if you don't check type recursively you open vuln i think traversal by object / values does not ensure there is no private types (especially nil typed values) it's why i did recursive on types when values traversal is handled by the function but after trying your code i saw it was passing the test so i added one to demonstrate what i mean:
func SaveNilSliceOfSliceOfPrivateToPublicRealm(cur realm) {
// nil value with type [][]*PrivateData
var arr [][]*PrivateData = nil
publicrealm.SaveYourInterface(cross, arr)
}
This succeed with your impl. when it should fails. Do you see a way to handle this in a better way than i did ? Thanks a lot for review :heart:
I think this could be a solution, but I'm not entirely certain. My sample doesn't handle empty elements, which should be improved. To address your issue maybe this (we should handle map too):
func (rlm *Realm) assertTypeCanBeOwned2(ty Type) {
var pkgPaths []string
fmt.Printf("assertTypeCanBeOwned2: %T %v\n", ty, ty)
switch tt := ty.(type) {
case nil:
case *FuncType: // maybe impossible
case FieldType:
case *SliceType, *ArrayType, *ChanType, *PointerType:
ty = tt.Elem()
Loop:
for { // When slice or array are empty we will not have any value in get child object so we should check all the declaration
switch tt := ty.(type) {
case *SliceType, *ArrayType, *ChanType, *PointerType:
ty = tt.Elem()
default:
pkgPaths = append(pkgPaths, ty.GetPkgPath())
break Loop
}
}
case *tupleType:
case *MapType: // maybe handle empty case
pkgPaths = append(pkgPaths, tt.Key.GetPkgPath())
pkgPaths = append(pkgPaths, tt.Elem().GetPkgPath())
case *InterfaceType:
pkgPaths = append(pkgPaths, tt.GetPkgPath())
case *StructType:
pkgPaths = append(pkgPaths, tt.GetPkgPath())
case *DeclaredType:
pkgPaths = append(pkgPaths, tt.GetPkgPath())
case *RefType:
pkgPaths = append(pkgPaths, tt.GetPkgPath())
case PrimitiveType, *TypeType, *PackageType, blockType, heapItemType:
default:
panic(fmt.Sprintf("assertTypeIsPublic: unhandled type %T", tt))
}
for _, pkgPath := range pkgPaths {
if pkgPath != "" && pkgPath != rlm.Path && IsPkgPrivateFromPkgPath(pkgPath) {
panic("cannot persist object of type defined in a private realm")
}
}
}
WDYT ?
[c73fde0](/gnolang/gno/pull/4422/commits/c73fde00596919ae9fc21eafd4555889b77572b8)
would not be safer to use a visited map ? : c73fde0 ? This way we still process everything without bypass possible on empty values but we mark them to avoid process same type multiple time.
If not already added (in a hurry now to check), we should refer to https://github.com/gnolang/gno/pull/4594 in the vm keeper where we make maketx run run using a private package
back to this now
Making a PR to fix docs links
-> https://github.com/gnolang/gno/pull/4826