tinygo
tinygo copied to clipboard
tinygo-org
Tinygo need serialization tools
We really need serialization tools for tinygo, because tiny go can not use serialization tools just like coding/json encoding/gob in official golang.
Agree, having Google ProtoBuf is must have from the embeded world point of view... I didn't try to compile ProtoBuf with TinyGo so far.
@aykevl, maybe mark with issue as docs and help wanted and maybe someone will take over this?
From the list of things that work and what they are blocked on, it looks like to enable a lot of them we need to get base64 working first before moving on to solve gob, etc. The error for base64 regarding branching on a const expression might be addressed in dev, according to a comment on #437 two days ago, so this is a good place to look. If this fixed base64, it probably unblocks work on other packages. Gob and json might benefit from the recent updates to reflection, so it's certainly something worth checking on dev so that we can get going with more encoders.
As of TinyGo v0.8.0 the encoding/base64 package still cannot be imported due to #437 so that is still not fixed.
As of TinyGo v0.8.0 the
encoding/base64package still cannot be imported due to #437 so that is still not fixed.
encoding/base64 seems to be supported now, but that's still not sufficient for ProtoBuf support. The culprit now being encoding/json:
# encoding/json
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:346:23: Map not declared by package sync
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1264:21: Map not declared by package sync
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1117:19: sf.Tag.Get undefined (type string has no field or method Get)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1130:11: ft.Name undefined (type reflect.Type has no field or method Name)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1187:41: ft.Name undefined (type reflect.Type has no field or method Name)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:767:16: PtrTo not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:708:11: t.Key undefined (type reflect.Type has no field or method Key)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:713:9: t.Key undefined (type reflect.Type has no field or method Key)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:392:7: t.Implements undefined (type reflect.Type has no field or method Implements)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:396:14: PtrTo not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:401:7: t.Implements undefined (type reflect.Type has no field or method Implements)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:405:14: PtrTo not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:365:11: WaitGroup not declared by package sync
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:916:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:945:6: v.SetBytes undefined (type reflect.Value has no field or method SetBytes)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:949:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:983:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:991:23: v.OverflowInt undefined (type reflect.Value has no field or method OverflowInt)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:999:23: v.OverflowUint undefined (type reflect.Value has no field or method OverflowUint)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:1007:23: v.OverflowFloat undefined (type reflect.Value has no field or method OverflowFloat)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:622:40: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:638:12: t.Key undefined (type reflect.Type has no field or method Key)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:643:16: PtrTo not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:643:24: t.Key undefined (type reflect.Type has no field or method Key)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:650:18: MakeMap not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:771:12: t.Key undefined (type reflect.Type has no field or method Key)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:775:31: reflect.ValueOf(key).Convert undefined (type reflect.Value has no field or method Convert)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:776:17: PtrTo not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:787:40: reflect.Zero(kt).OverflowInt undefined (type reflect.Value has no field or method OverflowInt)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:791:30: reflect.ValueOf(n).Convert undefined (type reflect.Value has no field or method Convert)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:795:40: reflect.Zero(kt).OverflowUint undefined (type reflect.Value has no field or method OverflowUint)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:799:30: reflect.ValueOf(n).Convert undefined (type reflect.Value has no field or method Convert)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:805:7: v.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:517:8: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:550:13: Copy not declared by package reflect
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:554:7: v.SetLen undefined (type reflect.Value has no field or method SetLen)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:591:6: v.SetLen undefined (type reflect.Value has no field or method SetLen)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:450:41: v.Type().Name undefined (type reflect.Type has no field or method Name)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:476:15: v.Type().NumMethod undefined (type reflect.Type has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:306:30: cannot convert nil (untyped nil value) to reflect.Type
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:309:39: d.errorContext.Struct.Name undefined (type reflect.Type has no field or method Name)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:291:26: cannot convert nil (untyped nil value) to reflect.Type
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:159:15: cannot convert nil (untyped nil value) to reflect.Type
Still, progress, thanks to everyone involved!
The 0.9 release gave us base32, base64, and csv. We'll probably need additional reflection support before json, xml, and gob. I expect ProtoBuf will have other things it needs. Great progress!
It's not necessary to use encoding/json to parse and encode JSON in Go. This library compiles on tinygo with -target wasm: https://github.com/buger/jsonparser
This is possible because it import graph is really small - the only thing that imports 'reflect' is its fmt dependency: https://godoc.org/github.com/buger/jsonparser?import-graph
I wasn't able to test jsonparser because I'd like to use it to handle a HTTP request response, but alas I don't see any examples for passing values from JS to exported WASM functions, even though I can call them in the JS callback.
This is a good tip. While most people using JSON probably don't want the tradeoffs in jsonparser, I'd be willing to bet most users of tinygo would want those tradeoffs, at least if they are using tinygo for tiny machines.
for golang/protobuf, apart from the json issue mention above, there are the following other issues:
../../vendor/github.com/golang/protobuf/proto/extensions.go:62:46: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/extensions.go:82:71: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/extensions.go:155:78: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/text_parser.go:765:20: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:785:18: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:469:14: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/text_parser.go:506:8: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:507:8: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:561:18: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:599:21: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:601:34: dst.Type().Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:655:8: dst.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/text.go:404:23: st.Field(i).Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/text.go:412:38: inner.Type().Field(0).Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/text.go:208:13: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text.go:209:12: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text.go:221:11: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1736:10: t.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1738:44: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1739:44: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1799:18: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1803:5: m.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:488:57: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:610:47: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:613:46: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:454:39: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:294:60: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:302:66: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:305:13: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:313:58: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:322:18: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:329:17: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:332:51: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:366:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:378:31: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:406:29: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:409:71: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:190:41: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_merge.go:212:59: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:500:53: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:502:58: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:572:55: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:581:22: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_merge.go:589:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:595:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:600:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:609:61: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2282:15: t.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2284:33: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2285:33: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:476:32: sf.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:446:30: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:420:15: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:316:13: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:324:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:346:25: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:370:12: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:374:12: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:285:40: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:372:30: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:374:18: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:394:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:410:27: sft.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:278:29: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:278:43: p.mtype.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/properties.go:278:64: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:284:20: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:286:43: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:289:16: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:300:15: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:159:37: reflect.ValueOf(&v).Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:152:20: p.v.Elem().Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:133:13: p.v.Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:113:5: s.SetLen undefined (type reflect.Value has no field or method SetLen)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:115:17: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:100:31: p.v.Elem().FieldByIndex undefined (type reflect.Value has no field or method FieldByIndex)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:54:11: f.Index undefined (type *reflect.StructField has no field or method Index)
../../vendor/github.com/golang/protobuf/proto/equal.go:119:15: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:120:13: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:126:15: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:127:13: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:133:11: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:139:11: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/discard.go:318:29: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/discard.go:332:13: v.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/clone.go:169:20: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:185:8: out.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/clone.go:212:8: out.SetBytes undefined (type reflect.Value has no field or method SetBytes)
../../vendor/github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:227:21: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:122:11: in.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/clone.go:128:7: out.FieldByName undefined (type reflect.Value has no field or method FieldByName)
For anyone else trying to get protobuf to work... it can. Sort of.
If you use gogoproto, you can have it generate marshal and unmarshal functions that work on the bytes, without reflection.
protoc --gogofaster_out=. *.proto
Then, you "just" need to stub the proto package with something like this
package proto
import "errors"
const Marshal = 0
const GoGoProtoPackageIsVersion3 = 0
type Message interface {
}
type InternalMessageInfo interface {
DiscardUnknown(interface{})
Merge(interface{}, interface{})
Marshal(interface{}, interface{}, interface{}) ([]byte, error)
}
func RegisterFile(string, []byte) {}
func RegisterType(interface{}, string) {}
func CompactTextString(interface{}) string {
return "nope"
}
You can then call Marshal and Unmarshal directly.
Just wanted to give a heads up. My tests have found that https://github.com/valyala/fastjson compiles successfully and runs with the TinyGo compiler targeting WASM/WASI.
Is there any other way to compile those packages that are not supported by tinygo? Must find support instead?
Is there any other way to compile those packages that are not supported by tinygo? Must find support instead?
The issue is the TinyGo compiler doesn't support 100% of Go's reflect package. And most JSON packages depend on those functions in some way.
So at the moment, the only work around is find JSON packages that don't need those unimplemented functions.
I am a third-party package using the xml package, which is not easy to replace.
Unfortunately, there isn't much that can be done right now until the necessary changes/fixes are made within the TinyGo compiler.
The tinylib/msgp messagepack code generation package works with tinygo. There's even a test in the msgp repo for it: https://github.com/tinylib/msgp/tree/master/tinygotest
I made a proof of concept demo to use code generation to implement json.Unmarshal https://github.com/json-iterator/tinygo Is there similar tools already? Is this path worth pursuing?
@dgryski all features of json.Unmarshal has been implemented https://github.com/json-iterator/tinygo
verified it is working properly in tinygo https://github.com/json-iterator/tinygo/tree/main/demo
It looks like mailru/easyjson works with tinygo if you remove the var _ easyjson.Marshaler line from the generate code that wants to bring in net/http and friends.
Since https://github.com/tinygo-org/tinygo/pull/2314 was merged, mailru/easyjson now compiles and works.
Since #2314 was merged,
mailru/easyjsonnow compiles and works.
Is there a chance to make it work on macos / M1?
I'm getting https://github.com/tinygo-org/tinygo/pull/2314#issuecomment-979148013
Removing var _ easyjson.Marshaler https://github.com/tinygo-org/tinygo/issues/447#issuecomment-997022972 helps,
but it's not super convenient ;-)
Is there a chance to make it work on macos / M1? I'm getting #2314 (comment)
The error you've linked is for x86_64, not arm64.
If you're only getting that error, it's unrelated to serialization and seems to be caused by cryptograpy-related packages (which might be imported by mailru/easyjson, I didn't check). For those errors, see: https://github.com/tinygo-org/tinygo/pull/2688
I haven't tried encoding/json yet but from the website I see:
Which is correct?
Ping @mathetake @jptosso
Attempting to use encoding/json resulted in the following error for me:
panic: unimplemented: (reflect.Type).NumMethod()
some experiments: https://github.com/cbrake/tinygo
karmem at least compiles. Could not get any protobuf stuff to compile -- may need to port nanopb or something ...
@cbrake VTProtobuf with the references to the reflection-based Google protobuf libraries removed - and one extra generator which creates .pb.go with just the structs & not the reflection stuff - would probably do the job. It's possible with just hacks, I think.
I am a third-party package using the xml package, which is not easy to replace.
Me too
