validator icon indicating copy to clipboard operation
validator copied to clipboard
verified publisher icon go-playground

Weird panic - Undefined validation function 'required' on field XXX

Open kubeplusplus opened this issue 2 years ago • 6 comments

  • [x] I have looked at the documentation here first?
  • [x] I have looked at the examples provided that may showcase my question here?

Package version eg. v9, v10: v10.15.4

image

Issue, Question or Enhancement:

I am developing a consumer that is using nats.io. In the consumer, I am using the validator to validate my message before process next step but got a weird bug panic: Undefined validation function 'required' on field 'Subject'. Definitely the validator must have the built-in required function so something went wrong

Code sample, to showcase or reproduce:

import govalidator "github.com/go-playground/validator/v10"

func UseConsumer() func(msg *nats.Msg) {
	v := govalidator.New(govalidator.WithRequiredStructEnabled())

	// some init tasks

	func(msg *nats.Msg) {
		event := natsMsgToEvent(msg)
		if err := v.Struct(event); err != nil {
			subscriber.logger.Errorw(err.Error(), "nats_msg", utils.Stringify(msg))
			if err := msg.Nak(); err != nil {
				// it's important to log entire event here because we can trace it in log
				subscriber.logger.Errorw(ErrSubNakFail.Error(), "nats_msg", utils.Stringify(msg))
			}
			return
		}
		// my logic here
	}
}

And the error I got

panic: Undefined validation function 'required' on field 'Subject'

goroutine 47 [running]:
github.com/go-playground/validator/v10.(*Validate).parseFieldTagsRecursive(0xc00089ca80, {0x13c1c96?, 0x167d31d?}, {0x13c1c75, 0x7}, {0x0, 0x0}, 0x0)
	/app/vendor/github.com/go-playground/validator/v10/cache.go:294 +0x9d9
github.com/go-playground/validator/v10.(*Validate).extractStructCache(0xc00089ca80, {0x15d6f40?, 0xc0187c79e0?, 0x0?}, {0x137eb47, 0x5})
	/app/vendor/github.com/go-playground/validator/v10/cache.go:155 +0x569
github.com/go-playground/validator/v10.(*validate).validateStruct(0xc01feffb00, {0x18d7060, 0x2cc0ec0}, {0x14a9840?, 0xc0187c79e0?, 0xffffffffffffffff?}, {0x15d6f40?, 0xc0187c79e0?, 0xc0025f4230?}, {0x18eeb48, ...}, ...)
	/app/vendor/github.com/go-playground/validator/v10/validator.go:37 +0x196
github.com/go-playground/validator/v10.(*Validate).StructCtx(0xc00089ca80, {0x18d7060, 0x2cc0ec0}, {0x14a9840?, 0xc0187c79e0?})
	/app/vendor/github.com/go-playground/validator/v10/validator_instance.go:401 +0x5ac
github.com/go-playground/validator/v10.(*Validate).Struct(0xc002b7f500?, {0x14a9840?, 0xc0187c79e0?})
	/app/vendor/github.com/go-playground/validator/v10/validator_instance.go:370 +0x31
github.com/scrapnode/kanthor/infrastructure/streaming.(*NatsSubscriberPushing).Sub.func1(0xc0011d40d0?)
	/app/infrastructure/streaming/nats_subscriber_pushing.go:105 +0x6d
github.com/nats-io/nats%2ego.(*Conn).waitForMsgs(0xc0003cd800, 0xc0011d40d0)
	/app/vendor/github.com/nats-io/nats.go/nats.go:3001 +0x3f2
created by github.com/nats-io/nats%2ego.(*Conn).subscribeLocked in goroutine 30
	/app/vendor/github.com/nats-io/nats.go/nats.go:4232 +0x3c8

After take a look in the source code, I found where the panic was threw and put some test code by myself and test again

image

There is the error out I got after added my debug code. You can see the has variable is false but in the map v.validations we have required key

2023/09/17 13:45:54  false | map[alpha:{fn:0x699860 runValidatinOnNil:false} alphanum:{fn:0x699860 runValidatinOnNil:false} alphanumunicode:{fn:0x699860 runValidatinOnNil:false} alphaunicode:{fn:0x699860 runValidatinOnNil:false} ascii:{fn:0x699860 runValidatinOnNil:false} base64:{fn:0x699860 runValidatinOnNil:false} base64rawurl:{fn:0x699860 runValidatinOnNil:false} base64url:{fn:0x699860 runValidatinOnNil:false} bcp47_language_tag:{fn:0x699860 runValidatinOnNil:false} bic:{fn:0x699860 runValidatinOnNil:false} boolean:{fn:0x699860 runValidatinOnNil:false} btc_addr:{fn:0x699860 runValidatinOnNil:false} btc_addr_bech32:{fn:0x699860 runValidatinOnNil:false} cidr:{fn:0x699860 runValidatinOnNil:false} cidrv4:{fn:0x699860 runValidatinOnNil:false} cidrv6:{fn:0x699860 runValidatinOnNil:false} contains:{fn:0x699860 runValidatinOnNil:false} containsany:{fn:0x699860 runValidatinOnNil:false} containsrune:{fn:0x699860 runValidatinOnNil:false} credit_card:{fn:0x699860 runValidatinOnNil:false} cron:{fn:0x699860 runValidatinOnNil:false} cve:{fn:0x699860 runValidatinOnNil:false} datauri:{fn:0x699860 runValidatinOnNil:false} datetime:{fn:0x699860 runValidatinOnNil:false} dir:{fn:0x699860 runValidatinOnNil:false} dirpath:{fn:0x699860 runValidatinOnNil:false} dns_rfc1035_label:{fn:0x699860 runValidatinOnNil:false} e164:{fn:0x699860 runValidatinOnNil:false} email:{fn:0x699860 runValidatinOnNil:false} endsnotwith:{fn:0x699860 runValidatinOnNil:false} endswith:{fn:0x699860 runValidatinOnNil:false} eq:{fn:0x699860 runValidatinOnNil:false} eq_ignore_case:{fn:0x699860 runValidatinOnNil:false} eqcsfield:{fn:0x699860 runValidatinOnNil:false} eqfield:{fn:0x699860 runValidatinOnNil:false} eth_addr:{fn:0x699860 runValidatinOnNil:false} eth_addr_checksum:{fn:0x699860 runValidatinOnNil:false} excluded_if:{fn:0x6998c0 runValidatinOnNil:true} excluded_unless:{fn:0x6998c0 runValidatinOnNil:true} excluded_with:{fn:0x6998c0 runValidatinOnNil:true} excluded_with_all:{fn:0x6998c0 runValidatinOnNil:true} excluded_without:{fn:0x6998c0 runValidatinOnNil:true} excluded_without_all:{fn:0x6998c0 runValidatinOnNil:true} excludes:{fn:0x699860 runValidatinOnNil:false} excludesall:{fn:0x699860 runValidatinOnNil:false} excludesrune:{fn:0x699860 runValidatinOnNil:false} fieldcontains:{fn:0x699860 runValidatinOnNil:false} fieldexcludes:{fn:0x699860 runValidatinOnNil:false} file:{fn:0x699860 runValidatinOnNil:false} filepath:{fn:0x699860 runValidatinOnNil:false} fqdn:{fn:0x699860 runValidatinOnNil:false} gt:{fn:0x699860 runValidatinOnNil:false} gtcsfield:{fn:0x699860 runValidatinOnNil:false} gte:{fn:0x699860 runValidatinOnNil:false} gtecsfield:{fn:0x699860 runValidatinOnNil:false} gtefield:{fn:0x699860 runValidatinOnNil:false} gtfield:{fn:0x699860 runValidatinOnNil:false} hexadecimal:{fn:0x699860 runValidatinOnNil:false} hexcolor:{fn:0x699860 runValidatinOnNil:false} hostname:{fn:0x699860 runValidatinOnNil:false} hostname_port:{fn:0x699860 runValidatinOnNil:false} hostname_rfc1123:{fn:0x699860 runValidatinOnNil:false} hsl:{fn:0x699860 runValidatinOnNil:false} hsla:{fn:0x699860 runValidatinOnNil:false} html:{fn:0x699860 runValidatinOnNil:false} html_encoded:{fn:0x699860 runValidatinOnNil:false} http_url:{fn:0x699860 runValidatinOnNil:false} image:{fn:0x699860 runValidatinOnNil:false} ip:{fn:0x699860 runValidatinOnNil:false} ip4_addr:{fn:0x699860 runValidatinOnNil:false} ip6_addr:{fn:0x699860 runValidatinOnNil:false} ip_addr:{fn:0x699860 runValidatinOnNil:false} ipv4:{fn:0x699860 runValidatinOnNil:false} ipv6:{fn:0x699860 runValidatinOnNil:false} isbn:{fn:0x699860 runValidatinOnNil:false} isbn10:{fn:0x699860 runValidatinOnNil:false} isbn13:{fn:0x699860 runValidatinOnNil:false} isdefault:{fn:0x699860 runValidatinOnNil:false} iso3166_1_alpha2:{fn:0x699860 runValidatinOnNil:false} iso3166_1_alpha3:{fn:0x699860 runValidatinOnNil:false} iso3166_1_alpha_numeric:{fn:0x699860 runValidatinOnNil:false} iso3166_2:{fn:0x699860 runValidatinOnNil:false} iso4217:{fn:0x699860 runValidatinOnNil:false} iso4217_numeric:{fn:0x699860 runValidatinOnNil:false} json:{fn:0x699860 runValidatinOnNil:false} jwt:{fn:0x699860 runValidatinOnNil:false} latitude:{fn:0x699860 runValidatinOnNil:false} len:{fn:0x699860 runValidatinOnNil:false} longitude:{fn:0x699860 runValidatinOnNil:false} lowercase:{fn:0x699860 runValidatinOnNil:false} lt:{fn:0x699860 runValidatinOnNil:false} ltcsfield:{fn:0x699860 runValidatinOnNil:false} lte:{fn:0x699860 runValidatinOnNil:false} ltecsfield:{fn:0x699860 runValidatinOnNil:false} ltefield:{fn:0x699860 runValidatinOnNil:false} ltfield:{fn:0x699860 runValidatinOnNil:false} luhn_checksum:{fn:0x699860 runValidatinOnNil:false} mac:{fn:0x699860 runValidatinOnNil:false} max:{fn:0x699860 runValidatinOnNil:false} md4:{fn:0x699860 runValidatinOnNil:false} md5:{fn:0x699860 runValidatinOnNil:false} min:{fn:0x699860 runValidatinOnNil:false} mongodb:{fn:0x699860 runValidatinOnNil:false} multibyte:{fn:0x699860 runValidatinOnNil:false} ne:{fn:0x699860 runValidatinOnNil:false} ne_ignore_case:{fn:0x699860 runValidatinOnNil:false} necsfield:{fn:0x699860 runValidatinOnNil:false} nefield:{fn:0x699860 runValidatinOnNil:false} number:{fn:0x699860 runValidatinOnNil:false} numeric:{fn:0x699860 runValidatinOnNil:false} oneof:{fn:0x699860 runValidatinOnNil:false} postcode_iso3166_alpha2:{fn:0x699860 runValidatinOnNil:false} postcode_iso3166_alpha2_field:{fn:0x699860 runValidatinOnNil:false} printascii:{fn:0x699860 runValidatinOnNil:false} required:{fn:0x699860 runValidatinOnNil:false} required_if:{fn:0x6998c0 runValidatinOnNil:true} required_unless:{fn:0x6998c0 runValidatinOnNil:true} required_with:{fn:0x6998c0 runValidatinOnNil:true} required_with_all:{fn:0x6998c0 runValidatinOnNil:true} required_without:{fn:0x6998c0 runValidatinOnNil:true} required_without_all:{fn:0x6998c0 runValidatinOnNil:true} rgb:{fn:0x699860 runValidatinOnNil:false} rgba:{fn:0x699860 runValidatinOnNil:false} ripemd128:{fn:0x699860 runValidatinOnNil:false} ripemd160:{fn:0x699860 runValidatinOnNil:false} semver:{fn:0x699860 runValidatinOnNil:false} sha256:{fn:0x699860 runValidatinOnNil:false} sha384:{fn:0x699860 runValidatinOnNil:false} sha512:{fn:0x699860 runValidatinOnNil:false} skip_unless:{fn:0x6998c0 runValidatinOnNil:true} spicedb:{fn:0x699860 runValidatinOnNil:false} ssn:{fn:0x699860 runValidatinOnNil:false} startsnotwith:{fn:0x699860 runValidatinOnNil:false} startswith:{fn:0x699860 runValidatinOnNil:false} tcp4_addr:{fn:0x699860 runValidatinOnNil:false} tcp6_addr:{fn:0x699860 runValidatinOnNil:false} tcp_addr:{fn:0x699860 runValidatinOnNil:false} tiger128:{fn:0x699860 runValidatinOnNil:false} tiger160:{fn:0x699860 runValidatinOnNil:false} tiger192:{fn:0x699860 runValidatinOnNil:false} timezone:{fn:0x699860 runValidatinOnNil:false} udp4_addr:{fn:0x699860 runValidatinOnNil:false} udp6_addr:{fn:0x699860 runValidatinOnNil:false} udp_addr:{fn:0x699860 runValidatinOnNil:false} ulid:{fn:0x699860 runValidatinOnNil:false} unique:{fn:0x699860 runValidatinOnNil:false} unix_addr:{fn:0x699860 runValidatinOnNil:false} uppercase:{fn:0x699860 runValidatinOnNil:false} uri:{fn:0x699860 runValidatinOnNil:false} url:{fn:0x699860 runValidatinOnNil:false} url_encoded:{fn:0x699860 runValidatinOnNil:false} urn_rfc2141:{fn:0x699860 runValidatinOnNil:false} uuid:{fn:0x699860 runValidatinOnNil:false} uuid3:{fn:0x699860 runValidatinOnNil:false} uuid3_rfc4122:{fn:0x699860 runValidatinOnNil:false} uuid4:{fn:0x699860 runValidatinOnNil:false} uuid4_rfc4122:{fn:0x699860 runValidatinOnNil:false} uuid5:{fn:0x699860 runValidatinOnNil:false} uuid5_rfc4122:{fn:0x699860 runValidatinOnNil:false} uuid_rfc4122:{fn:0x699860 runValidatinOnNil:false}]
panic: Undefined validation function 'required' on field 'Subject'

goroutine 29 [running]:
github.com/go-playground/validator/v10.(*Validate).parseFieldTagsRecursive(0xc00041ad90, {0x13c1c91?, 0x167d2de?}, {0x13c1c70, 0x7}, {0x0, 0x0}, 0x0)
	/app/vendor/github.com/go-playground/validator/v10/cache.go:297 +0xade
github.com/go-playground/validator/v10.(*Validate).extractStructCache(0xc00041ad90, {0x15d6f00?, 0xc0224f43c0?, 0xc000f81ba8?}, {0x137eb47, 0x5})
	/app/vendor/github.com/go-playground/validator/v10/cache.go:156 +0x569
github.com/go-playground/validator/v10.(*validate).validateStruct(0xc01e0f0000, {0x18d7080, 0x2cc0ee0}, {0x14a9800?, 0xc0224f43c0?, 0x1685ccb?}, {0x15d6f00?, 0xc0224f43c0?, 0x19?}, {0x18eece8, ...}, ...)
	/app/vendor/github.com/go-playground/validator/v10/validator.go:37 +0x196
github.com/go-playground/validator/v10.(*Validate).StructCtx(0xc00041ad90, {0x18d7080, 0x2cc0ee0}, {0x14a9800?, 0xc0224f43c0?})
	/app/vendor/github.com/go-playground/validator/v10/validator_instance.go:393 +0x445
github.com/go-playground/validator/v10.(*Validate).Struct(0xc00930ae70?, {0x14a9800?, 0xc0224f43c0?})
	/app/vendor/github.com/go-playground/validator/v10/validator_instance.go:366 +0x31
github.com/scrapnode/kanthor/infrastructure/streaming.(*NatsSubscriberPushing).Sub.func1(0xc0010b2750?)
	/app/infrastructure/streaming/nats_subscriber_pushing.go:105 +0x6d
github.com/nats-io/nats%2ego.(*Conn).waitForMsgs(0xc0006f5100, 0xc0010b2750)
	/app/vendor/github.com/nats-io/nats.go/nats.go:3001 +0x3f2
created by github.com/nats-io/nats%2ego.(*Conn).subscribeLocked in goroutine 26
	/app/vendor/github.com/nats-io/nats.go/nats.go:4232 +0x3c8

kubeplusplus avatar Sep 17 '23 13:09 kubeplusplus

Can you please post what event struct looks like and the validations on it. I can’t really help without it.

deankarn avatar Sep 17 '23 14:09 deankarn

Is it possible the required tag contains an invisible char? I'd try deleting the tag and rewriting it just in case.

Ex.:

func TestNastyInvisibleChar(t *testing.T) {
	v := validator.New(validator.WithRequiredStructEnabled())

	type myTypeWithInvisibleChar struct {
                // The "required" tag here contains an invisible char at the end
		V string `validate:"required­"`
	}

	type myType struct {
		V string `validate:"required"`
	}

	// All good
	if err := v.Struct(myType{"potato"}); err != nil {
		t.Fatal(err)
	}

	// Panics with "Undefined validation function 'required­' on field 'V'"
	if err := v.Struct(myTypeWithInvisibleChar{"potato"}); err != nil {
		t.Fatal(err)
	}
}

MysteriousPotato avatar Sep 18 '23 00:09 MysteriousPotato

@deankarn sorry about missing the event, here it is


type Event struct {
	Subject string `json:"subject" validate:"required"`
	AppId   string `json:"app_id" validate:"required"`
	Type    string `json:"type" validate:"required"`

	Id       string            `json:"id" validate:"required"`
	Data     []byte            `json:"data" validate:"required"`
	Metadata map[string]string `json:"metadata"`
}

And an example of actual data that was dumped when the bug was happen

{
  "subject": "kanthor.default.msg.app_2VXx8Qpkd9MAaKhYhEwnBLSliYJ.testing.traffic.stability",
  "app_id": "app_2VXx8Qpkd9MAaKhYhEwnBLSliYJ",
  "type": "testing.traffic.stability",
  "id": "msg_2VXxhrM959iCT4pLFJI7TlOojZC",
  "data": "eyJpZCI6Im1zZ18yVlh4aHJNOTU5aUNUNHBMRkpJN1RsT29qWkMiLCJ0aW1lc3RhbXAiOjE2OTQ5OTYzNDc4MjUsImJ1Y2tldCI6IjIwMjMwOTE4IiwiYXR0X2lkIjoiYXR0XzJWWHhocVROZzdaMEp1cmlpdnk3MFNya0xrSiIsInRpZXIiOiJkZWZhdWx0IiwiYXBwX2lkIjoiYXBwXzJWWHg4UXBrZDlNQWFLaFloRXduQkxTbGlZSiIsInR5cGUiOiJ0ZXN0aW5nLnRyYWZmaWMuc3RhYmlsaXR5IiwibWV0YWRhdGEiOnsia2FudGhvci5pZGVtcG90ZW5jeV9rZXkiOiI0MmY1NjUwNi1lN2ZmLTQ3MGYtODBhMS01Y2Q2M2Q4MTAyNjMifSwiaGVhZGVycyI6eyJIZWFkZXIiOnsiWC1DbGllbnQiOlsiazYuaW8iXX19LCJib2R5IjoiZXlKMWMyVnlibUZ0WlNJNkluVmZkbXhpWTNwNmEzRm5jVzF2Ym1weWIyZGhZMmRyZDNKdGFuTnNaRUJyWVc1MGFHOXliR0ZpY3k1amIyMGlmUT09In0=",
  "metadata": {}
}

Some other information that may help

  • The bug only happens during my stress test (traffic grows up from 0 to 2k event/s in 15mins). Sometime it happened after 30s, sometime after 10mins the test. In normal traffic, it's just fine.
  • Sometime a bug fatal error: gcmarknewobject called while doing checkmark was raised instead of the bug I reported
  • Test environment is k3s version v1.27.4+k3s1, no limit resources (both cpu and ram)
  • Average throughput is 2652 event/s
  • Average message size is 2kb/even
  • A result from a pprof image

kubeplusplus avatar Sep 18 '23 01:09 kubeplusplus

Is it possible the required tag contains an invisible char? I'd try deleting the tag and rewriting it just in case.

Ex.:

func TestNastyInvisibleChar(t *testing.T) {
	v := validator.New(validator.WithRequiredStructEnabled())

	type myTypeWithInvisibleChar struct {
                // The "required" tag here contains an invisible char at the end
		V string `validate:"required­"`
	}

	type myType struct {
		V string `validate:"required"`
	}

	// All good
	if err := v.Struct(myType{"potato"}); err != nil {
		t.Fatal(err)
	}

	// Panics with "Undefined validation function 'required­' on field 'V'"
	if err := v.Struct(myTypeWithInvisibleChar{"potato"}); err != nil {
		t.Fatal(err)
	}
}

I have test with this debug log image And it shows the normal required text image

I also did a check with online tool Your example shows this output image And this is the mine image

Note: And the bug only happen during my stress test. In normal traffic, it's just fine

kubeplusplus avatar Sep 18 '23 01:09 kubeplusplus