opcua icon indicating copy to clipboard operation
opcua copied to clipboard

Working Server Example

Open danomagnum opened this issue 8 months ago • 27 comments

I've been working on getting the server working and I've got a version of it working on the server branch of my fork here.

There is still quite a bit to implement to get it production ready, but as a proof-of-concept it is working. I've been (manually) testing it using ignition and pyopcua.

I tried to keep as much of the new code in /examples/server/ as I could.

What (mostly) works:

  • Browse
  • Read Attribute (value, type)
  • Write Attribute (value)
  • Create Subscription
  • Create Monitored Items
  • Publish (values only. timed only - not event based/ on change)

I would like to contribute this back to the project, but I've got to do quite a bit of refactoring first and would like some recommendations on what the refactored code should look like before I start on that so that I don't have to refactor it again to match what you all are looking for in the server code.

As an example, one thing that will have to change is I am ignoring the concept as nodes as much as possible - I've got a map[string]any providing the backed and only using string node IDs as the keys to the map. This is a lot more convenient for the actual use case I've got that prompted me to start working on this so I'd like to keep that functionality in the end, but I think I should be able to make a pseudo-node that would hold the map and provide the references, etc... to make it invisible to the server. Maybe.

Anyway, like I said I'm really looking for feedback, suggestions, and recommendations on what you'd like to see. Obviously my use case is really what I need to achieve but if I can make it work for merging back in that would be a win-win.

danomagnum avatar Dec 08 '23 19:12 danomagnum

I ended up splitting the address space up by namespace and made a namespace interface so different back-ends can be added at the namespace level. I've ported the browse and read/write attribute handlers back into the main server and it is working for regular nodes and my map-based "nodes".

The pub/sub stuff is next, but I'll have to think about the best way to do that for a bit before I start moving it back in.

danomagnum avatar Dec 12 '23 04:12 danomagnum

Wow 🥇 Thank you @danomagnum !!! That might be the push to get us off the ground with the server. I'll have a look.

magiconair avatar Dec 14 '23 08:12 magiconair

CC: @kung-foo

magiconair avatar Dec 14 '23 08:12 magiconair

So I've got quite a bit of it working now. I've re-integrated it back into the server module of the library - nothing in the example folder except for an actual example using the library to host a server now.

Assuming the simplest use case is the most common, I think it's pretty usable at this point.

I tested it now against ignition's UA client, pyopc's client, and UaExpert. They're all reading/writing/subscribing to data and working correctly.

I'm sure there are still bugs to work out of course.

Once you get a chance to look at it let me know and I'll get it merged back up to the latest master client revision and I can do a pull request.

Here's the current status of all the services in the ua spec:

Discovery Service - unchanged (assumed working)
	Find Servers
	Find Servers On Network
	Get Endpoints
	Register Server
	Register Server 2
Secure Channel Service - unchanged (assumed working)
	Open Secure Channel
	Close Secure Channel
Session Service - Unchanged (assumed working)
	Create Session
	Activate Session
	Close Session
	Cancel
Node Management Service - Not implemented from the UA api - you can do it through the go function calls though.
	Add Nodes
	Add References
	Delete Nodes
	Delete References
View Service
	Browse - Working.  Does not support partial browse/BrowseNext.
	BrowseNext - Not implemented
	TranslateBrowsePathsToNodeIds - Not implemented
	RegisterNodes - Not implemented
	UnregisterNodes - Not implemented
Query Service - Not implemented
	QueryFirst
	QueryNext
Attribute Service
	Read - Working
	HistoryRead - Not implemented
	Write - Working
	HistoryUpdate - Not implemented
Method Service - Not implemented, but it would be easy enough to add by  adding a Call method to the Namespace interface.
	Call
Monitored Item Service
	Create Monitored Items - Working
	Modify Monitored Items - Working
	Set Monitoring Mode - Working
	Set Triggering - Not implemented
	Delete Monitored Items - Working
Subscription Service
	Create Subscription - Working
	Modify Subscription - Working
	Set Publishing Mode - Not implemented
	Publish - Not implemented
	Re-Publish - Not implemented
	Transfer Subscriptions - Not implemented
	Delete Subscriptions - Working

danomagnum avatar Dec 15 '23 20:12 danomagnum

The thing I got stuck at was to get the basic node structure in the prosys browser. The Root, Objects, Types, Views and so forth. Maybe you can get this to work and then I'm all for merging this back to main and marking this experimental. But I'd like people to play with this and drive this forward. So once we have a basic programming model where someone can write a server and make it do something then we can iterate on this.

gopcua server

image

standard opcua server

image

magiconair avatar Dec 18 '23 16:12 magiconair

I think you need to implement browsing for this.

magiconair avatar Dec 18 '23 16:12 magiconair

So browsing (edit: partially - no continuation/browsenext) works, but you've got to have all the node references defined for it to know what to return as the browse results.

If I manually set them up, they start showing up.

// add the namespaces to the server, and add a reference to them
	root_ns, _ := s.Namespace(0)
	root_obj := root_ns.Root()

	// add the "Types" folder
	tf := root_ns.Node(ua.NewNumericNodeID(0, 86))
	root_obj.AddRef(tf)
	dtf := root_ns.Node(ua.NewNumericNodeID(0, 90))
	tf.AddRef(dtf)
	base_type := root_ns.Node(ua.NewNumericNodeID(0, 24))
	dtf.AddRef(base_type)
	bool_type := root_ns.Node(ua.NewNumericNodeID(0, 1))
	ByteString_type := root_ns.Node(ua.NewNumericNodeID(0, 15))
	DataValue_type := root_ns.Node(ua.NewNumericNodeID(0, 23))
	DateTime_type := root_ns.Node(ua.NewNumericNodeID(0, 13))
	DiagnosticInfo_type := root_ns.Node(ua.NewNumericNodeID(0, 25))
	Enumeration_type := root_ns.Node(ua.NewNumericNodeID(0, 29))
	Uint32_type := root_ns.Node(ua.NewNumericNodeID(0, 7))
	dtf.AddRef(bool_type)
	dtf.AddRef(DataValue_type)
	dtf.AddRef(ByteString_type)
	dtf.AddRef(DateTime_type)
	dtf.AddRef(DiagnosticInfo_type)
	dtf.AddRef(Enumeration_type)
	dtf.AddRef(Uint32_type)


	root_obj = root_ns.Objects()

	server_obj := root_ns.Node(ua.NewNumericNodeID(0, 2253))
	root_obj.AddRef(server_obj)

image

I only did a handful of the references to see if it worked or not. There is an absolute ton of references that are needed to get everything filled in properly. I'll have to take a look at how nodes_gen is being generated to see if we can build those references automatically.

danomagnum avatar Dec 18 '23 21:12 danomagnum

I made an adjustment to the generation file and while it's not perfect yet it's browsing the pre-defined nodes automatically now. Need to do some more investigation on why they are showing up as X's instead of folder icons. There are some loops in the references too that need to be sorted out.

image

danomagnum avatar Dec 18 '23 23:12 danomagnum

How about merging your changes on top of our server branch and giving you access to this repo? Then we can collaborate on your changes and I can help with refactoring. The collaboration on branches is something I really miss from Gerrit unless I'm missing something. I'll send you an invite.

magiconair avatar Jan 02 '24 13:01 magiconair

To get the full nodes generated we should probably import a NodeSpec2.xml file and generate the code from that. Sooner or later we need to add support for importing a node set anyway.

magiconair avatar Jan 02 '24 13:01 magiconair

@danomagnum I've sent you an invite. You should use the second one since I've canceled the first one.

magiconair avatar Jan 02 '24 13:01 magiconair

Invite accepted.

I'll try and get the changes merged properly onto the server branch in the next couple days and I'll let you know. I will probably bring in the changes from the main branch too, just so the server branch gets caught back up.

I've been away from this project for the holidays but intend to pick it back up in the next week or two. Before the break I was able to get a lot (but still not all) of the auto-import from xml working using the PredefinedNodes.xml file (which looks like it is generated by the opc-ua dotnet project from the nodeset2.xml file?). At one point the generated .go file it created was too large and the compiler choked on it so I had to refactor how it was generating the cross-references to re-use references instead of generating them anew for each node.

The strangest thing going on right now is that the FolderType reference is showing up as a child item when browsing, even though as far as I can tell there isn't any difference between the packets gopcua is sending and the packets other similar opc servers send. So far getting the predefined nodes more correct has fixed this kind of thing so that's where I'm betting the problem is.

As you suggest, It may probably be easier to get everything correct if we start with the nodeset2.xml file directly instead of going through the generated xml file.

danomagnum avatar Jan 02 '24 18:01 danomagnum

Maybe we can embed the Nodeset2.xml file and parse it on startup.

magiconair avatar Jan 03 '24 07:01 magiconair

Switching to the nodeset2 xml file was a good move since there is a schema definition file available and I was able to auto-generate the go structs for it using xgen which saved a lot of time parsing it. I feel much more confident the references are correct using this method.

It is embedded in the schema sub package and is now loading on server startup instead of using the generated file.

Unfortunately, the strange behavior with the FolderType showing up under every folder during browse with some clients remains.

I merged everything into the server branch of the main repo.

So far I had been testing without encryption so I could easily wireshark the packets. I tried using encryption today and that was not an immediate success so there may be some work to do there also.

danomagnum avatar Jan 04 '24 03:01 danomagnum

Awesome. I'll have a look

magiconair avatar Jan 04 '24 08:01 magiconair

Hmm, it doesn't compile. Did you push all of your changes?

frank@piet2 ~/s/g/g/opcua (server)> make
go test -count=1 -race ./...
# github.com/gopcua/opcua/uasc
package github.com/gopcua/opcua/uasc
	imports github.com/gopcua/opcua/uatest
	imports github.com/gopcua/opcua/server
	imports github.com/gopcua/opcua/uasc: import cycle not allowed in test
FAIL	github.com/gopcua/opcua/uasc [setup failed]
# github.com/gopcua/opcua
./client.go:766:54: cannot use func(v interface{}) error {…} (value of type func(v interface{}) error) as uasc.ResponseHandler value in argument to c.SecureChannel().SendRequest
./client.go:879:77: cannot use func(v interface{}) error {…} (value of type func(v interface{}) error) as uasc.ResponseHandler value in argument to c.SecureChannel().SendRequest
./client.go:942:66: cannot use h (variable of type func(interface{}) error) as uasc.ResponseHandler value in argument to c.sendWithTimeout
./client_sub.go:181:88: cannot use func(v interface{}) error {…} (value of type func(v interface{}) error) as uasc.ResponseHandler value in argument to c.SecureChannel().SendRequest

magiconair avatar Jan 04 '24 09:01 magiconair

I'm here.

commit 1ddfa35 (HEAD -> server, origin/server)
Author: danomagnum <[email protected]>
Date:   Wed Jan 3 21:01:03 2024 -0600

    Added server info to the readme.md

magiconair avatar Jan 04 '24 09:01 magiconair

Which timezone are you in? Just for my info. I'm Europe/Stockholm (UTC+1)

magiconair avatar Jan 04 '24 09:01 magiconair

I see. I haven't been building everything with make, just doing go run in the specific examples I was testing. Looks like when I merged in the main to the server branch I didn't get the change to the ua.Response interface everywhere somehow. Hopefully all taken care of now.

I'm in US central (UTC-6).

danomagnum avatar Jan 04 '24 14:01 danomagnum

Getting test timeouts

Listening on 0.0.0.0:4841
--- FAIL: TestNamespace (10.08s)
panic: opcua: timeout [recovered]
	panic: opcua: timeout

goroutine 54 [running]:
testing.tRunner.func1.2({0x102fb5c40, 0xc00000e408})
	/Users/frank/sdk/go1.21.0/src/testing/testing.go:1545 +0x274
testing.tRunner.func1()
	/Users/frank/sdk/go1.21.0/src/testing/testing.go:1548 +0x448
panic({0x102fb5c40?, 0xc00000e408?})
	/Users/frank/sdk/go1.21.0/src/runtime/panic.go:920 +0x26c
github.com/gopcua/opcua/uatest.NewPythonServer({0x102ebe80b, 0xc})
	/Users/frank/src/github.com/gopcua/opcua/uatest/py_server.go:46 +0x124
github.com/gopcua/opcua/uatest.TestNamespace(0xc0002149c0)
	/Users/frank/src/github.com/gopcua/opcua/uatest/namespace_test.go:18 +0x3c
testing.tRunner(0xc0002149c0, 0x102ffd420)
	/Users/frank/sdk/go1.21.0/src/testing/testing.go:1595 +0x198
created by testing.(*T).Run in goroutine 1
	/Users/frank/sdk/go1.21.0/src/testing/testing.go:1648 +0x5dc

magiconair avatar Jan 04 '24 15:01 magiconair

Looks like the integration server python file was changed after the initial server branch was created to use port 4841 instead of 4840. I set it back and it should be OK now.

danomagnum avatar Jan 04 '24 17:01 danomagnum

Heya, so just wondering - is there anyone working on this feature rn? There is a draft PR for it but that's from a different branch. But the code on the server branch looks very close to finished, at least at first glance. So what's the status on this? Any progress?

megakoresh avatar Apr 04 '24 06:04 megakoresh

I haven't worked on it in a little bit because it's (the server branch) mostly implemented enough for my needs. I've had it running in a non-critical production application for a couple months now with no issues.

I didn't ever comment about it in this issue, but I did fix the FolderType bug and everything looks pretty good now when browsing from a client. Read, Write, Subscribe, and Browse all work, so it's pretty functional.

The two things I was working on when I got pulled in a different direction were:

  • Getting the integration tests set up using gopcua for both client and server (in addition to rather than instead of the python opc server it is currently using).
  • Getting the crypto working properly.

I was able to change a couple lines in the code and get specific encryption/signing methods to work, but I wasn't having much luck picking up on the correct method programmatically.

If you need a server, you should be able to specifically pull the server branch - it's up to date with master except for some readme changes.

I'm open to issuing a pull request to merge it into the main branch if everyone else is good with it.

danomagnum avatar Apr 04 '24 12:04 danomagnum

I would love for that branch to be merged. If you know of some issues with it, then why not just merge it to main and immediately create the issues so they can be tracked? And if the server is already working in production for you, then IMO it's good to go as a first version. Better than not having it at all. Just put a usage example to the examples folda and if the crypto is not fully working, it can just be marked with panic("unimplemented")

megakoresh avatar Apr 05 '24 15:04 megakoresh