PermissionDenied since Dishy update to 9f4d05a4

[MrNiceforest]

Describe the bug Since the latest Dishy update earlier this morning to 9f4d05a4-65b6-411b-ba6b-8dd91c64e02c.uterm.release the exporter is unable to collect data via gRPC. Running in a standard Starlink setup without further parameters it successfully gets the dish ID

time="2021-06-24T22:15:59+02:00" level=info msg="dish id: ut01000000-00000000-00013135" source="main.go:28"

However, each time /metrics of the exporter is queried, it returns the following:

# HELP starlink_dish_up Was the last query of Starlink dish successful.
# TYPE starlink_dish_up gauge
starlink_dish_up 0

The console/log shows the following error:

time="2021-06-24T22:15:59+02:00" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"

It worked fine with the previous release a4908729-5051-4d3b-aacc-446ef9b26cdc.uterm.release. Data is still available fine when accessing the web UI normally. I didn't have time yet to look further into it, so sorry for the somewhat unqualified bug ticket.

To Reproduce Steps to reproduce the behavior:

1. Get a Dishy with the mentioned software version
2. Run the exporter to collect data from given Dishy and fetch metrics If needed, I can provide a sample Wireshark pcapng dump of this.

Expected behavior The extractor to collect/poll the required metrics.

Screenshots Not applicable, see log excerpt above.

Additional context Add any other context about the problem here.

• OS: Windows Server 2019
• Starlink Exporter Version v0.0.3

[dwillcocks]

Ok, this sounds fun, I will have a look on the weekend, I hope they have not added authentication to all the gRPC methods ☹️

[MrNiceforest]

As an update, last night Dishy updated to 990b956d-a7fc-4c9b-a3ab-c09a309ac425.uterm.release and it works normally again, so I would assume an error (or test?) in their software. I still have the tcpdump file if you still want to look into it, but otherwise I think this can be closed in the meantime.

[awlx]

I have the same error with 4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release:

starlink_exporter_1   | time="2021-06-25T07:56:28Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:31Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:34Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:37Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:40Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:43Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:46Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"
starlink_exporter_1   | time="2021-06-25T07:56:49Z" level=error msg="failed to collect context from dish: rpc error: code = PermissionDenied desc = " source="exporter.go:290"

I started this morning as you can see on my public dashboard (https://starlink.awlnx.space).

[dwillcocks]

I'm on 4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release and seeing the same error as @awlx. I will look into why but it looks like Starlink have fixed the issue in release 990b956d-a7fc-4c9b-a3ab-c09a309ac425.uterm.release.

They have been making lots of additional gRPC methods available but none of them seemed usable yet. Maybe they tried to deprecate the old methods but have reverted that in the latest dish release.

[virtuallynathan]

It still seems like having this merged wouldn't hurt anything in the case that they do revert it? All of my data collection is now broken. Also, FWIW, 990b956d-a7fc-4c9b-a3ab-c09a309ac425.uterm.release was released before 4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release

[dwillcocks]

4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release

@virtuallynathan would you say this is the latest release, as far as you can tell?

I'm looking at it now, and splitting out too different gRPC methods and will add the fail open logic to all calls so it will return what it can.

[dwillcocks]

I have merged and built @awlx PR - https://github.com/danopstech/starlink_exporter/releases/tag/v0.0.4

Thank you @awlx

This gives me more time to work around the permissions:

1. Get data from a different gRPC method
2. Add authentication - bonus we then get location data once authenticated

[virtuallynathan]

Yes, there’s a community maintained spreadsheet here: https://docs.google.com/spreadsheets/d/1nsdLZ34VVX1qNVlDlAErzLov-fb_ZWgpYAQJWp_W8ic/edit

Great, re: merge!

On Wed, Jun 30, 2021 at 6:01 AM Daniel Willcocks @.***> wrote:

4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release

@virtuallynathan https://github.com/virtuallynathan would you say this is the latest release, as far as you can tell?

I'm looking at it now, and splitting out too different gRPC methods and will add the fail open logic to all calls so it will return what it can.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/danopstech/starlink_exporter/issues/32#issuecomment-871383817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2PWLOWE5L4RMMLJDZLUE3TVMITDANCNFSM47IT4BJQ .

[MrNiceforest]

By the way, one day after my post about the update to 990b956d-a7fc-4c9b-a3ab-c09a309ac425.uterm.release and it working again, Dishy updated to 4990ce8d-5028-4e51-a015-e9ab1b1ebe1a.uterm.release which broke it again, but the fix from @awlx worked fine - also as the recent v0.0.4 of course.

[dwillcocks]

Response from support 😢

Our engineering teams will be continuously adjusting the availability and security of both our published and unpublished APIs throughout the development of Starlink. If you have any further feedback, please let us know in next survey sent out to you.

I will look at fixing the missing metrics. just hope they don't start locking everything down!