make error: resty.go:13:2: found packages publicsuffix ...

[colemar]

Since latest released stc does not implement events subcommand, I am tring to build it:

$ git clone https://github.com/tenox7/stc.git
Cloning into 'stc'...
remote: Enumerating objects: 388, done.
remote: Counting objects: 100% (78/78), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 388 (delta 39), reused 45 (delta 18), pack-reused 310 (from 1)
Receiving objects: 100% (388/388), 67.76 KiB | 9.68 MiB/s, done.
Resolving deltas: 100% (233/233), done.

$ cd stc/

$ make
go build -ldflags "-X main.GitTag=1.5.3-3-g0f55212" -o stc .
../../gopath/pkg/mod/github.com/go-resty/resty/[email protected]/resty.go:13:2: found packages publicsuffix (example_test.go) and main (gen.go) in /home/admin/gopath/pkg/mod/golang.org/x/[email protected]/publicsuffix
make: *** [Makefile:7: stc] Error 1

[tenox7]

events command was added in this PR https://github.com/tenox7/stc/pull/12 I have made a release with this, can you double check?

[colemar]

$ ./stc-amd64-linux -version
stc version 1.5.4

$ ./stc-amd64-linux events | jq '.[0]'
{
  "id": 21043,
  "globalID": 21090,
  "time": "2025-02-13T00:15:27.494503753+01:00",
  "type": "DownloadProgress",
  "data": {
    "qthtp-fwoqb": {
      "The Last Ship_ The Complete Series_S04E09_Detect, Deceive, Destroy.mp4": {
        "total": 1505,
        "reused": 0,
        "copiedFromOrigin": 0,
        "copiedFromOriginShifted": 0,
        "copiedFromElsewhere": 0,
        "pulled": 859,
        "pulling": 17,
        "bytesDone": 900262923,
        "bytesTotal": 1577294178
      },
      "The Last Ship_ The Complete Series_S04E10_Endgame.mp4": {
        "total": 1219,
        "reused": 0,
        "copiedFromOrigin": 0,
        "copiedFromOriginShifted": 0,
        "copiedFromElsewhere": 0,
        "pulled": 38,
        "pulling": 18,
        "bytesDone": 39837825,
        "bytesTotal": 1277955474
      }
    }
  }
}


$ ./stc-amd64-linux events DownloadProgress
[]

Is the last syntax correct?

[tenox7]

Not sure. @tombh can you elaborate? Thank you!

[tombh]

That's definitely the correct syntax. I just rebased main and tried on my local machine and go run . events LocalChangeDetected is working. So I have 2 suggestions:

1. Did the download finish before you ran ./stc-amd64-linux events DownloadProgress?
2. The API docs definitely mention DownloadProgress as a valid type: https://docs.syncthing.net/events/downloadprogress.html But maybe the API endpoint itself isn't accepting or returning the data? To verify that would require forming and running the raw API request yourself, which I'm not sure how to do. I wonder if stc has a debugging option for that? Or perhaps you can setup https://mitmproxy.org

[colemar]

@tombh

1. Yes, no downloads were active.
2. The command stc events DownloadProgress just waits for about 1 minute, then it spits out an empty json array. stc events LocalChangeDetected does the same.

I decided to use httptap to tap into stc dialog:

It seems that, while events works ok, events LocalChangeDetected just sits there doing nothing.

[tombh]

1. Maybe there's a misunderstanding, my thinking was that if there isn't an active download then that's the reason nothing shows up in the response.
2. My idea there was that if you could recreate the API requests outside of stc, say with curl, then you could verify if the issue was in stc or in Syncthing itself.

But from what you're reporting I think it may just be that you don't have any DownloadProgress or LocalChangeDetected in the moment of the request. Or can you verify that ./stc-amd64-linux events always works and ./stc-amd64-linux events DownloadProgress always doesn't, even when they're run at the same time?

[colemar]

Sorry, I don't follow your reasoning. There is evidence (see above) that stc events shows all the possible events, including a DownloadProgress event with two folders. How can you conjecture that I don't have any DownloadProgress?

[tombh]

Sorry for not being clear.

I know it's unlikely, but there is at least a small possibility that you happen to run ./stc-amd64-linux events during an active download but then run ./stc-amd64-linux events DownloadProgress after the download has finished.

Considering how unlikely that is, it is of course very curious that ./stc-amd64-linux events shows the events but ./stc-amd64-linux events DownloadProgress doesn't. So my next thought is who is failing to return the events, is it stc or Syncthing. That is where a curl request and response would come in useful.

[colemar]

$ stc events | md5sum
053d1e06b4c87fe7590b89300935f695  -

# https://docs.syncthing.net/rest/events-get.html
# https://docs.syncthing.net/dev/rest.html

# As shown by md5sum, this API call gives the same output as [stc events]
$ curl -s -X GET -H "X-API-Key: Qwhatever9" http://localhost:8384/rest/events | md5sum
053d1e06b4c87fe7590b89300935f695  -

# Let see the available event types by filtering the json output with jq
$ curl -s -X GET -H "X-API-Key: Qwhatever9" http://localhost:8384/rest/events | jq '.[].type'
"Starting"
"StartupComplete"
"ListenAddressesChanged"
"ListenAddressesChanged"
"StateChanged"
"StateChanged"
"StateChanged"
"DeviceConnected"
"ClusterConfigReceived"
"FolderSummary"
"FolderCompletion"
"DeviceDiscovered"
"DeviceDiscovered"
"ListenAddressesChanged"
"ConfigSaved"
"ConfigSaved"
"ClusterConfigReceived"
"DeviceDisconnected"
"DeviceConnected"
"ClusterConfigReceived"
"DeviceDisconnected"
"DeviceConnected"
"ClusterConfigReceived"
"DeviceDisconnected"
"DeviceConnected"
"ClusterConfigReceived"
"DeviceDisconnected"
"DeviceConnected"
"ClusterConfigReceived"
"DeviceDisconnected"
"DeviceConnected"
"ClusterConfigReceived"

# Let see the StartupComplete event and also see how much time the API call takes
$ time curl -s -X GET -H "X-API-Key: Qwhatever9" "http://localhost:8384/rest/events?events=StartupComplete"
[]

real    1m0.013s
user    0m0.000s
sys     0m0.012s

After a full minute the API call returns an empty array. We can conclude that this is a Synchting's issue.

[colemar]

In case you wonder about the StartupComplete event, here it is:

$ curl -s -X GET -H "X-API-Key: Qwhatever9" "http://localhost:8384/rest/events" | jq '.[1]'
{
  "id": 2,
  "globalID": 2,
  "time": "2025-02-14T22:33:13.374990962+01:00",
  "type": "StartupComplete",
  "data": {
    "myID": "OWSULVX-HC6RKID-VVBHPQ2-OV2QVFB-7XL5GCW-ZI2A5MV-KFK5YOL-DM4X4Q5"
  }
}

The output of curl is a json (zero based) array, where each element is an event. jq '.[1]' selects the array element with index 1 (that is, the second one in the array).

The API call /rest/events?events=StartupComplete could block and wait for 60 seconds, but not when the parameter "since" is omitted, If I understood well this excerpt: The optional parameter since=<lastSeenID> sets the ID of the last event you’ve already seen. Syncthing returns a JSON encoded array of event objects, starting at the event just after the one with this last seen ID. The default value is 0, which returns all events.

[colemar]

I see now that the intent of this kind of API call is poorly documented, and the above behaviour is (perhaps) expected: https://github.com/syncthing/syncthing/issues/8902

Nonetheless, I cannot explain why the second call below blocks for 60 seconds and then returns nothing, given that the first gives a ClusterConfigReceived with id=32:

$ time curl -s -X GET -H "X-API-Key: Q...9" "http://localhost:8384/rest/events?since=31"
[
  {
    "id": 32,
    "globalID": 32,
    "time": "2025-02-15T22:11:43.848897534+01:00",
    "type": "ClusterConfigReceived",
    "data": {
      "device": "DLCFWAV-5YIL3ZS-DDYTSKS-AOE3X4E-ZTJGWTW-MFI6IIV-LJFHKSQ-YKFO2AJ"
    }
  }
]

real    0m0.052s
user    0m0.007s
sys     0m0.000s

$ time curl -s -X GET -H "X-API-Key: Q...9" "http://localhost:8384/rest/events?since=31&events=ClusterConfigReceived"
[]

real    1m0.029s
user    0m0.006s
sys     0m0.006s

Also see my post in the Syncthing's forum.

[tombh]

Oh, I didn't know that. Yes the docs are confusing! So after all that you still can't see your download progress? Basically what they're saying is that you only get one chance to subscribe and listen? Once the event has passed its irretrievable?

[colemar]

Oh, I didn't know that. Yes the docs are confusing! So after all that you still can't see your download progress? Basically what they're saying is that you only get one chance to subscribe and listen? Once the event has passed its irretrievable?

It seems so.

When the (special) API call goes through the Syncthing server generates a subscription where the first event (of the requested type) to happen will have id=1 (or perhaps id=0). If the API call is asking for since=31 then it will have to wait for 30 events to be generated before completing and returning an answer. If the API call omits the parameter since then this is equivalent to since=0 and it will have to wait for the next event to be generated before completing and returning an answer. I believe a second API call will be matched with the same subscription (but I don't know how, is the server referring the same subscription irrespective of who calls or where from the API call comes?), so you have at least the chance to recover all the events happened after the first call with that same event type.

If I understand correctly.

[tombh]

I share your curiosity about how it remembers you for the second request. I guess it can only do it with the IP address as there's no cookies or anything. I mean, if it even remembers you, maybe it assumes there's only one possible subscriber in the world.

I'm a bit lost I must admit.

What I can tell you is that the stc code always adds since=0 and limit=0 if they're not set. So maybe that's not helping? But it sounds like it's confusing even with curl 😬