[systemcore] TimeSyncServer not connecting

[pjreiniger]

Describe the bug After finally having the motivation to figure out how to get photonvision building in our alpha testing repo, we get a timesync error, which causes the the photonvision updates to get bounced when trying to add them to the pose estimator.

PS: Consolidated documentation / publishing the 2027 builds to maven would be super helpful. I had to look at 2 month old discord messages.

5:15:06.670 PM
Warning  1  PhotonVision coprocessor at path /photonvision/Right Camera is not connected to the TimeSyncServer? It's been 9223372036854.78s since the coprocessor last heard a pong.

Check /photonvision/.timesync/{COPROCESSOR_HOSTNAME} for more information.  org.photonvision.PhotonCamera.checkTimeSyncOrWarn(PhotonCamera.java:346) 

To Reproduce

Platform:

• Orange pi powered by VRM
• photonlib: v2025.3.2-39-gcf7413a1. Incorperated into build with checked in jars opposed to vendordep
• systemcore set to a static IP of 10.35.4.2

[samfreund]

Your version of photon-lib seems to be out of sync with the version of PV that you're running on your opi. Can you try updating such that they're the same and see if that helps?

[pjreiniger]

What jar do I need to put on the orange pi? The latest build failed some parts so I'm not sure if that artifact actually gets produced or where to find it

[pjreiniger]

PS we did get it working by hacking the timestamp and just ignoring the latency.

     @Override
     public void addVisionMeasurement(Pose2d visionRobotPoseMeters, double timestampSeconds, Matrix<N3, N1> stds) {
-        super.addVisionMeasurement(visionRobotPoseMeters, Utils.fpgaToCurrentTime(timestampSeconds), stds);
+        // TODO weird
+        double myTimestamp = Microseconds.of(NetworkTablesJNI.now()).in(Seconds);
+        super.addVisionMeasurement(visionRobotPoseMeters, myTimestamp, stds);
         m_oldPoseEstimator.addVisionMeasurement(visionRobotPoseMeters, timestampSeconds, stds);
     }

[samfreund]

What jar do I need to put on the orange pi? The latest build failed some parts so I'm not sure if that artifact actually gets produced or where to find it

There's a new build running right now, so you should be able to grab the artifacts from that one when it finishes.

[pjreiniger]

I don't think a new build of the 2027 branch got kicked off, did you mean grab the latest from the main branch that ran an hour ago?

[samfreund]

I don't think a new build of the 2027 branch got kicked off, did you mean grab the latest from the main branch that ran an hour ago?

Oh sorry, I was thinking of the main branch. Let me start a new build on 2027 for you.

[samfreund]

Upon further investigation, it looks like our gradle build on the 2027 branch is broken. I'd recommend using the dev build from the main branch.

[Gold856]

Yeah, so 2027 has kinda just been me YOLOing it. I haven't felt too strongly about fixing the dependencies in favor of trying to migrate to a more long term solution (upstreaming mrcal_java to WPILib.) I fixed up the photonlib build so teams could at least use it to communicate with their PV coprocs, but haven't made the full push to get the JAR building. The version difference shouldn't be an issue, so I'm wondering what NT looks like. Could you post what's under the /photonvision/.timesync table?

[mcm001]

I think I'd need to see a wireshark/tcpdump capture of the time sync traffic from the coprocessor to SC to see if we're seeing pings but not pongs, or what's up. The protocol is pretty trivial, just two messages, so debugging shouldn't be too bad

[pjreiniger]

I went out of town so I won't be able to try anything till Thursday, but I can run tcpdump if it is preinstalled on systemcore and get a wpilog with the NT info

[pjreiniger]

I had to rename them to .txt for github to accept them.

Each hostname is incrementing its TX ping. Systemcore doesn't have tcpdump installed so I got onto one of the orangepi's and ran the command there

FRC_20251016_214604.wpilog.txt timesync.pcap.txt

[Gold856]

Could you also download the logs from the PV coproc? The TimeSyncManager will print out a debug message when the client connects to the server, so I'm curious if that shows up ever. Based on this, it seems like it doesn't, and I might need to dig into ntcore's behavior to how connections are being reported.

[pjreiniger]

Downloaded from the UI. There is a color camera that is no longer on the robot, so the logs might be pretty noisy trying to connect to that.

photonvision-logs.zip

[mcm001]

So why isn't the robot program running a timesync server?

[mcm001]

Can you set a breakpoint here: https://github.com/PhotonVision/photonvision/blob/373b61024e0aed5b0e980560fdb17c0ada4b54d4/photon-lib/src/main/java/org/photonvision/PhotonCamera.java#L168C1-L169C34

        // HACK - start a TimeSyncServer, if we haven't yet.
        TimeSyncSingleton.load();

And make sure that that loads and starts running?