Prometheus not accessible

[Daemonslayer2048]

I was trying out this niffty project and when I tried to access the prometheus instance at :9090/metrics I end up with an "unable to connect" message. I also noticed there does not appear to be anything in the docker command to tell port 9090 to bind to anything so maybe I am missing something? I should also note I had do have cockpit running on this machine (which also uses port 9090) however I did disable it for testing. Below is the exact command used, am I mssing anything?

docker run   
    --network=bridge                           
    --publish 8443:4443                         
    --publish 8080:8080                          
    --volume `pwd`/certs:/certs:ro             
    --volume `pwd`/datadir:/datadir            
    --read-only                               
    --user `id -u`:`id -g`                     
    --cap-drop=all                             
    ndt-server                                 
    -cert /certs/cert.pem                      
    -key /certs/key.pem                        
    -datadir /datadir                          
    -ndt7_addr :4443                           
    -ndt7_addr_cleartext :8080

Also, I did notice the ndt5 test loads but can not run, tested on both firefox, and chrome and firefox simply loads the page and does nothing. Chrome fails with "Test failed: 152". I am happy to afdop the v7 if its the future but it sounds like v5 still has more features. Please let me know if this side note should be a different issue.

Thank you for reading this far.

[robertodauria]

Thank you for reporting this!

To access the Prometheus port, you should add an additional --publish 9990:9990 to the above command. e.g.

docker run --network=bridge                \
           --publish 443:4443              \
           --publish 80:8080               \
           --publish 9990:9990             \
           --volume `pwd`/certs:/certs:ro  \
           --volume `pwd`/datadir:/datadir \
           --read-only                     \
           --user `id -u`:`id -g`          \
           --cap-drop=all                  \
           ndt-server                      \
           -cert /certs/cert.pem           \
           -key /certs/key.pem             \
           -datadir /datadir               \
           -ndt7_addr :4443                \
           -ndt7_addr_cleartext :8080

Then http://localhost:9990/metrics should work.

I think the reason it's not in the README is that not everyone wants the Prometheus metrics to be externally accessible by default, but it might also be an oversight.

Regarding the NDT test, I've experimented with it a bit and I've been able to make it work by changing --network=bridge to --network=host. That will make every port the ndt-server daemon listens on accessible from the host, and you can also remove the --publish flags as they'd be ignored by Docker anyway. e.g.

docker run --network=host                \
           --volume `pwd`/certs:/certs:ro  \
           --volume `pwd`/datadir:/datadir \
           --read-only                     \
           --user `id -u`:`id -g`          \
           --cap-drop=all                  \
           ndt-server                      \
           -cert /certs/cert.pem           \
           -key /certs/key.pem             \
           -datadir /datadir               \
           -ndt7_addr :4443                \
           -ndt7_addr_cleartext :8080

The reason for that seems to be that NDT listens on a different (random) port every time a new test is requested, and that port must be accessible from the test client. That does not work well with --publish and a static list of ports.

The ndt7 protocol is much simpler in many ways, and one of them is that it only uses one HTTP and one HTTPS port.

[Daemonslayer2048]

Your comment regarding Promethus make sense, I get why everyone may not want that accessible. Also I kept tying to publish to port 9090, I apparently need to read documentation better next time.

Is the port range NDT uses configurable? Could I set an upper and lower limit?

[robertodauria]

Is the port range NDT uses configurable? Could I set an upper and lower limit?

It's not possible with just command line flags. In the ndt5 protocol code, there are several points where a random TCP port is used, such as https://github.com/m-lab/ndt-server/blob/b4f7984895461c3ed8d84b1ba82e4fd1ff22702a/ndt5/singleserving/server.go#L117

If you wanted to instead have your instance of ndt-server use explicitly defined port ranges, you would have to find all instances in the code and maintain those changes in your fork. @pboothe @soltesz would you have more to add about where the additional code changes would need to be made?

On a separate note: if you would be willing to expand on what you think is missing from the ndt7 protocol, I think it would be good feedback for the team.

[pboothe]

We would love to hear about what is preventing you from using v7. It's better all-around from our perspective, and we want to move people to it and away from ndt5.

[Daemonslayer2048]

My issue so far is not with ndtv7 protocol so far as I can tell. I have more of an issue with the clients for v5 and v7. I have been playing with the clients in a lab like setup on some ARM chip sets. From what I have found so far the v7 go client on ARM seems to peg the CPU harder and produces lower results, than the v5 client. I am not sure if this is the right forum exactly to talk about this but I would love to share my findings (in the correct location) if you would find it useful.

[bassosimone]

We know that Go code does not have crypto assembly for armv7. Can you check whether you have similar issues with arm64? Can you check whether using the cleartext ndt7 is better?

[pboothe]

This is a perfectly fine place to talk about it. Both Simone and I are on the thread, and we are the two people who are most likely to actually implement the fix.