Cyanite does not return all metrics when query multi metrics

[avneradania]

When querying Graphite-API to get multi metrics, it returns only one metric.

Example of tcpdump output:

# wget -O - 'http://localhost:8066/render?from=-15min&until=now&t**arget=AIO.DIAG.*.cpu.cpu*.usage**&format=csv'
--2017-07-23 16:07:03--  http://localhost:8066/render?from=-15min&until=now&target=AIO.DIAG.*.cpu.cpu*.usage&format=csv
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8066... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8066... connected.
HTTP request sent, awaiting response... <B^@
GET /paths?query=AIO.DIAG.%2A.cpu.cpu%2A.usage HTTP/1.1
Host: 127.0.0.1:18080
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.7.0 CPython/2.7.5 Linux/2.6.32-504.30.3.el6.x86_64
HTTP/1.1 200 OK
Content-Type: application/json
[{"text":"usage","id":"**AIO.DIAG.diag2.cpu.cpu1.usage**","path":"AIO.DIAG.diag2.cpu.cpu1.usage","allowChildren":false,"expandable":false,"leaf":true},{"text":"usage","id":"**AIO.DIAG.diag2.cpu.cpu.usage**","path":"AIO.DIAG.diag2.cpu.cpu.usage","allowChildren":false,"expandable":false,"leaf":true},{"text":"usage","id":"**AIO.DIAG.diag2.cpu.cpu0.usage**","path":"AIO.DIAG.diag2.cpu.cpu0.usage","allowChildren":false,"expandable":false,"leaf":true}]w
GET /metrics?path=**AIO.DIAG.diag2.cpu.cpu.usage**&path=**AIO.DIAG.diag2.cpu.cpu0.usage**&path=**AIO.DIAG.diag2.cpu.cpu1.usage**&from=1500814323&to=1500815223 HTTP/1.1
Host: 127.0.0.1:18080
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.7.0 CPython/2.7.5 Linux/2.6.32-504.30.3.el6.x86_64
HTTP/1.1 200 OK
Content-Type: application/json
({"from":1500814320,"to":1500815220,"step":60,"series":{"**AIO.DIAG.diag2.cpu.cpu.usage**":[5.612457354863484,5.973420302073161,5.450943231582642,5.963113149007161,5.773936112721761,6.122448921203613,5.4597095648447675,5.780772129694621,5.752193371454875,5.6122448444366455,5.6019376913706465,5.964653889338176,6.1088831424713135,5.6122448444366455,null,null]}}w
200 OK
Length: 1059 (1.0K) [text/csv]
Saving to: “STDOUT”

 0% [                                                                                                                                                                          ] 0           --.-K/s             
**AIO.DIAG.diag2.cpu.cpu.usage**,2017-07-23 15:52:00,5.612457354863484
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:53:00,5.973420302073161
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:54:00,5.450943231582642
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:55:00,5.963113149007161
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:56:00,5.773936112721761
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:57:00,6.122448921203613
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:58:00,5.4597095648447675
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 15:59:00,5.780772129694621
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:00:00,5.752193371454875
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:01:00,5.6122448444366455
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:02:00,5.6019376913706465
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:03:00,5.964653889338176
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:04:00,6.1088831424713135
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:05:00,5.6122448444366455
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:06:00,
AIO.DIAG.diag2.cpu.cpu.usage,2017-07-23 16:07:00,

[avneradania]

@pyr, @ifesdjeen, are you familiar with such behavior?

[ifesdjeen]

@avneradania it's clearly a bug. Although I didn't yet have a chance to take a closer look at it. Not sure if that could be because of the duplicated * or not (just looks suspicious). Unit tests related to multiplexing also pass, so might be an integration issue.

I'll try to find time for that this week.

[ifesdjeen]

@avneradania I could not reproduce it so far.

Since you're using an external dependency (graphite-api) I can't really help out. If you send what exactly graphite-api queries, I'll check if the things get lost on our side.

[awkspace]

Ran into this issue recently and did a little digging on its history.

It looks like this was first brought up in brutasse/graphite-cyanite#16 and subsequently fixed in #168. However #265 broke it again. Building off of commits prior to that merge results in working multi-metric queries from grafana/graphite-api, but anything later gives me only the first metric.

[ifesdjeen]

@awkspace most likely it was fixed by https://github.com/pyr/net/commit/fa22b8c0c8e64f25c949ad5680d42ef598771f6d but there was no library release...