/metrics/find is not compliant with what graphite-web expects

[xneo64]

I've been trying to integrate carbonserver + carbonzipper with graphite-web, and it seems that the response by carbonserver for the /metrics/find query made towards CLUSTER_SERVERS by graphite-web is incompatible with the equivalent response graphite-web issues:

In the example below, the receiver on port 8080 is graphite-web and 8081 is carbonserver. The same happens when both queries are issued in 'format=pickle' instead of json

$ curl "bcarbon00:8080/metrics/find/?local=1&format=treejson&query=%2A"
[{"text": "carbon", "expandable": 1, "leaf": 0, "id": "carbon", "allowChildren": 1}, {"text": "mlt3", "expandable": 1, "leaf": 0, "id": "mlt3", "allowChildren": 1}, {"text": "mlt3-ppc1", "expandable": 1, "leaf": 0, "id": "mlt3-ppc1", "allowChildren": 1}, {"text": "mlt3-ppc2", "expandable": 1, "leaf": 0, "id": "mlt3-ppc2", "allowChildren": 1}, {"text": "mlt3-ppc3", "expandable": 1, "leaf": 0, "id": "mlt3-ppc3", "allowChildren": 1}]

$ curl "bcarbon00:8081/metrics/find/?local=1&format=json&query=%2A"
{"name":"*","matches":[{"path":"carbon","isLeaf":false},{"path":"graphite.db","isLeaf":false},{"path":"mlt3","isLeaf":false},{"path":"mlt3-ppc1","isLeaf":false},{"path":"mlt3-ppc2","isLeaf":false},{"path":"mlt3-ppc3","isLeaf":false}]}

The request issued by graphite-web is in pickle format, but I've shown the output in json here for readability. The actual request is made in pickle, and the response by carbonserver causes graphite-web to complain and fail to parse.

I know there's carbonapi, but there are some other separate issues there which are currently impeding me from switching to it fully. Am I doing this wrong? I If so please advise, I'd really like to migrate to carbonserver/zipper!

[grobian]

Hmmm, as far as I remember carbonserver doesn't talk pickle, it only talks json and a protobuf protocol that only carbonserver, carbonzipper and carbonapi talk.

[xneo64]

It does speak pickle protocol, from line 284. Also it does return seemingly valid data when asked for format=pickle Which is in line with it to be "a drop-in replacement" for the graphite-web on the stores.

[grobian]

oh, I see, thanks for clarifying my code ;) So what does graphite-web complain about? IIRC treejson is different from json, it may be that a recent release of graphite-web changed something here.

[xneo64]

No worries :) I used treejson for illustration purposes only. The flow usually is like this:

Webclient (web-based graphite metric browser) issues: GET /metrics/find/?_dc=1479381707026&query=*&format=treejson&path=&node=GraphiteTree or Grafana issues: GET /metrics/find/?query=*

This request is processed by graphite-web acting as cluster-head, and the latter issues a request to all members in CLUSTER_SERVERS with this: GET /metrics/find/?local=1&format=pickle&query=%2A

I get a 500 with the following trace when debug is enabled on graphite-web:

Traceback:

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/opt/graphite/webapp/graphite/metrics/views.py" in find_view
  116.     matches = list( STORE.find(query, fromTime, untilTime, local=local_only) )

File "/opt/graphite/webapp/graphite/storage.py" in find
  54.         for node in request.get_results():

File "/opt/graphite/webapp/graphite/remote_storage.py" in get_results
  115.         node = BranchNode(node_info['path'])

Exception Type: KeyError at /metrics/find/
Exception Value: 'path'

If you want I can also supply you with individual packet captures and full stack traces from graphite-web

[Civil]

If I'm correct graphite-web 0.9.x and graphite-web@master have different format to answer those questions.

There is an effort to make carbonserver compatibility layer (basically carbonserver with some fixes, cache support and converted to be a module for go-carbon) in go-carbon: https://github.com/lomik/go-carbon/ (it also works as drop-in replacement for carbon-cache). We have this thing in testing since approx. November and in production since end of December.

You can try to switch to it and to see if your problem is fixed there.

@grobian if you want you also get grab the commits that was done by lomik to fix pickle support and import them to this repo.

[grobian]

@Civil would it be useful to give you push rights to this repo?

[grobian]

or better: deprecate this project as soon as go-carbon is ok enough for you?

[Civil]

Well, it's your project so it's up to you. I can backport changes if you think there is a need for that, but right now go-carbon implements the same functionality + have cache support.

[grobian]

yup, and go-carbon seems to me the way to go ;)

[Civil]

I think it's really better to deprecate carbonserver in favor to go-carbon as at this moment we are moving to protobuf3 and carbonzipper will no longer be compatible with carbonserver.

Even though I can port all the changes here (most of them are straightforward) but I don't know if it's worth it. You can still use only carbonserver from go-carbon for example.