Resource Usage in dashboard saying something went wrong with version v4.2.2

[thenucleophile]

Resource Usage in dashboard saying something went wrong with version v4.2.2 while ceilometer resource-list is giving output as below

root@controller:~# ceilometer resource-list +----------------------------------------------+--------+----------------------------------+----------------------------------+ | Resource ID | Source | User ID | Project ID | +----------------------------------------------+--------+----------------------------------+----------------------------------+ | 50a9a09e-a71a-477e-816a-3ac8a8a1b055 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | 6c794811-73fa-4f4c-bef4-ce5fea20d9fc | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | 8813ff3d-6b65-45c5-b0ca-f104bab41f7c | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | 8f81eefb-5f28-4b5c-b283-58911710adcd | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | 99344452-5fee-4607-acb7-74b257bd948a | | None | 5b3463337426443fbc85633051c2e972 | | 9d16cb95-fd6d-4132-8a5c-ec06ebbb7319 | | None | 5b3463337426443fbc85633051c2e972 | | aff4e933-8e83-4aa0-83c0-c9a559dacdaa | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | b1206217-cc37-4780-a882-6013b885cc79 | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | b912ab7f-380a-49dd-abe6-347635660429 | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | cbc0c113-45e2-427b-8ec4-8ddf3a3c6bf5 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | cbf18e9f-33ed-439b-89e7-a8f106fdc537 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | cc3f8efa-ed18-49b1-b696-2fa3a58115a8 | | None | 5b3463337426443fbc85633051c2e972 | | d33f8089-b751-456f-9054-98de74560feb | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | d85e264d-8bef-4679-b17f-65575192f558 | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | d919a729-90f3-46d7-994d-86371c47b37b | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | e9e0e8ba-4420-4b05-9759-1c68c626356c | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | f2f30735-196f-4db8-90d7-9b9eb6eca732 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | nova-instance-instance-00000001-fa163eb56aa0 | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | nova-instance-instance-00000003-fa163e9a105d | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | nova-instance-instance-00000005-fa163e7b328f | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | nova-instance-instance-0000000b-fa163efdd281 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | nova-instance-instance-0000001b-fa163e2ed3ee | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | +----------------------------------------------+--------+----------------------------------+----------------------------------+

root@controller:/var/log/ceilometer# cat ceilometer-api.log 2014-04-25 11:02:48.869 7826 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail: Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in get_resources raise NotImplementedError(_('metaquery not implemented'))

NotImplementedError: metaquery not implemented

Output of cat ceilometer-agent-compute.log

2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(**kwargs) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common chanmap, None, timeout=timeout, 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.method_reader.read_method() 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common raise m 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common error: [Errno 110] Connection timed out 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common 2014-04-25 11:29:13.813 29155 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.20.0.175:5672 2014-04-25 11:29:13.833 29155 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.20.0.175:5672

Please help!

[cloudnull]

Couple of questions:

• what os are you running?
• have the packages been upgraded outside of the cookbooks and the installation?
• when was the installation done?
• Is this an HA setup?
• what is the version of the packages you have installed (dpkg -l | grep ceilometer || rpm -qa | grep ceilometer)

Get back to us with all of the information you can about your environment and we'll see what needs to be done to get it back online.

[thenucleophile]

Ubuntu 12.04 upgraded packages after cookbook installation initially it was 4.2.0 installation was done 1 month back It is not HA setup. One of each chef server, controller node (chef-client) & compute node (chef-client)

Controller Node: root@controller:~# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer ii ceilometer-agent-central 2013.2.2-0ubuntu1~cloud0 ceilometer central agent rc ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent ii ceilometer-api 2013.2.2-0ubuntu1~cloud0 ceilometer api service ii ceilometer-collector 2013.2.2-0ubuntu1~cloud0 ceilometer collector service ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

Compute Node: root@compute:~# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer ii ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

It is showing something went wrong in dashboard in resource usage tab

root@controller:/var/log/ceilometer# cat ceilometer-api.log

2014-04-28 11:29:39.235 1207 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail: Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in get_resources raise NotImplementedError(_('metaquery not implemented'))

NotImplementedError: metaquery not implemented

As per below link https://github.com/rcbops/chef-cookbooks/issues/799

this problem should not come as cookbook version is upgraded to 4.2.2

Please help! Thanks.

[thenucleophile]

i installed ceilometer-agent-compute on controller by mistake..it is already there on compute node...it can be removed as well..but resource usage tab doesn't appear..

Also all services are running fine.

root@controller:/etc/nova# nova-manage service list Binary Host Zone Status State Updated_At nova-conductor controller internal enabled :-) 2014-04-28 12:12:57 nova-cert controller internal enabled :-) 2014-04-28 12:12:58 nova-consoleauth controller internal enabled :-) 2014-04-28 12:12:58 nova-scheduler controller internal enabled :-) 2014-04-28 12:12:58 nova-network compute internal enabled :-) 2014-04-28 12:13:04 nova-compute compute nova enabled :-) 2014-04-28 12:12:08

[thenucleophile]

Do not why but nova-network is going down after sometime after rebooting compute node.

root@controller:/etc# nova-manage service list Binary Host Zone Status State Updated_At nova-conductor controller internal enabled :-) 2014-04-29 07:02:52 nova-cert controller internal enabled :-) 2014-04-29 07:03:01 nova-consoleauth controller internal enabled :-) 2014-04-29 07:03:00 nova-scheduler controller internal enabled :-) 2014-04-29 07:02:58 nova-network compute internal enabled XXX 2014-04-29 06:25:37 nova-compute compute nova enabled :-) 2014-04-29 07:02:52

[breu]

Ubuntu and RedHat packages were updated that caused our hand patch of ceilometer to fail.

RedHat
openstack-ceilometer-collector-2013.2.3-1.el6.noarch
python-ceilometerclient-1.0.8-1.el6.noarch
python-ceilometer-2013.2.3-1.el6.noarch
openstack-ceilometer-api-2013.2.3-1.el6.noarch
openstack-ceilometer-common-2013.2.3-1.el6.noarch
openstack-ceilometer-central-2013.2.3-1.el6.noarch

Ubuntu
ii ceilometer-agent-central 2013.2.2-0ubuntu1~cloud0 ceilometer central agent
rc ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent
ii ceilometer-api 2013.2.2-0ubuntu1~cloud0 ceilometer api service
ii ceilometer-collector 2013.2.2-0ubuntu1~cloud0 ceilometer collector service
ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files
ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries
ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

We will need to patch the patcher again..

[breu]

@thenucleophile Your nova-network problem is something separate from ceilometer. Check the nova-network log file but don't lump this into the same github issue.

[thenucleophile]

@rackerjoe Thanks!! Please update once the patcher is patched..

[sstent]

I'm seeing this on Rhel 6.5 as well

[runningman84]

Any news here? This problem still exists in Ubuntu 12.04 LTS

[claco]

@runningman84 Yes! There's a patch merged into master and the v4.2.3rc branches of ceilometer to fix this:

https://github.com/rcbops-cookbooks/ceilometer/pull/48 https://github.com/rcbops-cookbooks/ceilometer/tree/v4.2.3rc