openstack-exporter icon indicating copy to clipboard operation
openstack-exporter copied to clipboard

Published 20 hours ago • verified publisher icon openstack-exporter

Running with restricted privileges

Open mkszuba opened this issue 3 years ago • 16 comments

Hello,

If I understand your documentation correctly, openstack-exporter is supposed to connect to OpenStack as a user with full admin rights in the projects under consideration. Sadly, this is at odds with security policies of our sites. Is there a list somewhere of what the exact privileges the exporter user must have?

mkszuba avatar Oct 12 '21 11:10 mkszuba

Hello @mkszuba , As far as exporter does not create, delete or modify something, RO permissions should be enough.

alexeymyltsev avatar Oct 12 '21 15:10 alexeymyltsev

Hello,

If I understand your documentation correctly, openstack-exporter is supposed to connect to OpenStack as a user with full admin rights in the projects under consideration. Sadly, this is at odds with security policies of our sites. Is there a list somewhere of what the exact privileges the exporter user must have?

You could indeed use a reader system scoped role and so at least limit security issues in terms of behavior if not in terms of discoverability.

gtherond avatar Oct 13 '21 15:10 gtherond

Hello, If I understand your documentation correctly, openstack-exporter is supposed to connect to OpenStack as a user with full admin rights in the projects under consideration. Sadly, this is at odds with security policies of our sites. Is there a list somewhere of what the exact privileges the exporter user must have?

You could indeed use a reader system scoped role and so at least limit security issues in terms of behavior if not in terms of discoverability.

Hello @gtherond . Do you use scoped tokens? Honestly speaking we tried move to scoped tokens but faced that it works not properly for all services.

alexeymyltsev avatar Oct 14 '21 03:10 alexeymyltsev

Finally had a moment to try running the exporter with 'reader' permissions. The bad news is, it's not enough - there are authorisation-related errors on output every time it tries to retrieve data from OpenStack. The good news, the messages make it (mostly) clear what additional privileges are required:

  • nova / os_compute_api:os-services
  • nova / os_compute_api:servers:detail:get_all_tenants
  • nova / os_compute_api:os-hypervisors
  • identity / identity:list_domains
  • identity / identity:list_users
  • identity / identity:list_groups
  • identity / identity:list_projects
  • cinder / volume_extension:services:index
  • cinder / scheduler_extension:scheduler_stats:get_pools

Or am I doing something wrong and the exporter should not require these privileges?

I've also been getting authentication errors associated with attempting to retrieve Object Store data - but since we use Ceph radosgw rather than Swift here, it's a different kettle of fish altogether.

mkszuba avatar Oct 20 '21 15:10 mkszuba

Regarding reader permissions. Do you have soped tokens? If yes, you should have reader on System Scope level.

alexeymyltsev avatar Oct 21 '21 07:10 alexeymyltsev

Hi, we have created an openstack with default roles.

When we use admin user, everything works. We created a user monitor:

openstack user create --password mypassword monitor
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | 4336f553239140949b0e0226b4ec19e0 |
| name                | monitor                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+
openstack role add --system all --user monitor reader
openstack role assignment list --names --system all
+---------------------+-----------------+--------------------------+---------+--------+--------+-----------+
| Role                | User            | Group                    | Project | Domain | System | Inherited |
+---------------------+-----------------+--------------------------+---------+--------+--------+-----------+
| admin               |                 | OpenStack Admins@Default |         |        | all    | False     |
| load-balancer_admin |                 | OpenStack Admins@Default |         |        | all    | False     |
| admin               | admin@Default   |                          |         |        | all    | False     |
| reader              | monitor@Default |                          |         |        | all    | False     |
+---------------------+-----------------+--------------------------+---------+--------+--------+-----------+

When we stated the exporter:

Jul 21 09:40:57 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:40:57Z" level=info msg="Build context (go=go1.16.3, user=, date=)" source="main.go:61"
Jul 21 09:40:57 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:40:57Z" level=debug msg="Setting Env var OS_CLIENT_CONFIG_FILE = /home/prom_os_exp/cloud.yml" source="main.go:64"
Jul 21 09:40:57 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:40:57Z" level=info msg="openstack exporter started in legacy mode" source="main.go:85"
Jul 21 09:40:57 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:40:57Z" level=info msg="Starting HTTP server on :9180" source="main.go:95"
Jul 21 09:41:03 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:03Z" level=info msg="Starting openstack exporter version (version=, branch=, revision=) for cloud: default" source="main.go:148"
Jul 21 09:41:03 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:03Z" level=info msg="Build context (go=go1.16.3, user=, date=)" source="main.go:149"
Jul 21 09:41:03 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:03Z" level=debug msg="Setting Env var OS_CLIENT_CONFIG_FILE = /home/prom_os_exp/cloud.yml" source="main.go:152"
Jul 21 09:41:03 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:03Z" level=error msg="enabling exporter for service network failed: Authentication failed" source="main.go:163"
Jul 21 09:41:04 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:04Z" level=error msg="enabling exporter for service volume failed: Authentication failed" source="main.go:163"
Jul 21 09:41:04 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:04Z" level=error msg="enabling exporter for service identity failed: Authentication failed" source="main.go:163"
Jul 21 09:41:04 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:04Z" level=error msg="enabling exporter for service load-balancer failed: Authentication failed" source="main.go:163"
Jul 21 09:41:05 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:05Z" level=error msg="enabling exporter for service dns failed: Authentication failed" source="main.go:163"
Jul 21 09:41:05 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:05Z" level=error msg="enabling exporter for service gnocchi failed: Authentication failed" source="main.go:163"
Jul 21 09:41:05 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:05Z" level=error msg="enabling exporter for service compute failed: Authentication failed" source="main.go:163"
Jul 21 09:41:06 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:06Z" level=error msg="enabling exporter for service image failed: Authentication failed" source="main.go:163"
Jul 21 09:41:06 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:06Z" level=error msg="enabling exporter for service orchestration failed: Authentication failed" source="main.go:163"
Jul 21 09:41:06 monitor01 golang-openstack-exporter.openstack-exporter[1112787]: time="2022-07-21T09:41:06Z" level=error msg="No exporter has been enabled, exiting" source="main.go:173"

Version:

golang-openstack-exporter.openstack-exporter --version
openstack-exporter, version  (branch: , revision: )
  build user:       
  build date:       
  go version:       go1.16.3
snap list |grep expo
golang-openstack-exporter  v1.4.0         97     latest/stable  niedbalski   -

tabacha avatar Jul 21 '22 11:07 tabacha

Hi all, I have made some tests and it seems like: 1) exporter doesn't know how to use scoped tokens, 2) got somewhere hardcoded usage of admin project.

Regarding reader permissions. Do you have scoped tokens? If yes, you should have reader on System Scope level.

Having openstack-exporter user configured as reader for system, with correct clouds.yaml file prepared for that user:

openstack role assignment list --names --user openstack-exporter
+--------+----------------------------+-------+--------------------+--------+--------+-----------+
| Role   | User                       | Group | Project            | Domain | System | Inherited |
+--------+----------------------------+-------+--------------------+--------+--------+-----------+
| reader | openstack-exporter@Default |       | admin@Default      |        |        | False     |
| reader | openstack-exporter@Default |       | monitoring@Default |        |        | False     |
| reader | openstack-exporter@Default |       |                    |        | all    | False     |
+--------+----------------------------+-------+--------------------+--------+--------+-----------+

cat /etc/openstack/clouds.yaml
---
clouds:
  system-mon:
    auth:
      auth_url: https://XXXXXXXXXXXXXXXXXXXXX:5000/v3/
      password: YYYYYYYYYYYY
      username: openstack-exporter
      user_domain_name: Default
      system_scope: all
    region_name: RegionOne
    identity_api_version: 3
    identity_interface: public
    verify: true

And knowing, that system scope reader role is required for listing endpoints:

oslopolicy-policy-generator --namespace keystone 2>&1 | grep identity:list_endpoints
"identity:list_endpoints": "role:reader and system_scope:all"

I'm able to execute system reader queries as standard user with system scope token and role reader: openstack --os-cloud system-mon endpoint list

+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                               |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------+
| XXXXXXX | RegionOne | cinderv3     | volumev3     | True    | internal  | https://XXXXXXXXXX:8776/v3         |

But openstack-exporter with using same clouds.yaml is unable to authenticate:

./openstack-exporter --disable-service.placement --disable-service.orchestration --disable-service.database --disable-service.gnocchi --disable-service.baremetal --disable-service.dns --disable-service.container-infra --disable-service.load-balancer --disable-service.object-store --disable-service.volume --disable-service.image --disable-service.compute --disable-service.network --log.level="debug" --web.listen-address=":9181"  system-mon
INFO[0000] Build context (go=go1.18.3, user=, date=)     source="main.go:62"
INFO[0000] openstack exporter started in legacy mode     source="main.go:86"
INFO[0000] Starting HTTP server on :9181                 source="main.go:101"
INFO[0001] Starting openstack exporter version (version=, branch=, revision=) for cloud: system-mon  source="main.go:154"
INFO[0001] Build context (go=go1.18.3, user=, date=)     source="main.go:155"
INFO[0001] Adding metric: domains to exporter: identity  source="exporter.go:178"
INFO[0001] Adding metric: users to exporter: identity    source="exporter.go:178"
INFO[0001] Adding metric: groups to exporter: identity   source="exporter.go:178"
INFO[0001] Adding metric: projects to exporter: identity  source="exporter.go:178"
INFO[0001] Adding metric: project_info to exporter: identity  source="exporter.go:178"
INFO[0001] Adding metric: regions to exporter: identity  source="exporter.go:178"
INFO[0001] Enabled exporter for service: identity        source="main.go:173"
INFO[0001] Collecting metrics for exporter: openstack_identity, metric: regions  source="exporter.go:97"
INFO[0001] Collected metrics for exporter: openstack_identity, metric: regions  source="exporter.go:104"
DEBU[0001] No function handler set for metric: up        source="exporter.go:117"
DEBU[0001] No function handler set for metric: openstack_metric_collect_seconds  source="exporter.go:117"
INFO[0001] Collecting metrics for exporter: openstack_identity, metric: domains  source="exporter.go:97"
ERRO[0001] Failed to collect metric for exporter: identity, error: failed to collect metric: domains, error: Request forbidden: [GET https://XXXXXXX:5000/v3/domains], error message: {"error":{"code":403,"message":"You are not authorized to perform the requested action: identity:list_domains.","title":"Forbidden"}}  source="exporter.go:123"
INFO[0001] Collecting metrics for exporter: openstack_identity, metric: users  source="exporter.go:97"
ERRO[0001] Failed to collect metric for exporter: identity, error: failed to collect metric: users, error: Request forbidden: [GET https://XXXXXXX:5000/v3/users], error message: {"error":{"code":403,"message":"You are not authorized to perform the requested action: identity:list_users.","title":"Forbidden"}}  source="exporter.go:123"
INFO[0001] Collecting metrics for exporter: openstack_identity, metric: groups  source="exporter.go:97"
ERRO[0001] Failed to collect metric for exporter: identity, error: failed to collect metric: groups, error: Request forbidden: [GET https://XXXXXXX:5000/v3/groups], error message: {"error":{"code":403,"message":"You are not authorized to perform the requested action: identity:list_groups.","title":"Forbidden"}}  source="exporter.go:123"
INFO[0001] Collecting metrics for exporter: openstack_identity, metric: projects  source="exporter.go:97"
ERRO[0001] Failed to collect metric for exporter: identity, error: failed to collect metric: projects, error: Request forbidden: [GET https://XXXXXXX:5000/v3/projects], error message: {"error":{"code":403,"message":"You are not authorized to perform the requested action: identity:list_projects.","title":"Forbidden"}}  source="exporter.go:123"
DEBU[0001] No function handler set for metric: project_info  source="exporter.go:117

And again, policy allows only system scoped reader tokens:

oslopolicy-policy-generator --namespace keystone 2>&1 | grep identity:list_domains
"identity:list_domains": "role:reader and system_scope:all"

Where openstack-exporter user tries to use project admin (as it would be hardcoded somewhere):

2024-01-22 13:45:04.846 1820200 DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-XXXXXXX YYYYYY ZZZZZZZ - - de
fault default] RBAC: auth_context: {'token': <TokenModel (audit_id=AAAAAAA, audit_chain_id=['BBBBBBBB']) at CCCCCCCC>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id
': None, 'domain_name': None, 'group_ids': [], 'user_id': '<<openstack-exporter user ID>>', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '<<admin project ID>>', 'project_domain_id': 'defau
lt', 'roles': ['reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /usr/lib/python3
/dist-packages/keystone/server/flask/request_processing/middleware/auth_context.py:483

When user openstack-exporter got no role in admin project, it still tries to authenticate to admin project, but fails in keystone as it has no role that would allow for that.

EDIT: This is tested on openstack-exporter 1.6.0

fprzewozny avatar Jan 22 '24 13:01 fprzewozny

Keystone system scope seems to be supported by gophercloud from 28.05.2020, so v0.25.0 (?): https://github.com/gophercloud/gophercloud/pull/1908/files and openstack-exporter seems to be using v1.5.0: https://github.com/openstack-exporter/openstack-exporter/blob/main/go.mod#L8C25-L8C36

fprzewozny avatar Jan 22 '24 18:01 fprzewozny

@odnobit I saw your activity under other issues, maybe you could have a look? Every openstack-exporter user could benefit from getting rid of admin user requirements.

fprzewozny avatar Feb 20 '24 11:02 fprzewozny

@odnobit I saw your activity under other issues, maybe you could have a look? Every openstack-exporter user could benefit from getting rid of admin user requirements.

Hello! If I remember correctly, the reader role hasn't been fully completed yet. If I were in your position, I would consider looking at another approach: modifying the policy.yaml file for services (create admin_readonly role)

odnobit avatar Feb 20 '24 12:02 odnobit

@odnobit I saw your activity under other issues, maybe you could have a look? Every openstack-exporter user could benefit from getting rid of admin user requirements.

Hello! If I remember correctly, the reader role hasn't been fully completed yet. If I were in your position, I would consider looking at another approach: modifying the policy.yaml file for services (create admin_readonly role)

Thanks for your reply, admin_readonly role requires a ton of policy rules, and it becomes real pain in my situation, where I manage multiple clusters running different versions. system scope reader role tested manually allows for getting all of the informations used by openstack-exporter, but exporter itself isn't able to manage the system scope role. Could you please have a look into gophercloud used by current release of openstack-exporter? My message from a month ago:

Keystone system scope seems to be supported by gophercloud from 28.05.2020, so v0.25.0 (?): https://github.com/gophercloud/gophercloud/pull/1908/files and openstack-exporter seems to be using v1.5.0: https://github.com/openstack-exporter/openstack-exporter/blob/main/go.mod#L8C25-L8C36

Thanks!

fprzewozny avatar Feb 20 '24 12:02 fprzewozny

Okay, I will try to use scoped-tokens on my stage and will write the answer

odnobit avatar Feb 20 '24 12:02 odnobit

it's working, but few nova metrics have failed to collect because policies, details below

ts=2024-02-20T12:40:19.448Z caller=exporter.go:126 level=error err="Failed to collect metric for exporter" exporter=nova error="failed to collect metric: limits_vcpus_max, error: Request forbidden: [GET http://10.20.0.111:8774/v2.1/limits?tenant_id=21197b95f90b4e50a6af1c1f09781be1], error message: {\"forbidden\": {\"code\": 403, \"message\": \"Forbidden\"}}"
ts=2024-02-20T12:40:19.560Z caller=exporter.go:126 level=error err="Failed to collect metric for exporter" exporter=nova error="failed to collect metric: server_local_gb, error: Request forbidden: [GET http://10.20.0.111:8774/v2.1/os-simple-tenant-usage?detailed=1], error message: {\"forbidden\": {\"code\": 403, \"message\": \"Forbidden\"}}

role assignment

$ openstack role add --user member --system all reader

$ openstack role assignment list --names
+-----------------+----------------------+-------+-------------------------+--------+--------+-----------+
| Role            | User                 | Group | Project                 | Domain | System | Inherited |
+-----------------+----------------------+-------+-------------------------+--------+--------+-----------+
| reader          | member@Default       |       |                         |        | all    | False     |
+-----------------+----------------------+-------+-------------------------+--------+--------+-----------+

clouds.yaml example

---
clouds:
  default:
    auth:
      auth_url: http://10.20.0.111:5000/v3
      system_scope: all
      username: member
      password: test
      user_domain_name: Default
    region_name: RegionOne
    interface: internal

odnobit avatar Feb 20 '24 12:02 odnobit

@odnobit That's interesting! Have you changed anything in exporter itself? Can you share what OpenStack version are you running?

fprzewozny avatar Feb 20 '24 14:02 fprzewozny

openstack version is Xena. Nope I've change nothing, just built it from master branch and run. Btw, v1.7.0 has been released so you can try new version if you don't want to build it from sources.

odnobit avatar Feb 20 '24 14:02 odnobit

It really did started working when upgraded to v1.7.0! I will test it fully in coming days, thanks.

fprzewozny avatar Feb 20 '24 14:02 fprzewozny