grimoirelab icon indicating copy to clipboard operation
grimoirelab copied to clipboard

Published 20 hours ago • verified publisher icon chaoss

Unable to use default dashboards for github enterprise

Open Dbz opened this issue 2 months ago • 1 comments

Hello, I was wondering if it is possible to use the default dashboards for github enterprise. I am able to get my data ingested, but I think there may be a problem with enriching the index or something similar. I'm trying to stick as close to the default configs as possible so that I can get it up and running. I've changed our github org and repo to "org/repo":

projects.cfg:

{
  "grimoirelab": {
    "meta": {
      "title": "GrimoireLab"
    },
    "github": [
      "https://username:[email protected]/org/repo"
    ]
  }
}

setup.cfg:

[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/grimoire/logs
bulk_size = 100
scroll_size = 100
aliases_file = /home/grimoire/aliases.json

[projects]
projects_file = /home/grimoire/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = nginx
user = root
password = root
port = 8000
path = /identities/api/
ssl = false
database = sortinghat_db
autoprofile = [github, git]
matching = [email,name,username]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true

[panels]
kibiter_time_from = now-90d
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.8.6-3

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git_demo_raw
enriched_index = git_demo_enriched
latest-items = true
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

# new stuff here
[github]
raw_index = git_demo_raw
enriched_index = git_demo_enriched
api-token = xxx
# category = pull_request
sleep-for-rate = true
no-archive = true
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

I see it having an error which may or may not be relevant. My thought is maybe I'm not using the right studies, and/or maybe I need to find a way to transform some of the data into what fields the dashboards are expecting github produces (maybe we have some custom github enterprise settings or something making our fields different names?):

2024-04-17 17:25:30,330 - GrimoireLab - grimoire_elk.elk - INFO - [github] Done enrichment for https://github.com/org/repo
2024-04-17 17:25:30,332 - GrimoireLab - sirmordred.task_enrich - INFO - [github] enrichment finished for https://github.com/org/repo
2024-04-17 17:25:30,332 - GrimoireLab - sirmordred.task_enrich - INFO - [github] enrichment phase finished in 0:00:03
2024-04-17 17:25:30,333 - GrimoireLab - sirmordred.task_enrich - INFO - [github] data retention start
2024-04-17 17:25:30,340 - GrimoireLab - sirmordred.task_enrich - INFO - [github] data retention end
2024-04-17 17:25:30,341 - GrimoireLab - sirmordred.task_enrich - INFO - [github] identities retention end
2024-04-17 17:25:30,341 - GrimoireLab - sirmordred.task_enrich - INFO - [github] autorefresh start
2024-04-17 17:25:30,355 - GrimoireLab - sirmordred.task_enrich - INFO - [github] Refreshing identities from 2024-04-15 17:23:57.356895
2024-04-17 17:25:31,466 - GrimoireLab - sirmordred.task_enrich - INFO - [github] Refreshed 165 identities in 0:00:01
2024-04-17 17:25:31,467 - GrimoireLab - sirmordred.task_enrich - INFO - [github] autorefresh end
2024-04-17 17:25:31,467 - GrimoireLab - sirmordred.task_enrich - INFO - [github] studies phase start
2024-04-17 17:25:33,545 - GrimoireLab - sirmordred.task_enrich - ERROR - [github] Wrong studies names: ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git']
2024-04-17 17:25:33,546 - GrimoireLab - sirmordred.task_manager - ERROR - [github] Exception in Task Manager ('Wrong studies names ', 'github', ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git'])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_manager.py", line 98, in run
    task.execute()
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 460, in execute
    raise e
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 450, in execute
    self.__studies(retention_time)
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 346, in __studies
    raise RuntimeError('Wrong studies names ', self.backend_section, studies)
RuntimeError: ('Wrong studies names ', 'github', ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git'])
Exception in thread github:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_manager.py", line 98, in run
    task.execute()
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 460, in execute
    raise e
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 450, in execute
    self.__studies(retention_time)
  File "/usr/local/lib/python3.8/site-packages/sirmordred/task_enrich.py", line 346, in __studies
    raise RuntimeError('Wrong studies names ', self.backend_section, studies)
RuntimeError: ('Wrong studies names ', 'github', ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git'])

Here is a json of a single document (with some "redacted"s):

{
  "_index": "git_demo_enriched",
  "_type": "_doc",
  "_id": "e46cf7fb98aafa951cb5745c90ddb4cb9b0e2217",
  "_score": 1,
  "_source": {
    "time_to_first_attention": null,
    "issue_url": "https://github.com/org/repo/pull/965",
    "assignee_data_domain": "",
    "assignee_data_user_name": "Unknown",
    "user_data_uuid": "4eadf680b14d01c735897967b144e5bff357458e",
    "demography_min_date": "2024-03-12T10:35:59.000Z",
    "item_type": "pull request",
    "metadata__gelk_backend_name": "GitHubEnrich",
    "project": "grimoirelab",
    "metadata__timestamp": "2024-04-17T16:44:16.311701+00:00",
    "time_to_close_days": 0.19,
    "repository": "https://github.com/org/repo",
    "uuid": "e46cf7fb98aafa951cb5745c90ddb4cb9b0e2217",
    "user_data_bot": false,
    "cm_title": "GrimoireLab",
    "user_domain": "gm.com",
    "user_data_multi_org_names": [
      "Unknown"
    ],
    "user_login": "redacted",
    "metadata__enriched_on": "2024-04-17T16:44:42.085469+00:00",
    "title_analyzed": "redacted",
    "assignee_data_name": "Unknown",
    "tag": "https://github.com/org/repo",
    "assignee_login": null,
    "id": 2227311311,
    "state": "closed",
    "assignee_data_bot": false,
    "author_bot": false,
    "assignee_data_gender": "",
    "author_org_name": "Unknown",
    "repository_labels": [],
    "project_1": "grimoirelab",
    "closed_at": "2024-04-05T12:18:33Z",
    "offset": null,
    "user_data_user_name": "",
    "user_data_id": "4eadf680b14d01c735897967b144e5bff357458e",
    "labels": [],
    "user_data_name": "redacted",
    "is_github_issue": 1,
    "assignee_geolocation": null,
    "user_data_org_name": "Unknown",
    "metadata__gelk_version": "1.0.0",
    "author_id": "4eadf680b14d01c735897967b144e5bff357458e",
    "author_name": "redacted",
    "author_gender": null,
    "author_multi_org_names": [
      "Unknown"
    ],
    "assignee_org": null,
    "assignee_data_gender_acc": null,
    "assignee_location": null,
    "time_open_days": 0.19,
    "user_name": "redacted",
    "origin": "https://github.com/org/repo",
    "created_at": "2024-04-05T07:48:58Z",
    "url_id": "org/repo/issues/965",
    "title": "redacted",
    "github_repo": "org/repo",
    "assignee_data_uuid": "",
    "assignee_name": null,
    "assignee_data_org_name": "Unknown",
    "user_org": null,
    "demography_max_date": "2024-04-14T12:59:35.000Z",
    "updated_at": "2024-04-05T12:18:34Z",
    "author_user_name": "",
    "id_in_repo": "965",
    "grimoire_creation_date": "2024-04-05T07:48:58+00:00",
    "assignee_data_multi_org_names": [
      ""
    ],
    "author_gender_acc": null,
    "author_domain": "redacted",
    "assignee_data_id": "",
    "pull_request": true,
    "user_data_domain": "redacted",
    "user_geolocation": null,
    "metadata__filter_raw": null,
    "author_uuid": "4eadf680b14d01c735897967b144e5bff357458e",
    "url": "https://github.com/org/repo/pull/965",
    "user_data_gender_acc": null,
    "user_data_gender": null,
    "metadata__updated_on": "2024-04-05T12:18:34+00:00",
    "user_location": null,
    "assignee_domain": null
  },
  "fields": {
    "closed_at": [
      "2024-04-05T12:18:33.000Z"
    ],
    "demography_min_date": [
      "2024-03-12T10:35:59.000Z"
    ],
    "created_at": [
      "2024-04-05T07:48:58.000Z"
    ],
    "metadata__timestamp": [
      "2024-04-17T16:44:16.311Z"
    ],
    "demography_max_date": [
      "2024-04-14T12:59:35.000Z"
    ],
    "updated_at": [
      "2024-04-05T12:18:34.000Z"
    ],
    "grimoire_creation_date": [
      "2024-04-05T07:48:58.000Z"
    ],
    "metadata__enriched_on": [
      "2024-04-17T16:44:42.085Z"
    ],
    "metadata__updated_on": [
      "2024-04-05T12:18:34.000Z"
    ]
  }
}

Are there any ideas on how I can get the default dashboards to work?

example: image

Thank you so much for your help even looking at this. Please let me know if I can update the documentation/donate/etc. after I work through getting this up and running. I'm always happy to contribute back. <3

Dbz avatar Apr 17 '24 17:04 Dbz

Hi @Dbz

2024-04-17 17:25:33,546 - GrimoireLab - sirmordred.task_manager - ERROR - [github] Exception in Task Manager ('Wrong studies names ', 'github', ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git']) ... ... RuntimeError: ('Wrong studies names ', 'github', ['enrich_demography:git', 'enrich_areas_of_code:git', 'enrich_onion:git'])`

Wrong study sections in github more info at https://github.com/chaoss/grimoirelab-sirmordred?tab=readme-ov-file#github- about github studies.

projects.json

  • Credentials are not needed because it will use the API token in the setup.cfg

setup.cfg

  • You are using git studies and index name in github

Please try again but using these projects.json and setup.cfg files.

  • projects.json
{
  "grimoirelab": {
    "meta": {
      "title": "GrimoireLab"
    },
    "github": [
      "https://github.com/org/repo"
    ]
  }
}
  • setup.cfg
[github]
raw_index = github_demo_raw
enriched_index = github_demo_enriched
api-token = xxx
sleep-for-rate = true
no-archive = true
studies = [enrich_demography:github]

[enrich_demography:github]

zhquan avatar Apr 17 '24 19:04 zhquan