Failed to setup code complexity analysis

[zzz945]

Hi:

After running docker-compose up -d, everything works fine except the cocom part. I found the following related error messages. Btw, I am using the most recent master branch.

Error messages in console

mordred_1 | 2022-03-23 23:44:33,744 - graal.graal - ERROR - Analysis failed at 83c4d1aa3637929df55670031c857857318a3f9b mordred_1 | 2022-03-23 23:44:33,745 - grimoire_elk.elk - ERROR - Error feeding raw from cocom (https://github.com/chaoss/grimoirelab): Impossible to checkout the worktree /tmp/worktrees/grimoirelab-git at 83c4d1aa3637929df55670031c857857318a3f9b mordred_1 | Traceback (most recent call last): mordred_1 | File "/usr/local/lib/python3.7/dist-packages/perceval/backends/core/git.py", line 1336, in _exec mordred_1 | cwd=cwd, env=env) mordred_1 | File "/usr/lib/python3.7/subprocess.py", line 775, in init mordred_1 | restore_signals, start_new_session) mordred_1 | File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child mordred_1 | raise child_exception_type(errno_num, err_msg, err_filename) mordred_1 | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/worktrees/grimoirelab-git': '/tmp/worktrees/grimoirelab-git' mordred_1 | mordred_1 | During handling of the above exception, another exception occurred: mordred_1 | mordred_1 | Traceback (most recent call last): mordred_1 | File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 334, in checkout mordred_1 | self._exec(cmd_checkout, cwd=self.worktreepath, env=self.gitenv) mordred_1 | File "/usr/local/lib/python3.7/dist-packages/perceval/backends/core/git.py", line 1339, in _exec mordred_1 | raise RepositoryError(cause=str(e)) mordred_1 | perceval.errors.RepositoryError: [Errno 2] No such file or directory: '/tmp/worktrees/grimoirelab-git': '/tmp/worktrees/grimoirelab-git' mordred_1 | mordred_1 | During handling of the above exception, another exception occurred: mordred_1 | mordred_1 | Traceback (most recent call last): mordred_1 | File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/elk.py", line 171, in feed_backend mordred_1 | ocean_backend.feed(**params) mordred_1 | File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/raw/elastic.py", line 231, in feed mordred_1 | self.feed_items(items) mordred_1 | File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/raw/elastic.py", line 247, in feed_items mordred_1 | for item in items: mordred_1 | File "/usr/local/lib/python3.7/dist-packages/perceval/backend.py", line 316, in fetch mordred_1 | for item in self.fetch_items(category, **kwargs): mordred_1 | File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 197, in fetch_items mordred_1 | raise e mordred_1 | File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 189, in fetch_items mordred_1 | self.graalRepo.checkout(commit['commit']) mordred_1 | File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 338, in checkout mordred_1 | raise RepositoryError(cause=cause) mordred_1 | perceval.errors.RepositoryError: Impossible to checkout the worktree /tmp/worktrees/grimoirelab-git at 83c4d1aa3637929df55670031c857857318a3f9b

My setup.cfg:

[general] short_name = GrimoireLab update = true min_update_delay = 60 debug = true logs_dir = /home/bitergia/logs aliases_file = /home/bitergia/conf/aliases.json

[projects] projects_file = /home/bitergia/conf/projects.json

[es_collection] url = http://elasticsearch:9200

[es_enrichment] url = http://elasticsearch:9200 autorefresh = true

[sortinghat] host = mariadb user = root password = database = demo_sh load_orgs = true orgs_file = /home/bitergia/conf/organizations.json autoprofile = [github, pipermail, git] matching = [email] sleep_for = 100 unaffiliated_group = Unknown affiliate = true strict_mapping = false reset_on_load = false identities_file = [/home/bitergia/conf/identities.yml] identities_format = grimoirelab

[panels] kibiter_time_from = now-5y kibiter_default_index = git kibiter_url = http://kibiter:5601 kibiter_version = 6.1.4-1 gitlab-issues = false gitlab-merges = false github-repos = true github-comments = true code-complexity = true

[phases] collection = true identities = true enrichment = true panels = true

#[bugzillarest] #raw_index = bugzillarest_demo-raw #enriched_index = bugzillarest_demo_enriched #no-archive = true

#[confluence] #no-archive = true #raw_index = confluence_demo-raw #enriched_index = confluence_demo_enriched

#[discourse] #raw_index = discourse_demo-raw #enriched_index = discourse_demo_enriched #no-archive = true

[git] raw_index = git-raw enriched_index = git_enriched latest-items = true studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

[github] api-token = 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 raw_index = github-raw enriched_index = github_issues_enrich no-archive = true sleep-for-rate = true category = issue studies = [enrich_onion:github, enrich_geolocation:user, enrich_geolocation:assignee, enrich_backlog_analysis, enrich_demography:github]

[github:repo] raw_index = github-repo_raw enriched_index = github-repo_enriched api-token = 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 category = repository sleep-for-rate = true no-archive = true studies = [enrich_demography:github]

[github2:issue] api-token = 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 raw_index = github2-issues_raw enriched_index = github2-issues_enriched sleep-for-rate = true category = issue no-archive = true studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_feelings]

[github2:pull] api-token = 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 raw_index = github2-pull_raw enriched_index = github2-pull_enriched sleep-for-rate = true category = pull_request no-archive = true studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_feelings]

[cocom] raw_index = cocom_chaoss enriched_index = cocom_chaoss_enrich category = code_complexity_lizard_file studies = [enrich_cocom_analysis] branches = master

#[gitlab:issues] #api-token = <YOUR_API_TOKEN_HERE> #raw_index = gitlab_issues_demo-raw #enriched_index = gitlab_issues_demo_enriched #no-archive = true #enterprise-url = <YOUR_GITLAB_INSTANCE_URL> #sleep-for-rate = true

#[gitlab:merge] #api-token = <YOUR_API_TOKEN_HERE> #raw_index = gitlab_merges_demo-raw #enriched_index = gitlab_merges_demo_enriched #no-archive = true #enterprise-url = <YOUR_GITLAB_INSTANCE_URL> #category = merge_request #sleep-for-rate = true

#[jira] #raw_index = jira_demo-raw #enriched_index = jira_demo_enriched #no-archive = true

#[pipermail] #raw_index = pipermail_demo-raw #enriched_index = pipermail_demo_enriched #no-verify = true

#[mediawiki] #raw_index = mediawiki_demo-raw #enriched_index = mediawiki_demo_enriched #no-archive = true

#[meetup] #raw_index = meetup_demo-raw #enriched_index = meetup_demo_enriched #api-token = <YOUR_API_TOKEN_WHERE> #no-archive = true #sleep-for-rate = true #sleep-time = "300"

#[stackexchange] #raw_index = stackexchange_demo-raw #enriched_index = stackexchange_demo_enriched #api-token = <YOUR_API_TOKEN_WHERE> #no-archive = true

#[slack] #raw_index = slack_demo-raw #enriched_index = slack_demo_enriched #api-token = <YOUR_API_TOKEN_WHERE> #no-archive = true

#[supybot] #raw_index = supybot_demo-raw #enriched_index = supybot_demo_enriched

#[twitter] #raw_index = twitter_demo-raw #enriched_index = twitter_demo_enriched #api-token = <YOUR_API_TOKEN_WHERE> #no-archive = true #sleep-for-rate = true #sleep-time = 300

#studies based on enriched indexes

[enrich_demography:git]

[enrich_areas_of_code:git] in_index = git-raw out_index = git-aoc_enriched

[enrich_onion:git] in_index = git out_index = git-onion_enriched contribs_field = hash

[enrich_onion:github] in_index_iss = github_issues_enrich in_index_prs = github_issues_enrich out_index_iss = github_issues_onion_enriched out_index_prs = github_prs_onion_enriched

[enrich_geolocation:user] location_field = user_location geolocation_field = user_geolocation

[enrich_geolocation:assignee] location_field = assignee_location geolocation_field = assignee_geolocation

#[enrich_extra_data:github] #json_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt

[enrich_backlog_analysis] out_index = github_enrich_backlog interval_days = 7 reduced_labels = [bug,enhancement] map_label = [others, bugs, enhancements]

[enrich_demography:github]

[enrich_feelings] attributes = [title, body] nlp_rest_url = http://localhost:2901

[enrich_cocom_analysis] out_index = cocom_chaoss_study interval_months = [3]

My projects.json:

{ "grimoirelab": { "meta": { "title": "GrimoireLab" }, "git": [ "https://github.com/chaoss/grimoirelab" ], "github": [ "https://github.com/chaoss/grimoirelab" ], "github:repo": [ "https://github.com/chaoss/grimoirelab" ], "github2:issue": [ "https://github.com/chaoss/grimoirelab" ], "github2:pull": [ "https://github.com/chaoss/grimoirelab" ], "cocom": [ "https://github.com/chaoss/grimoirelab" ]
} }

ES indices:

Cocom panel error:

Let me know if you need more details. Thank you for your time to inspect this problem in advance.

[zhquan]

Hi @zzz945

Did you read this https://github.com/chaoss/grimoirelab-sirmordred#cocom- and https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables?

I hope it helps you.

Best, Quan

[zzz945]

Thank you for this quick response. Yes. I have installed Cloc in the mordred docker. And it seems it failed during checking out a commit, before doing any code analysis. Nomos and ScanCode are also needed to be installed, right?

[zhquan]

@zzz945 I think so.

I have no issue executing cocom setup.cfg

[cocom]
raw_index = cocom_chaoss
enriched_index = cocom_chaoss_enrich
category = code_complexity_lizard_file
studies = [enrich_cocom_analysis]
branches = master

[enrich_cocom_analysis]
out_index = cocom_chaoss_study
interval_months = [3]

projects.json

{
  "chaoss": {
    "cocom": [
      "https://github.com/chaoss/grimoirelab"
    ]
  }
}

Result

$ curl -k https://xxx:xxx@localhost:9200/_cat/indices | grep cocom
yellow open cocom_chaoss_enrich                  qLFWTrWcTOqwfpq5qfeDZA 1 1 2584   5   6.5mb   6.5mb
yellow open cocom_chaoss                         WY091YhZTBW5-t2WsX1_pQ 1 1  506   1   1.7mb   1.7mb
yellow open cocom_chaoss_study                   q7hdbRHQRz-KE_sQZlqjiQ 1 1  638 100 247.4kb 247.4kb

perceval.errors.RepositoryError: [Errno 2] No such file or directory: '/tmp/worktrees/grimoirelab-git'

Maybe you have no permission to write on the dir /tmp/worktrees (by default). You can change that by adding ´worktree-path´ in the setup

[cocom]
raw_index = cocom_chaoss
enriched_index = cocom_chaoss_enrich
category = code_complexity_lizard_file
studies = [enrich_cocom_analysis]
branches = master
worktree-path = /PATH/worktrees/

[vchrombie]

Hi @zzz945, did you get some time to try the @zhquan comment.

[zzz945]

I have tried @zhquan comment but unfortunately it doesn't work for me. My setup.cfg is almost the same and I tried several work-tree paths but none of them works. I tried to inspect the issue and went into the docker and found nothing in /tmp/worktrees, as the following picture shows.

By the way, I am using Macbook Pro with Intel Core i5 chip. I wonder if this matters. I appreciate you guys following up with me.

[zhquan]

Hi @zzz945

It seems that Mordred cannot write on /tmp/worktrees so try another directory as worktree-path = /home/bitergia/worktrees/.

Some executions:

Set worktree-path = /home/bitergia/worktrees/

2022-04-08 10:30:38,554 - sirmordred.task_collection - INFO - [cocom] collection starts for https://github.com/chaoss/grimoirelab
2022-04-08 10:30:38,668 - grimoire_elk.raw.elastic - INFO - [cocom] Incremental from: None for https://github.com/chaoss/grimoirelab
2022-04-08 10:38:34,322 - graal.graal - INFO - Fetch process completed: 509 commits inspected
2022-04-08 10:38:35,487 - grimoire_elk.elk - INFO - [cocom] Done collection for https://github.com/chaoss/grimoirelab
2022-04-08 10:38:35,487 - sirmordred.task_collection - INFO - [cocom] collection finished for https://github.com/chaoss/grimoirelab
2022-04-08 10:38:35,488 - sirmordred.task_collection - INFO - [cocom] collection phase finished in 00:07:56
2022-04-08 10:38:45,559 - sirmordred.task_enrich - INFO - [cocom] enrichment phase starts
2022-04-08 10:38:45,596 - sirmordred.task_enrich - INFO - [cocom] enrichment starts for https://github.com/chaoss/grimoirelab
2022-04-08 10:38:52,384 - grimoire_elk.enriched.cocom - INFO - [cocom] 2592 items inserted

Without set worktree-path (by default is /tmp/worktrees)

2022-04-08 10:28:28,487 - sirmordred.task_collection - INFO - [cocom] collection phase starts
2022-04-08 10:28:28,490 - sirmordred.task_collection - INFO - [cocom] collection starts for https://github.com/chaoss/grimoirelab
2022-04-08 10:28:28,607 - grimoire_elk.raw.elastic - INFO - [cocom] Incremental from: None for https://github.com/chaoss/grimoirelab
2022-04-08 10:28:29,123 - graal.graal - ERROR - Analysis failed at 83c4d1aa3637929df55670031c857857318a3f9b
2022-04-08 10:28:29,123 - grimoire_elk.elk - ERROR - Error feeding raw from cocom (https://github.com/chaoss/grimoirelab): Impossible to checkout the worktree /tmp/worktrees/grimoirelab-git at 83c4d1aa3637929df5
5670031c857857318a3f9b
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/perceval/backends/core/git.py", line 1341, in _exec
    cwd=cwd, env=env)
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/worktrees/grimoirelab-git': '/tmp/worktrees/grimoirelab-git'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 334, in checkout
    self._exec(cmd_checkout, cwd=self.worktreepath, env=self.gitenv)
  File "/usr/local/lib/python3.7/dist-packages/perceval/backends/core/git.py", line 1344, in _exec
    raise RepositoryError(cause=str(e))
perceval.errors.RepositoryError: [Errno 2] No such file or directory: '/tmp/worktrees/grimoirelab-git': '/tmp/worktrees/grimoirelab-git'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/elk.py", line 171, in feed_backend
    ocean_backend.feed(**params)
  File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/raw/elastic.py", line 231, in feed
    self.feed_items(items)
  File "/usr/local/lib/python3.7/dist-packages/grimoire_elk/raw/elastic.py", line 247, in feed_items
    for item in items:
  File "/usr/local/lib/python3.7/dist-packages/perceval/backend.py", line 316, in fetch
    for item in self.fetch_items(category, **kwargs):
  File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 197, in fetch_items
    raise e
  File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 189, in fetch_items
    self.graalRepo.checkout(commit['commit'])
  File "/usr/local/lib/python3.7/dist-packages/graal/graal.py", line 338, in checkout
    raise RepositoryError(cause=cause)
perceval.errors.RepositoryError: Impossible to checkout the worktree /tmp/worktrees/grimoirelab-git at 83c4d1aa3637929df55670031c857857318a3f9b

[zzz945]

Update:

Good news is I able to worked around this problem using micro.py thanks to the above link mentioned by zhquan. My way is:

1. Run grimoirelab using docker compose: docker-compose up -d
2. Login into mordred docker: docker exec -it 【your_mordred_docker_id】 /bin/sh
3. Install cloc (see this)
4. Clone grimoirelab-sirmordred since I need micro.py: git clone https://github.com/chaoss/grimoirelab-sirmordred
5. sudo python3 ./grimoirelab-sirmordred/sirmordred/utils/micro.py --raw --enrich --cfg ./conf/setup.cfg --backends cocom

However, if I didn't add sudo in step5 command, I would get the same error as before "Impossible to checkout the worktree". Therefore, the problem must have something to do with permissions. Fixing it is beyond my skills, so hopefully the team could fix it in some days. Please let me know if you gays need more details about this problem.

[lbergmans]

Hi, I ran into the same problem, while I am attempting to run the whole setup. so far I have not been able to get things running. What appears to be the case, is that the system removes the folder where it wants to clone, before actually doing this; when you create the folder manually, it will be gone after running mordred.... perhaps someone has a clue which code might do this (seems like a cleaning action that is too thorough...) In fact, I once timed the creation of the folder such that it started producing stuff for one cycle--never again, alas..

An additional note, somewhat related: when I tried to install cloc, I received the error message E: Unable to locate package cloc; the solution to that issue is to first run sudo apt-get update, and then sudo apt-get install cloc (found on stackoverflow, worked for me). Perhaps we (I?) can add this to the tutorial notes?

I would prefer a solution without micro.py; just running stuff out of the box seems much desirable, and should be more portable to others.. Also, I am not able to do a sudo from my regular terminal (on a somewhat restricted company MacBook).... If someone has further insights or suggestions, that would be great!

[vchrombie]

An additional note, somewhat related: when I tried to install cloc, I received the error message E: Unable to locate package cloc; the solution to that issue is to first run sudo apt-get update, and then sudo apt-get install cloc (found on stackoverflow, worked for me). Perhaps we (I?) can add this to the tutorial notes?

I think this is already present in the tutorial, we have a troubleshooting page. https://chaoss.github.io/grimoirelab-tutorial/docs/getting-started/troubleshooting/#cloc-installation

Do you think we should add it anywhere else? Please let us know.

[lbergmans]

I missed your quick response :-) What I was trying to point out (I am formatting it in the tutorial style):

Diagnosis

when trying to install cloc with apt-get you may encounter E: Unable to locate package cloc

Solution

First execute the following command: sudo apt-get update And after that execute sudo apt-get install cloc

But perhaps it is most useful to add this as a tip to the current cloc installation section?

[vchrombie]

Now, I understand it.

But perhaps it is most useful to add this as a tip to the current cloc installation section?

Please feel free to open a PR for adding it.

[lbergmans]

meanwhile: is it worthwhile to check out the mystery of the missing-because-auto-deleted worktrees folder?? there must be some code that does this?

[sduenas]

Closing this ticket due to no activity. Feel free to reopen it again if you work on this.