noobaa-core icon indicating copy to clipboard operation
noobaa-core copied to clipboard

NSFS | NC| CLI | noobaa node command taking longer time to exit

Open PravinRanjan10 opened this issue 1 year ago • 6 comments

Environment info

  • NooBaa Version: noobaa-core-5.15.0-20240102.el8.x86_64.rpm

Actual behavior

  1. All node commands taking longer time to exit after printing the status

Expected behavior

  1. it should exit immediately after printing the status

Steps to reproduce

  1. run any node command

More information - Screenshots / Logs / Other output

PravinRanjan10 avatar Jan 05 '24 16:01 PravinRanjan10

intermittent issue, try to reproduce it in @PravinRanjan10 machine but could not reproduce it, will try again once it is reproducible in his system.

naveenpaul1 avatar Jan 12 '24 05:01 naveenpaul1

@PravinRanjan10 @naveenpaul1 do you still see this issue? or can we close this bug?

romayalon avatar Jan 22 '24 15:01 romayalon

Did a test with @PravinRanjan10 in his rhel 9 and 8 machine, We could reproduce the issue with rhel8 but not with rhel9, And confirm that it is an issue with rhel8, I am trying to fix this in rhel8 with Pravin.

naveenpaul1 avatar Feb 07 '24 11:02 naveenpaul1

@PravinRanjan10 @naveenpaul1 I tried it with @udaysk23 (Thanks Uday!) on a RHEL8 VM (noobaa-core-5.15.0-20240207.el8.x86_64.rpm upstream) and we couldn't reproduce it..

romayalon avatar Feb 08 '24 13:02 romayalon

This issue is still reproducible on our cluster of rhel8. @naveenpaul1 is looking into it. FYI @romayalon

PravinRanjan10 avatar Feb 13 '24 10:02 PravinRanjan10

Hi @PravinRanjan10 we identified a possible cause for this issue and the fix would be to disable random seed read. A temporary workaround for disabling random seed read is adding env variable DISABLE_DEV_RANDOM_SEED to true for each manage nsfs command. Something similar to below. DISABLE_DEV_RANDOM_SEED='true' usr/local/noobaa-core/bin/node usr/local/noobaa-core/src/cmd/manage_nsfs account list This will be a temporary solution for setting env DISABLE_DEV_RANDOM_SEED, We will come up with a proper solution ASAP. special thanks to @guymguym @romayalon

naveenpaul1 avatar Feb 15 '24 11:02 naveenpaul1

@PravinRanjan10 @naveenpaul1 The issue was fixed on #7864, @PravinRanjan10 can you please validate it? The fix is currently on master, and should be backported to 4.15.1 in the near future

romayalon avatar Mar 10 '24 09:03 romayalon

Tested with build : noobaa-core-5.15.2-20240425-stage_5.15.3.el8.x86_64.rpm on rhel8 This issue is fixed!!.

cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.8 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.8" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.8" [root@prscale-12 ~]# time noobaa-cli bucket list 2>/dev/null { "response": { "code": "BucketList", "reply": [] } }

real 0m0.649s user 0m0.789s sys 0m0.065s

PravinRanjan10 avatar May 03 '24 18:05 PravinRanjan10