nebari icon indicating copy to clipboard operation
nebari copied to clipboard

[DOC] - Best practice for cleaning up conda-store pod

Open rsignell-usgs opened this issue 2 years ago • 4 comments

The ESIP Qhub conda-store is 80% full, and there are 12 old versions of the Pangeo environment:

``` (base) root@qhub-conda-store-worker-7f59cd69cb-2wmn5:/home/conda/users# ls 03a9d7aacb095fed84981a6e1dedbb702107f55898a3855459f4955c21631dba-20220725-170028-003799-110-BSL 0d2848b4ecab74128f07396bd22ebff1c8809e2a87e088f8c48b7ee5deb02028-20220716-151743-645051-90-pangeo 0fa467f5b20567c030a25626ce18c65498cf2eead525370c4ee1998cc9fabe62-20220722-183729-277894-106-pangeo 27e71f29bb63110a83f91e67b52a5f030d1eb53b3ae5d0b265461d5265e5e376-20220617-203553-416161-63-FI_summer 2a9c5a0d4a694196f9125acb790ea612686af75d78b623998ac6eb8a5f9fd990-20220602-185627-319221-32-bitinfo 4aabcb8ddab4b2a0f53f9ad165c61089f2eab6411ac1cfc8b1a083953440aad2-20220721-155516-216619-93-seppo 4aabcb8ddab4b2a0f53f9ad165c61089f2eab6411ac1cfc8b1a083953440aad2-20220721-155556-006639-94-seppo 64c916200203461f09131925b582ad8933ba986658d231fe64d81be91ffada1c-20220711-220404-252528-79-pangeo 7245b4ab07ea53a6d8b4d19208190c53d32f3303bab3604fc5946b89ca61158c-20220628-234826-045803-70-fi_intern 8b076ea83db2ce07cbf2ef2027201858cf30237f2fad1d9e0024fbca1715ccf3-20220711-220648-096267-80-pangeo 969d249aa987e53c821cae92e2922fe84cb7edad9aac2bdf4679e8e2b1d4ba67-20220718-213830-670314-91-pangeo 969d249aa987e53c821cae92e2922fe84cb7edad9aac2bdf4679e8e2b1d4ba67-20220718-213945-480289-92-pangeo af5bdbe0a7790d697b1783344c0610cbf9663446e8be97297bef4d120b77968e-20220715-153627-084835-84-pangeo bc35e257182bf6669e333014ea24dc79ac1d4ab1a6f6596e50712c3a2db124d8-20220722-184233-887037-107-gdptools bc35e257182bf6669e333014ea24dc79ac1d4ab1a6f6596e50712c3a2db124d8-20220722-184408-103755-108-gdptools bc35e257182bf6669e333014ea24dc79ac1d4ab1a6f6596e50712c3a2db124d8-20220722-184451-097450-109-gdptools bfc0e3ef7a0c1747b953f56f573768548e1167eb648541cc7474d76e128523cf-20220628-234703-684079-68-users-gkov c76828f5865f7ae55777df8e7d636e9fc586779db61c0f7a14eb31f02c1ed118-20220628-234738-891576-69-gkov ca19e52a6d5b5801536b3dcaf12bd5ca62d1f57fd274737d16f25c2f10038f4b-20220725-214332-802894-114-pangeo d21c1aca8d8b426a7b5d1df9c60856a7097c8de10e78316ac22fd2d8a95d9b11-20220811-143100-388871-119-pangeo d21c1aca8d8b426a7b5d1df9c60856a7097c8de10e78316ac22fd2d8a95d9b11-20220811-143100-415932-120-pangeo dae6e0cc72969d8a896b60dbf08153ec5da18bd7e7f8353ab81736f3db5c8c21-20220617-203205-039229-61-fi_intern dae6e0cc72969d8a896b60dbf08153ec5da18bd7e7f8353ab81736f3db5c8c21-20220617-203444-306118-62-fi_intern de5847a7ab98e8382085dd5a31ee5b6a0c7eb8fc3e9d390903f1361e5c61ceb1-20220525-124936-571313-31-seppo e6aa7ecb5d1fdfdc96b013e60f37b5b6476dfafe607fc361e6c796f9050d1e70-20220726-152312-460987-115-pangeo e6aa7ecb5d1fdfdc96b013e60f37b5b6476dfafe607fc361e6c796f9050d1e70-20220801-103922-189931-116-pangeo ```

Best practice used to be to ssh into the conda-store-worker pod (e.g. using k9s) and just deleting any no-longer-needed directories (environments).

Is that still okay to do?

Is it now preferable to use the conda-store gui to delete?

It would be nice to document this. If it's documented already, I apologize in advance -- I didn't find it.

rsignell-usgs avatar Aug 21 '22 16:08 rsignell-usgs

@HarshCasper can you check on this and also update docs.

I'm reasonably sure that the k9s method is still the best current practice. I'm not sure that deleting from the GUI actually deletes the data (vs just hiding it). Also, now that we have argo workflows we really need to look into a triggered or scheduled job to do this.

dharhas avatar Aug 22 '22 12:08 dharhas

Okay, so I can still delete old envs by doing rm -rf <old env> on the conda-store worker pod using k9s? (I just wanted to make sure I'm not going to mess anything up)

rsignell-usgs avatar Aug 22 '22 18:08 rsignell-usgs

Just a note that removing conda environments by using k9s to shell into the conda-worker pod and using rm -rf on the environments in /home/conda/ worked nicely.

I'm now at 34% full on the conda-store pod instead of 94%!

rsignell-usgs avatar Aug 23 '22 18:08 rsignell-usgs

Hey @rsignell-usgs I am just following up here - is there anything else we can help with? If this issue is resolved I will go ahead and close this.

Also - this looks like a nitfy trick for folks to be aware of. Would it be ok if I add it to our Troubleshooting section? https://www.nebari.dev/troubleshooting

Just a note that removing conda environments by using k9s to shell into the conda-worker pod and using rm -rf on the environments in /home/conda/ worked nicely.

I'm now at 34% full on the conda-store pod instead of 94%!

trallard avatar Oct 04 '22 15:10 trallard

We have documented the process in the FAQ section of the docs here.

iameskild avatar Aug 07 '23 16:08 iameskild