ondemand
ondemand copied to clipboard
OSC
run tests without scl
How do I run rake tests in /var/www/ood/apps/sys/dashboard wihtout scl since I am installing in ubuntu?
┆Issue is synchronized with this Asana task by Unito
a .deb file but from about 6 weeks ago.
I ask for the ability to run the tests because we cannot seem to get our slurm service recognized.
cluster.yml:
v2: metadata: title: "GPU_Cluster" login: host: "gpu" job: adapter: "slurm" host: "gpu" cluster: "cluster" bin: "/usr/bin" conf: "/etc/slurm/slurm.conf" acls:
- adapter: "group"
groups:
- "gpu_users" type: "whitelist" # optional, one of "whitelist" or "blacklist" batch_connect: basic: script_wrapper: | %s set_host: "host=$(hostname -A | awk '{print $2}')" vnc: script_wrapper: | %s set_host: "host=$(hostname -A | awk '{print $2}')"
-john
On Fri, July 15, 2022 9:52 am, Jeff Ohrstrom wrote:
Did you install from the source or did you use a nightly
.deb?-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2125#issuecomment-1185568183 You are receiving this because you authored the thread.
Message ID: @.***>
-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright
It looks like you have the gpu_users ACL. I'd ask if you are indeed in that group. To get that stuff to run you'll have to source something out of /opt/ood/ to get the right Ruby environment variables. I'd find /opt/ood -name enable to find the file.
I'm hacking a container for the same so I should have the correct command soon.
This is the command, but I'm now finding that it is dangerous to run this as root on nightlies. So be sure to run this as an unprivileged user because it may try to write/overwrite your assets.
source /opt/ood/ondemand/enable && rake test:jobs
That enable file is not in my distribution. -john
On Fri, July 15, 2022 12:58 pm, Jeff Ohrstrom wrote:
This is the command, but I'm now finding that it is dangerous to run this as root on nightlies. So be sure to run this as an unprivileged user because it may try to write/overwrite your assets.
source /opt/ood/ondemand/enable && rake test:jobs-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2125#issuecomment-1185729363 You are receiving this because you authored the thread.
Message ID: @.***>
-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright
OK, this is all that file is. Do you have an /opt/ood/gems directory?
export GEM_PATH="/opt/ood/gems:${GEM_PATH:+:${GEM_PATH}}"
export RUBYLIB="/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:${RUBYLIB:+:${RUBYLIB}}"
some progress. typo in the gpu_cluster acl which should have been gpu-cluster group. In any case, I dropped that section and though jobs still do not appear, the cluster is in the cluster drop down menu.
running the rake tests as a non priveledged user in /var/www/ood/apps/sys/dashboard fails with:
Error reading app/assets/builds/application.css.map: permission denied.
though the file is world readable.
Alsop tries to write many updates to /var/www/ood/apps/sys/dashboard/app/assets/builds/
-john
On Fri, July 15, 2022 1:28 pm, Jeff Ohrstrom wrote:
OK, this is all that file is. Do you have an
/opt/ood/gemsdirectory?export GEM_PATH="/opt/ood/gems:${GEM_PATH:+:${GEM_PATH}}" export RUBYLIB="/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:${RUBYLIB:+:${RUBYLIB}}"-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2125#issuecomment-1185752783 You are receiving this because you authored the thread.
Message ID: @.***>
-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright
ok some progress. Another admin noticed error messages where a process was trying to write to var/log/ood_slurm.log which didn't exist and couldn't be created. After it was created an opened, we could finally see job listings.
-john
On Fri, July 15, 2022 1:28 pm, Jeff Ohrstrom wrote:
OK, this is all that file is. Do you have an
/opt/ood/gemsdirectory?export GEM_PATH="/opt/ood/gems:${GEM_PATH:+:${GEM_PATH}}" export RUBYLIB="/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:${RUBYLIB:+:${RUBYLIB}}"-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2125#issuecomment-1185752783 You are receiving this because you authored the thread.
Message ID: @.***>
-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright
Yea unfortunately there's a bug here, so you may not be able to run it. I'm working on a patch right now.
After it was created an opened, we could finally see job listings.
Very good! Let me know if you need anything else. I'm closing this as I made a bug ticket for running rake test:jobs in #2133, but you can reopen it or open another if you need.
yes I am but it doesn't matter whether I have the acl section or not. Nothing displays. Seems to be no way of debugging. I have put a custom logger in slurm.rb but it is not triggered so I think slurm.rb is not even being invoked even though the adapter is slurm.
-john
On Fri, July 15, 2022 12:50 pm, Jeff Ohrstrom wrote:
It looks like you have the
gpu_usersACL. I'd ask if you are indeed in that group. To get that stuff to run you'll have to source something out of/opt/ood/to get the right Ruby environment variables. I'dfind /opt/ood -name enableto find the file.I'm hacking a container for the same so I should have the correct command soon.
-- Reply to this email directly or view it on GitHub: https://github.com/OSC/ondemand/issues/2125#issuecomment-1185722103 You are receiving this because you authored the thread.
Message ID: @.***>
-- Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright
Can you attach your cluster.d file? there could be something broken there, though logs in /var/log/ondemand-nginx/$USER would have an error message around the same.
