document conda environment behavior and consider not enabling conda in AWS custom AMIs
source : https://github.com/skypilot-org/skypilot/pull/5303#discussion_r2057023025
While working on https://github.com/skypilot-org/skypilot/pull/5303#discussion_r2057023025, it's become apparent that we don't really document what conda environment is set up where (and why), and it is quite confusing to navigate it.
The fix to https://github.com/skypilot-org/skypilot/issues/4814 and https://github.com/skypilot-org/skypilot/issues/5265 involved deactivating conda environment that we activate if the image is a docker image. Do we need to do that, or can we get away with not activating the conda environment in the first place? I don't know, and I didn't want to tackle it in that PR - so here is the ticket to track this work.
Another question to consider - users don't expect conda environments to be set by us when using docker images, because the point of using a docker image is that they already have the environment they want and we should not modify it. Does the similar logic apply to custom AMIs in AWS, and in that case, should we apply a similar behavior for custom AMIs?