manageiq-pods
manageiq-pods copied to clipboard
ManageIQ
Allow vddk to be dynamically added without rebuilding container images
The current VDDK installation in pods involves rebuilding the container images, but that is not convenient for users. As discussed, we need an easier solution.
~~Redistribute the VDDK~~
- ~~Need to be a Select level member of the VMware TAP program[ref]~~
~~ConfigMap~~
- ~~Limited to 1MiB, file is ~20MiB~~
Keep the current solution of rebuilding the images
- High maintenance for upgrades
Single PersistentVolume mounted to all pods
- limits the backing storage types that we can run on
- More things to worry about for backup and restore
- Convenient that we only need to upload to a single place
Multiple PersistentVolumes (one per worker pod)
- Orchestrator will have to set them up copy file(s) in
- More things to worry about for backup and restore
- Don't need to worry about storage types, but requires the cluster to have multiple volumes available
BinaryBlobs
- Allows a single place to upload files
- Don't need to worry about backup / restore
- Don't need to worry about available volumes or their backing storage
- May be tricky to coordinate the initial upload and restarting of workers to make use of the new blob
- Every pod start would require a download and install
Download on demand
- Allow the user to specify a location to download the VDDK, like an FTP server or something
- User would have to permanently store the VDDK somwhere.
- Adding on the fly would require a worker restart on podified
- Every pod start would require a download and install
Thoughts @Fryguy & @agrare ?
I'd also like @jrafanie and @chessbyte to add their thoughts here.
ConfigMap is right out, because of the size limitations, so I crossed it out in the OP.
I'd prefer whatever solution we use to be consistent across both appliances and podified, so that we can reduce any speciality. That's another upside to BinaryBlobs, because we can document it as the preferred way for both appliances and podified.
Slight downside to BinaryBlobs is we will need to have a specific UI for uploading the file, which hopefully we can do in a pluggable way. We probably should block that to super-admins only or introduce a new RBAC setting or something for that.
In the private discussion, @agrare mentioned that we really only need the VDDK on the operation worker, which, I think, there can only be one replica of, however there is one per VMware EMS instance.
Another advantage for BinaryBlobs option is that the admin doesn't have to install the package on every appliance that could run a Vmware operations worker, single upload and they're done. This could also simplify updates to the package.
Slight downside to BinaryBlobs is we will need to have a specific UI for uploading the file, which hopefully we can do in a pluggable way. We probably should block that to super-admins only or introduce a new RBAC setting or something for that.
Yeah, I was thinking a simple HTTP PUT to the API, but UI would be nice too. I agree, it should be super-admin only
If we do BinaryBlobs, we have to make sure that the blob record is attached to something or it will get purged...the docs would have to tell the user to attach it to the enterprise or something.
HTTP PUT is a cool idea. I'm concerned they may try to upload multiple and we'd have to either disallow that or figure out the "right" one. Perhaps, instead of directly on a /binary_blobs endpoint, we could have an API on the EMS or perhaps the Enterprise? That API could do the right thing under the hood.
We'll probably need an API for the UI anyway, so that's a good start.
If we do BinaryBlobs, we have to make sure that the blob record is attached to something or it will get purged...the docs would have to tell the user to attach it to the enterprise or something.
Why not attach it to the VMware Provider EMS?
Why not attach it to the VMware Provider EMS?
The user would be required to upload it multiple times per VMware EMS instance, if they have multiple VMwares.
Why not attach it to the VMware Provider EMS?
Because we only need one instance of it
Just thought of another option, which I believe is along the lines of what the v2v team did.
Allow the user to specify a location to download the VDDK, like an FTP server or something. This could be set up as an extra tab in the vmware DDF form adding an endpoint/authentication for the vddk location. Then, when the operations worker starts, if there's a vddk defined, it can download and install it in the right place.
If we have the tgz file in BinaryBlobs then we will have to pull it down and extract it to the target dir. On an appliance this will only have to be done once but on podified it will have to be done everytime the pod starts up. Not a dealbreaker but not ideal
Yes I was comparing to the "mount a volume' option (unless that's already been NAK'd?)
On an appliance this will only have to be done once
On an appliance this will only have to be done once... per server
No, we haven't NAK'd yet - still trying to capture all the pros/cons and also new ideas.
Also we should technically add https://code.vmware.com/docs/6635/virtual-disk-development-kit-programming-guide/doc/GUID-282C600E-D986-4592-9206-70BF60DBF684.html to the OP to show we've considered it 😆
Added the download on demand and the redistribute options to the OP and also added @agrare's concern about the "on every pod startup"
What's required to tell the underlying code where the vddk is installed if we're not rebuilding images? I understand that uploading the files to the right location is important but I'd imagine most of our usages will be people having pods run with the installed vddk wherever it's located so runtime steps we need to take is pretty important too.
Allow the user to specify a location to download the VDDK, like an FTP server or something. This could be set up as an extra tab in the vmware DDF form adding an endpoint/authentication for the vddk location. Then, when the operations worker starts, if there's a vddk defined, it can download and install it in the right place.
The issue I see with this approach is that we will eventually have to support every type of file sharing under the sun (FTP, SFTP, Cloud Storage, ...). The BinaryBlob approach offers a single way to do it
@jrafanie @agrare how challenging is the install on first use of the Operations Worker?
@Fryguy with that approach, are we going to have security issues of what files are writable and where?
What's required to tell the underlying code where the vddk is installed if we're not rebuilding images? I understand that uploading the files to the right location is important but I'd imagine most of our usages will be people having pods run with the installed vddk wherever it's located so runtime steps we need to take is pretty important too.
We can set LD_LIBRARY_PATH to the target directory even if the it doesn't exist. That way when we extract the files there the env should already be set up.
I should note that I haven't gotten this working on an appliance yet, LD_LIBRARY_PATH doesn't seem to be honored or something.
I recall there being hacks that Jerry worked on that might be overriding the LD_LIBRARY_PATH
https://github.com/ManageIQ/vmware_web_service/blob/3273f463d7090f734ea14657d4565807b53e3209/lib/VMwareWebService/VixDiskLib/VixDiskLib.rb#L101
That looks like it just tacks on a specific directory to LD_LIBRARY_PATH, in my case I already have the proper directory in LD_LIBRARY_PATH
If we have the tgz file in BinaryBlobs then we will have to pull it down and extract it to the target dir. On an appliance this will only have to be done once but on podified it will have to be done everytime the pod starts up. Not a dealbreaker but not ideal
I think this could be an advantage. Yes, it will slow down boot time slightly, but it's super easy to update the package in the future. Just replace the BinaryBlob with the newer version and the next time a pod starts it gets the latest. I was thinking that appliances could do the same at appliance boot.
Allow the user to specify a location to download the VDDK, like an FTP server or something. This could be set up as an extra tab in the vmware DDF form adding an endpoint/authentication for the vddk location. Then, when the operations worker starts, if there's a vddk defined, it can download and install it in the right place.
The issue I see with this approach is that we will eventually have to support every type of file sharing under the sun (FTP, SFTP, Cloud Storage, ...). The BinaryBlob approach offers a single way to do it
I agree, and we're trying to drop similar things (like backup and restore to/from all of these locations)
Ah sorry, use of the VDDK is by the smartproxy not the vmware operations worker. I had originally had the entire smartstate scan performed by the vmware operations worker but changed it prior to merge to maintain the same performance of having multiple smart proxies.
@agrare I think I'm thinking of this PR (from 2014! https://github.com/ManageIQ/manageiq/pull/277)
I think this could be an advantage. Yes, it will slow down boot time slightly, but it's super easy to update the package in the future. Just replace the BinaryBlob with the newer version and the next time a pod starts it gets the latest. I was thinking that appliances could do the same at appliance boot.
...on podified. On appliances that will be complicated as we will need to compare what's on disk with what's in the database.
Okay I had vddk 7.0.2 which was "too new" so ffi-vix_disk_lib wasn't picking it up. Once I switched to 7.0.1 it worked.
Also need to note: the target directory doesn't have to exist for the worker to start up, but if it doesn't exist at startup then it is created after then it isn't picked up by ld. If the target directory does exist but is empty when the process starts up, then the .so is created after, then ld does pick it up. This just means whatever we want to use as the target directory has to exist on worker initialization even if it is empty if we got the BinaryBlob extract on startup approach.
We could also design it generically enough with BinaryBlobs to allow us to upload code patches as well (solving another podified problem).
I think this could be an advantage. Yes, it will slow down boot time slightly, but it's super easy to update the package in the future. Just replace the BinaryBlob with the newer version and the next time a pod starts it gets the latest. I was thinking that appliances could do the same at appliance boot.
...on podified. On appliances that will be complicated as we will need to compare what's on disk with what's in the database.
I was thinking we would overwrite it on every appliance boot too.