sftp-gcs
sftp-gcs copied to clipboard
kolban-google
Multiple users or ability to access subfolders in buckets
Hi !
Thanks for your package!
Although I encounter the No supported authentication methods available (server sent: ) issue in every sftp client apart from the shell (I don't understand why FileZilla sends a none authentication method though)...
I wondered : since you can't launch the daemon on the same port (I thought of maintaining a bucket per user), but you can't also give a subdirectory as a bucket (I tried it gave me a Couldn't read directory: Failure error : How can I use this to manage a single instance of the daemon for multiple users ?
Howdy my friend. At your service!! I'm always delighted to partner to get problems solved so realize that I am here and reading/thinking about all posts. Am I sensing two puzzles in the post? One about the authentication message and one on user permissions?
I'm thinking about the later one first. As it currently stands, the demon runs as a single GCP service identity. This means that all interactions with GCS are done under the auspices of a single identity. This means that whomever you are ... if you pass the SFTP login access, then you will be reading and writing to GCS using one single identity.
What might we do to resolve this?
Option 1 We could potentially use GCP Authentication where the SFTP access asked for a users Google userid/password, authenticated as that user and then your access to GCS would be done as that user.
Option 2 We realize that this is an open source project and will never be as full featured or functional as a vendor product. There are some for fee based offerings that may have substantially more functions. For example:
- https://docs.trillo.io/trillo-applications/filemanager-and-sftp
- https://www.filemage.io/
Both of these are available (I believe) for immediate deployment on GCP through the GCP Marketplace.
That's commitment ! Such a fast reply (Edit : forgot to THANK YOU) The first problem is not a real problem since I guess it's a matter of inherent bug for sFTP Filezilla version ? I see ... Of course it is an open source project, it worked so magically OOB I thought it might as well solve all my issues ! So is this a restriction from Google (the subdirectory thing) ? Or would it be possible to code it ?
It should be possible to code it up. I have to think about it some. Let's make sure that I have understood the requirements. Lets imagine you have two GCP users. User "A" and user "B". Let's now also imagine you have a GCS bucket called gs://my-bucket. I'm assuming that there might be two folders ... /user-A-files and /user-B-files. We want user "A" to login and do work against their folder and not be allowed to do work against user "B" ... and visa versa. I'm assuming that security of "who is allowed to do what" (otherwise known as authorization) is going to be governed by GCS security and GCP IAM. As such, the puzzle from the implementation of this project becomes "How do we make requests to GCS as user 'A' vs user 'B' as opposed to the current story which is a configured constant user?". The only way I could see that ever working is if the demon logically performs requests "as" those users which means that the demon has to "become" those users (even transiently). I have never researched (and hence attempted) to perform such logic before so don't know what would be involved (yet). My guess is that we would have to learn the passwords for the users which would mean that we would have to supply the passwords via SFTP login.
IF this were implemented (and I'm not committing to that :-) ... does this sound/feel like a solution to the puzzle for your needs?
I love open-source projects omg Alright that would actually be almost too perfect indeed ! Couldn't we hack it more easily? As in : if we launch the server with a subdirectory gs://bucket/userA -> that would be seen as root for user A; so we don't even have to mess with IAM right do we ? Worst case scenario I maintain a sFTP port per user (since its with network tags, it's not actually an issue) ; but I wanted to avoid a extra bucket maintenance ?
Because if so; since - as you said - this is not a vendor product and it is not meant to be one for this project therefore I don't think that could be an issue as I don't need 1,000 users -> opening a specific port would make us launch a daemon per user per port; but with one single bucket
Do you think this workaround would be ok ?
It would be relatively easy to constrain file access to forced root folders on per instance basis. When a request is made to access file "X", that could be remapped to "/userA/X". We would also have disable relative folders (eg. "..") so we can't access "../userB/Y".
How would you be asking your users to connect? Via a userid/password or a public key? If via userid/password, one option would be that we could supply a configuration file that maps users to passwords and root directories?
Ok I believe you found the way So the bucket will always serve a subdirectory user1 connects to bucket gs://bucket-name, it automatically reroutes to gs://bucket-name/user1/ -> sweet
User/password seems the easiest here Now : we could eventually mimic a sftp group users with indeed a configuration file with simple user:password configurations. But as you said; one daemon process will always mean a single identity ? Or would we be able to actually authenticates on-the-fly ?
The "authenticates on the fly" was what I originally had in mind but I don't know how to do that yet (nor know if its possible). The demon would appear to have to know the userid/password for a user and then authenticate with GCP as that user and then run the GCS APIs as that identity once authorized by GCP. I've never walked that path ... it may be possible or it may not. To be continued on that front. In the story with "remapping" paths, the demon would still run as one identity that would have full authority over the bucket. That would still be a single identity.
I dont mind maintaining an open port/sftp process per user tbh, this sounds like a great workaround already Doing the auth on the fly is too much of a headache ! Where should I look in the code to find out about the subfolder remapping ? I'll just fork it !
Sadly, its not going to be one single place (as the code currently exists). Scattered througout the code (search on the string "bucket") you will find lots of places where GCP bucket access is performed. It seems all of these take a file name as input. Each place would have to be found and replace with a call to a function that does not yet exist. For example .. if today we had:
stream = bucket.openStream(fileName);
we would have to change that to:
stream = bucket.openStream(mapFileName(fileName));
where we would then add:
function mapFileName(fileName) {
return globalMapPath + fileName;
}
Now ... I haven't thought all this through yet ... so if you do decide to fork and play ... go for it and enjoy. But I need to think more on this whole notion before I'll accept a pull request to merge it all back in if you change. I'd prefer to get the design/thinking right than make the demon have hundreds of options that contradict each other.
Oh wow for sure ! I didn't mean to mingle with this basecode hence the fork How could we move on from here?
Since this is a pet project and not one for work, it has to be outside of work. Next week I have to resit one of the GCP certifications (they expire after 2 years) so I have to swot all weekend otherwise I could have studied then. As it stands, we might be looking a few weeks out. I'm assuming this is for a personal project or some small/medium sized business? If it was corporate, I'd have said email at [email protected] your business details and I could reach out to your Google colleagues and maybe get approval to carve out some work hours based on that.
It's actually for one of my clients ! You are a great developer, this is what I love the most about open-sourcing, kudos!
I'll type you an email then with my corporate address from this client and continue from there ?