FluentFTP
FluentFTP copied to clipboard
robinrodricks
How to test FluentFTP against all supported FTP servers?
UPDATE: We now have extensive automated integration testing for many FTP servers. See the Automated Testing page for more info. You can help by adding more FTP server dockers!
We now support over two dozen servers, but our tests have not kept up the pace. We don't have a way to test FTP functionality against all types of FTP servers. Each server has different handling, so its not possible to setup one server alone, we need to setup most/all of these servers in some way and create automated tests.
I ask you, community, do you have any suggestions?
Some ideas:
- Create docker containers for individual FTP servers and create setup scripts to help setup a full test-suite locally
- Create steps to setup servers on local machines so advanced developers can set it up manually
- Create automated tests for all main methods, against all FTP servers that we support (or at least the major ones)
- Create a few test files that test various cases of upload and download (small files, large files, binary/text)
Methods/workflows that need to be tested
- Connection/disconnection
- File upload/download in binary and ASCII mode
- Folder upload/download
- Get listing in recursive and non-recursive modes
- Getting file exists, modified time, chmod, permissions
- File transfer with weird file names (including special chars, spaces, unicode chars)
- Get hash/checksum
- Renaming/moving files and folders
Hmmm - in the case of IBM z/OS and OS/400, these are licensed software and need specialised hardware. There actually IS an emulator for IBM System/z, which one could run in a docker container but there is no operating system available that needs no license and provides a modern z/OS CS FTP server. So no dice for these two server types. I currently run a test program against my own z/OS system, which is not publicly available.
I am also looking into this though at a smaller scale. For e.g. I want to test on a server that support TLS only, and one that do not have any; one that allow anonymous and one the requires login.
Right now I simply do manual test by switching settings from a FileZilla Server. Will just leave a comment here to learn any useful trick.
I would suggest a somewhat parallel path with more proper unit tests. Which might be the most practical way to (mostly) test some implementations (like OS/400, as FanDjango points out).
For integration tests against real ftp implementations I would suggest writing the tests with a default target of some portable ftp server that can be easily included (for local execution). And make a build pipeline to test each of the other feasible ftp servers.
I would think that would be the way to reliably run the tests often.
We now support over two dozen servers,
The idiosyncracies of different servers and supporting these, reminds me of other projects of mine, where it was also impossible to handle quality issues by pre-testing. As much as I am impressed by the amount of effort put into modern unit testing, I still believe that nothing beats exposing your code to a user and his environment - one of the most hostile scenarios imaginable.
But that also means being a little more responsive - a one or two week cycle for a reaction to a glitch is a bit of a problem.
I agree that exposing your code to real world users is the ultimate test. Though avoiding that your end users experience your errors is generally the goal. After all, experiencing errors is annoying and results in lost productivity.
I would say that unit tests are a good first line of defense. They are quite decent at showing side effects of changes you make. And they are good for reproducing bugs and check the fix. Quite a lot of developers (including me) will find that comforting as it reduces the odds of introducing unfortunate side effects.
A basic test suite of operations that should "work everywhere" against as many real ftp servers as is feasible should give the same effect for the differences between servers, early warning if we break something.
early warning if we break something
I definitely agree.
I think docker is the way to go, then, just as robin originally posted. Lots of dockers....
Perhaps one could also use (as a level-one acid test) one (or more) publicly available servers - there are some out there that are pretty standard and seem to be quite permanent. These would also have realistic round trip times, to help test against timing problems.
+1 for docker. I would think there are open source builds and/or ready made containers read to pull for most of the servers. More can easily be created for those that don't exit. The work here can be split up by software and OS. Most CI/CD pipelines have support for running docker.
With that said, is it it overkill to test OS and software combinations? For example, the first server listed as supported, PureFTPd, has two Docker builds:
- https://github.com/stilliard/docker-pure-ftpd
- https://github.com/crazy-max/docker-pure-ftpd
I assume there are more. Same for other software and investigation can be done into each.
The crazy-max readme declares support for:
- linux/amd64
- linux/arm/v6
- linux/arm/v7
- linux/arm64
- linux/386
- linux/ppc64le
In addition to OS and software combinations, is the CPU architecture something else that should be tested?
To start off, it would be good to start with the most common config of a given FTP server.
Something like:
| FTP software | Platform | Bits | Architecture |
|---|---|---|---|
| PureFTPd | Unix | x64 | AMD |
| VsFTPd | Unix | x64 | AMD |
| ProFTPD | Unix | x64 | AMD |
| WuFTPd | Unix | x64 | AMD |
| FileZilla Server | Unix | x64 | AMD |
| OpenVMS | VMS | ? | ? |
| Windows Server/IIS | Windows Server | x64 | Intel/AMD |
| Windows CE | Windows CE | ARM? | Intel/AMD |
| GlobalScape EFT | Unix | x64 | AMD |
| HP NonStop/Tandem | Tandem | ? | ? |
| Serv-U | Unix | x64 | AMD |
| Cerberus | Windows Server | x64 | Intel/AMD |
| CrushFTP | Unix | x64 | AMD |
| glFTPd | Unix | x64 | AMD |
| Homegate FTP | Unix | x64 | AMD |
| BFTPd | Unix | x64 | AMD |
| FTP2S3 Gateway | Unix | x64 | AMD |
| XLight | Windows | x64 | Intel/AMD |
| Solaris FTP | Unix | x64 | AMD |
| IBM z/OS | zOS | ? | ? |
| FritzBox | RTOS | 32 | MIPS32 |
| WS_FTP | Windows | x64 | Intel/AMD |
| PyFtpdLib | Unix | x64 | AMD |
| Titan FTP | Windows | x64 | Intel/AMD |
If there are mistakes in this, please correct it and reply, this is the source:
| FTP software | Platform | Bits | Architecture
| ------------------ | ------- | --- | --------- |
| PureFTPd | Unix | x64 | AMD |
| VsFTPd | Unix | x64 | AMD |
| ProFTPD | Unix | x64 | AMD |
| WuFTPd | Unix | x64 | AMD |
| FileZilla Server | Unix | x64 | AMD |
| OpenVMS | VMS | ? | ? |
| Windows Server/IIS | Windows Server | x64 | Intel/AMD |
| Windows CE | Windows CE | ARM? | Intel/AMD |
| GlobalScape EFT | Unix | x64 | AMD |
| HP NonStop/Tandem | Tandem | ? | ? |
| Serv-U | Unix | x64 | AMD |
| Cerberus | Windows Server | x64 | Intel/AMD|
| CrushFTP | Unix | x64 | AMD |
| glFTPd | Unix | x64 | AMD |
| Homegate FTP | Unix | x64 | AMD |
| BFTPd | Unix | x64 | AMD |
| FTP2S3 Gateway | Unix | x64 | AMD |
| XLight | Windows | x64 | Intel/AMD |
| Solaris FTP | Unix | x64 | AMD |
| IBM z/OS | zOS | ? | ? |
| FritzBox | RTOS | 32 | MIPS32 |
| WS_FTP | Windows | x64 | Intel/AMD |
| PyFtpdLib | Unix | x64 | AMD |
| Titan FTP | Windows | x64 | Intel/AMD |
Maybe you should add FileZilla Server on windows ?
| FTP software | Platform | Bits | Architecture |
|---|---|---|---|
| FileZilla Server | Unix | x64 | AMD |
| FileZilla Server | Windows | x64 | AMD |
UPDATE: We now have extensive automated integration testing for many FTP servers. See the Automated Testing page for more info.
You can help by adding more FTP server dockers!
Special thanks to tommysor for his major PR adding support for integration tests using Docker.