Implemented proposal: `download_dir` to download a whole directory

[sylvaticus]

Hello, I implemented a function to download all files in a given FTP directory, recursively or not.

Let me know if you would be interested, I'll clean it up a bit, make a docstring and create a pull request if it is the case... Note that this function uses the modified readdir function that I proposed in #122. Also, due to the bug in #121, download_dir can't be used if before to its call the cd function has been used, as all files to download are internally given using their full path.

This implementation answers my original request in #119 .

The function:

function download_dir(ftp,dir="";as="",verbosity=0, recursive=true, force=false, dryrun=false, mode=binary_mode, ftp_basepath="",dest_basepath="" ) # latest two used for recursion only
    (dir == "")  && (dir = pwd(ftp))
    if as == ""
        if endswith(dir,'/')
            as = split(dir,'/')[end-1]
        else
            as = split(dir,'/')[end]
        end
    end
    if ftp_basepath == ""
        if startswith(dir,"/")
            ftp_basepath = dir
        else
            ftp_basepath = joinpath(pwd(ftp),dir)
        end
    end
    (dest_basepath == "") && (dest_basepath = as)
    verbosity > 0       && println("Processing ftp directory `$(ftp_basepath)`:")
    items = FTP_readdir(ftp,dir;details=true)
    if !isempty(items)
        if !ispath(dest_basepath)
            mkdir(dest_basepath)
        elseif force
            rm(dest_basepath,recursive=true)
            mkdir(dest_basepath)
        else
            error("`$(dest_basepath)` exists on local system. Use `force=true` to override.")
        end
    end
    for item in items
        ftpname  = joinpath(ftp_basepath,item.name)
        destname = joinpath(dest_basepath,item.name)
        if(item.type != 'd')
            verbosity > 1 &&  print(" - downloading ftp file `$(ftpname)` as local file `$(destname)`...")
            if ispath(destname)
                if force
                    rm(destname)
                else
                    error("`$(destname)` exists on local system. Use `force=true` to override.")
                end
            end      
            dryrun ? write(destname,ftpname)  : download(ftp, ftpname, destname,mode=mode)
            verbosity > 1 && println(" done!")
        elseif recursive
            newdir = joinpath(ftp_basepath,item.name)
            download_dir(ftp,newdir;as=destname,ftp_basepath=newdir,dest_basepath=destname, verbosity=verbosity, recursive=recursive, force=force, mode=mode, dryrun=dryrun)
        end
    end
end

The usage:

julia> using FTPClient

julia> ftp = FTP(hostname = "palantir.boku.ac.at", username = "anonymous", password = "");

julia> download_dir(ftp,"/Public/ImprovedForestCharacteristics/",as="test",verbosity=2, force=true,recursive=true, dryrun=true)
Processing ftp directory `/Public/ImprovedForestCharacteristics/`:
Processing ftp directory `/Public/ImprovedForestCharacteristics/Age`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl.tif` as local file `test/Age/agecl.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_1_perc.tif` as local file `test/Age/agecl_1_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_2_perc.tif` as local file `test/Age/agecl_2_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_3_perc.tif` as local file `test/Age/agecl_3_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_4_perc.tif` as local file `test/Age/agecl_4_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_5_perc.tif` as local file `test/Age/agecl_5_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_6_perc.tif` as local file `test/Age/agecl_6_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_7_perc.tif` as local file `test/Age/agecl_7_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Age/agecl_8_perc.tif` as local file `test/Age/agecl_8_perc.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Alive_Tree_Carbon`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Alive_Tree_Carbon/tree_c.tif` as local file `test/Alive_Tree_Carbon/tree_c.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Alive_Tree_Carbon/tree_c_sd.tif` as local file `test/Alive_Tree_Carbon/tree_c_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Basal_Area`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Basal_Area/ba.tif` as local file `test/Basal_Area/ba.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Basal_Area/ba_sd.tif` as local file `test/Basal_Area/ba_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Biomass`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/bra_bm.tif` as local file `test/Biomass/bra_bm.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/bra_bm_sd.tif` as local file `test/Biomass/bra_bm_sd.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/fol_bm.tif` as local file `test/Biomass/fol_bm.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/fol_bm_sd.tif` as local file `test/Biomass/fol_bm_sd.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/root_bm.tif` as local file `test/Biomass/root_bm.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/root_bm_sd.tif` as local file `test/Biomass/root_bm_sd.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/stem_bm.tif` as local file `test/Biomass/stem_bm.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Biomass/stem_bm_sd.tif` as local file `test/Biomass/stem_bm_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Diameter`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Diameter/dbh.tif` as local file `test/Diameter/dbh.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Diameter/dbh_sd.tif` as local file `test/Diameter/dbh_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Height`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Height/heigth.tif` as local file `test/Height/heigth.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Height/heigth_sd.tif` as local file `test/Height/heigth_sd.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Readme.txt` as local file `test/Readme.txt`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Stand_Density_Index`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Stand_Density_Index/sdi.tif` as local file `test/Stand_Density_Index/sdi.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Stand_Density_Index/sdi_sd.tif` as local file `test/Stand_Density_Index/sdi_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Stem_Number`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Stem_Number/nha.tif` as local file `test/Stem_Number/nha.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Stem_Number/nha_sd.tif` as local file `test/Stem_Number/nha_sd.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Tree_Species_Group`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/Species_in_Group.txt` as local file `test/Tree_Species_Group/Species_in_Group.txt`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg.tif` as local file `test/Tree_Species_Group/tsg.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_1_perc.tif` as local file `test/Tree_Species_Group/tsg_1_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_2_perc.tif` as local file `test/Tree_Species_Group/tsg_2_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_3_perc.tif` as local file `test/Tree_Species_Group/tsg_3_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_4_perc.tif` as local file `test/Tree_Species_Group/tsg_4_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_5_perc.tif` as local file `test/Tree_Species_Group/tsg_5_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_6_perc.tif` as local file `test/Tree_Species_Group/tsg_6_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_7_perc.tif` as local file `test/Tree_Species_Group/tsg_7_perc.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Tree_Species_Group/tsg_8_perc.tif` as local file `test/Tree_Species_Group/tsg_8_perc.tif`... done!
Processing ftp directory `/Public/ImprovedForestCharacteristics/Volume`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Volume/vol.tif` as local file `test/Volume/vol.tif`... done!
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Volume/vol_sd.tif` as local file `test/Volume/vol_sd.tif`... done!

julia> download_dir(ftp,"Public/ImprovedForestCharacteristics", verbosity=2, force=true, recursive=false, dryrun=false)
Processing ftp directory `/Public/ImprovedForestCharacteristics`:
 - downloading ftp file `/Public/ImprovedForestCharacteristics/Readme.txt` as local file `ImprovedForestCharacteristics/Readme.txt`... done!

Note that the dryrun option does actually write to the local filesystem, but the file is a text file containing just the pathname on the FTP server and not the actual file.