Download to windows - file get corrupted

[rwinhold]

Or at least I think the file is getting corrupted. Not often but say once every couple of thousand downloads.

What I'm seeing is that once I download a file with a command similar to:

Sync-SfItem -ShareFilePath "sfmoveFile:" -Download -OverWrite -Move -LocalPath "$localDownload$dirname"

the file appears in the windows directory as a zero byte file. Now through detailed logs I'm collecting, I know that the file is good in Sharefile but after the download it is 0 bytes.

A workaround I thought of is copy, verify, then delete rather than the sync with the -Move option. But, I have no idea how to delete a file from Sharefile.

Any thought on a real fix or how to delete a file?

Thanks.

[vkvikaskmr]

You can delete a file from ShareFile by using the following command : Send-SFRequest –Client $sfClient –Method DELETE 'url/of/file/to/be/deleted' Assuming that you have $sfClient initialised to your ShareFile client.

[digitalexpl0it]

same issue, copy or sync the file is created but the files are 0 bytes. The files that are created are empty files/

Sync-SfItem -ShareFilePath $ShareFileHomeFolder -Synchronize -Download -LocalPath $LocalPath -Recursive

[rwinhold]

Here is what I did in the end to get around the problem. This is a section of my powershell script that does the job. There is a directory in ShareFile (that's the "fo1383b4-5b30-4dfd-a5cc-b4b687503e89" direcotry id) that has a number of sub-directories. The script will copy the files in the sub directories to my local drive. Then check if the file has content. Then delete from ShareFile if it does. I've go the script scheduled to run every 20 minutes so if the file does fail to copy (and that still happens every once in a while) it will get picked up again on the next run. Also, thanks to vkvikaskmr for the delete tip.

$localrootdir = "C:\dowload"


Function DownSync-files
{

    $childDir = Send-SfRequest $sfClient -Method GET -Entity Items -Id "fo1383b4-5b30-4dfd-a5cc-b4b687503e89" -Expand "Children,Owner"

    foreach($dir in $childDir.Children)
    {
        $dirname = $dir.filename
        $file = Send-SfRequest $sfClient -Method GET -Entity Items -Id $dir.id -Expand "Children,Owner"
        foreach($fileToMove in $file.children)
        {
            $fname = $fileToMove.filename
            if($fileToMove.filesizeinkb -gt 0)
            {
                $UnprocessedFile = (Send-SfRequest $sfClient -Entity Items -id $fileToMove.id).url

                New-PSDrive -Name sfmoveFile -PSProvider Sharefile -Client $sfClient -Root "/" -RootUri $UnprocessedFile
                Sync-SfItem -ShareFilePath "sfmoveFile:" -Download -OverWrite -LocalPath "$localrootdir\$dirname"
                Remove-PSDrive sfmoveFile
                $filesize = (Get-Item $localrootdir\$dirname\$fname).length
                if($filesize -gt 0)
                {
                    Send-SfRequest $sfClient -Method DELETE -Entity Items -Id $fileToMove.id
                }
                else
                {
                    Remove-Item $localrootdir\$dirname\$fname
                }
            }
        }
    }
}

[digitalexpl0it]

So you are pulling by ID vs the file name, interesting

[rwinhold]

Yes. In my case, some other process is dropping files into sharefile. I don't really care about the filenames. I just want to collect them all for processing.

[digitalexpl0it]

I am trying to make a copy of all the files in share file so I can back them up. Testing from my account, but would change over to the a higher account that see's all the files/folders and pull them

[rwinhold]

Okay. So you aren't concerned with deleting them then? You just want to make sure you have a file with the same content? (as in not a zero byte file).

[digitalexpl0it]

correct, have had some issues of endusers deleting files then 6 months to a year later they need the file.

[digitalexpl0it]

I tried your code, but it downloads the files with 0 bytes still. Thre is only two files in sharefile and they both have a few hundred Kb to the file

`$authFile = ((Join-Path $env:USERPROFILE "Documents") + "\leavittgroup.sfps")

if (-NOT (Test-Path $authFile )) { New-SfClient -Name $authFile }

$sfClient = Get-SfClient -Name $authFile

$localrootdir = "C:\Sharefile"

$ShareFileHomeFolder = "syncdrive:/" + (Send-SfRequest $sfClient -Entity Items).Name

$childDir = Send-SfRequest $sfClient -Method GET -Entity Items -Id "fohddddfc-9f6a-45a2-9cae-ce32aab5617d" -Expand "Children,Owner"

foreach($dir in $childDir.Children) { $dirname = $dir.filename $file = Send-SfRequest $sfClient -Method GET -Entity Items -Id $dir.id -Expand "Children,Owner" write-host "Found:" $file.filename

foreach($fileToMove in $file){
	$fname = $fileToMove.filename
	$UnprocessedFile = (Send-SfRequest $sfClient -Entity Items -id $fileToMove.id).url
	write-host "URL:" $UnprocessedFile

	write-host "Creating New PSDrive...."
	New-PSDrive -Name syncDrive -PSProvider ShareFile -Client $sfClient -Root "/" -RootUri $UnprocessedFile

	Sync-SfItem -ShareFilePath "syncDrive:/" -Synchronize -Download -LocalPath "$localrootdir" -Recursive
	
	
	Remove-PSDrive syncDrive
}

}

`

[rwinhold]

I also still had the same problem with my code. The difference was that I was able to tell when a file didn't download. My issue was I was doing the mass download and delete. So it would download some and only create a 0 byte file for others and then delete the source. So it was lost forever.

I wrote the code so that I could control whether or not to delete the source. So it still fails occasionally but when that happens I choose not to delete.

There are times when it takes a few runs of the script to download successfully.

Not sure what to suggest without being there. Have you tried the script using the account with elevated privilege? Do you have access to the ShareFile logs?

One other thing we noticed here is that we have two different places where the Sharefile directories can be. One is within our local network and another was in some cloud. The local one was always very sketchy as far as uploads and downloads (the rest of the scripts upload the files to ShareFile once they're processed and there are times when there is no indication that an upload was even attempted let alone successful when copying to the local sharefile).

So, maybe there are other forces at work? It might be good to work with the ShareFile admin to see if there is anything going on behind the scenes.

[digitalexpl0it]

Thank you for your help and insight. We use sharefile on-prem. Its been a nightmare trying to get backups of this data not encrypted and meet our RPO. I will look more into it and see if this will work for us. Again thanks for the help. Also I am using the 64 bit snapin, wonder if I need to use the 32 bit? Something I can test out.

[digitalexpl0it]

UPDATE: The 32 bit version is working fine. Something with the 64 bit snapin isn't downloading the files correctly.

[rwinhold]

That's some good insight there. I just checked and we're using the 32 bit version.

[poshbrs]

I'm fighting with 0KB files as well. Using 32bit or 64bit does not make any difference. Noticed that data got transfered when I used copy-sfitem, but that lacks all the logic that sync-sfitem offers around syncing, moving, keeping folder and so on. So would very much like to have sync-sfitem working. StorageZone is on-premise.

[mgalpin01]

Hi guys, please can someone provide an update / help with this issue?

We are using Sync-SfItem in production code and it has suddenly stopped working last week and now downloading 0KB files. It runs perfectly if we step through the powershell script in the Powershell IDE but not when calling in the automated way from the command line.

This is the script we're running (a cut down version of it)

    StartLogging "Getfiles_Sharefile"

    Add-PSSnapIn ShareFile

    $sfClient = Get-SfClient -Name $ShareFile_Subdomain

    #ShareFile directory is relative to the root of the account
    #get the current user's home folder to use as the starting point
    $ShareFileHomeFolder = "sf:/" + (Send-SfRequest $sfClient -Entity Items).Name

    # if Drive already mapped, remove

    if (Get-PSDrive sf -ErrorAction SilentlyContinue)
          {Remove-PSDrive -Name sf}

    #Create a PowerShell provider for ShareFile at the location specified
    New-PSDrive -Name sf -PSProvider ShareFile -Root / -Client $sfClient

    Set-Location $ShareFileHomeFolder/Managers


    Send-SfRequest -Client $sfClient -Entity Items -Navigation Children

   # Get details of each folder and copy one folder a time to inbox. 
   # Sync_Sfitem does not seem to work when there are sub-folders so need to do each  sub-folder one at a time
    $DataFileFolders = gci

     foreach ($DataFileFolder in $DataFileFolders) 
    {
        if (($DataFileFolder.psiscontainer -eq "True") -and ($DataFileFolder.Name -eq "Test"))
        {
         $FolderVariable = $DataFileFolder.filename
         WriteLogInfo "Downloading files from Sharefile folder $FolderVariable and saving to LocalPath: $ShareFile_ProdSubscriberRoot\Inbox\$foldervariable"
         Sync-SfItem -ShareFilePath "$ShareFileHomeFolder/Managers/$foldervariable" -Download -LocalPath $ShareFile_ProdSubscriberRoot\Inbox -Move -Overwrite -verbose
         #copy-SfItem -ShareFilePath "$ShareFileHomeFolder/Managers/$foldervariable" -Download -LocalPath $ShareFile_ProdSubscriberRoot\Inbox -verbose #-CreateRoot
          #Copy-SfItem -Path "$ShareFileHomeFolder/Managers/$foldervariable" -Destination $ShareFile_ProdSubscriberRoot\Inbox -Force 1
 
        }
     }
     #
  

    FinishLogging

And how we're calling it C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "D:\Batch\Scripts\Sharefile\GetFiles.ps1"

Have tried everything recommended in online forums and in this thread - eg

• reinstalling powershell snap in (32 and 64 bit versions)

• running with powershell 64 bit

• Using Copy-SfItem instead of Sync-SfItem

• Adding a check at end of download that file isn't locked (and a wait process)

Any help appreciated as our production batch process is basically broke and it doesn't appear there's any other alternative to using the powershell snap in (eg SFCLI deprecated)

thanks

[riegerd]

Hello guys,

after debugging this error, I figured out a solution for myself. Just add the following two lines at the start of your script or at least before creating the sf client:

[System.Net.SecurityProtocolType] $SecurityProtocolType = [System.Net.SecurityProtocolType]::Tls12
[System.Net.ServicePointManager]::SecurityProtocol = $SecurityProtocolType

This will enable TLS1.2 which in my case was required by the sf server.