PowerShell: Check if File was Fully Copied

Estimating How Much of a File was Copied

@August 9, 2023

We recently encountered an issue where files were copied but it wasn’t certain that the transfer was uninterrupted.

The Windows box copies large files (>50Gb) off-site over a relatively slow link. We thus needed a quick estimate from the copying machine if any files were obviously not copied completely.

A Quick Warning on Comparing Files by Size

Kindly allow us to introduce a warning here for anyone that assumes that this is the proper way to compare even files that are assumed to be the same. Every single bit could have been toggled in the copying process resulting in a file of the same size that is different - whether this was done intentionally or not does not matter if you need an exact replica.

⚠️

Comparing file sizes is not a proper way of comparing files. For court-room-proof comparisons use cryptographically-secure checksum algorithms like SHA256 or stronger.

In this case, we do also copy SHA256 checksums for each file in a separate .sha256 sidecar file, but validating those from the source of the copying machine requires copying the whole data back for verification purposes. Validation is thus a task better suited to the destination platform. Also, our 7z file type this case integrates CRC32 checksums for each individual file contained in the archive. Validation can then be performed on a single file even without copying the whole archive but alas, CRC32 is not cryptographically secure and could be fooled by a malicious actor - also validating the archive file-by-file is of course much slower than just copying the whole archive back for comparison.

Establishing the Copied Amount

Now, since Windows “lies” about the true file size and only reports the allocated file size, we can’t just compare those sizes. However, when right-clicking on a file and Explorer will tell us both the Size and the “Size on disk” which is the part that was actually copied.

Comparing from Explorer was not a practical option due to the number of files, thus PowerShell comes to the rescue.

A file where transfer was interrupted after 48.7 out of 55.8GB

We also noted that the date was left in 1980 - not quite sure why this date, since the UNIX epoch starts in 1970 (the target system is running Linux) and, if I’m not mistaken, the Windows epoch starts in 1601. Thus, checking if the date was too far back may have been another option, but did not feel as confident and also does not say how much of the file was left uncopied.

The Scripted Solution

Since PowerShell’s Get-Item only supports the Length parameter (”Size” in Explorer) but no “LenghOnDisk", we were looking for a way to access that property. Fortunately someone was kind enough to post the PowerShell code needed to access that information (link at bottom.)

We thus now boldly compare if the “Size on disk” is greater than the file size (which must be the case due to e.g. the cluster size)

⚠️

The function was not tested on compressed disks/folders. In this case we suspect that the “Size on disk” might be smaller than the file size for files that compress well.

Using the function Get-IsFileFullyCopied -FilePath "R:\Foo\file.7z" on the remote shared drive, we can now quickly gauge how much was actually copied. The function returns a boolean but also prints out for which files the sizes differ. It is thus suitable for a loop to check the whole directory including sub-directories:

# Define the directory to search
$directoryPath = "R:\Foo"

$files = Get-ChildItem -Path $directoryPath -File -Recurse

foreach ($file in $files) {
    $isFullyCopied = Get-IsFileFullyCopied -filePath $file.FullName
    if (-not $isFullyCopied) {
        # Handle error on $file
    }
}

Thanks to StackOverflow User CB., the original code contributor to [Win32Functions.ExtendedFileInfo]::GetFileSizeOnDisk

https://stackoverflow.com/questions/22507523/getting-size-on-disk-for-small-files-in-powershell/22508299#22508299