PowerShell: Check if a File Was Fully Copied

I recently ran into files that had been copied, but with no certainty that the transfer had finished uninterrupted.

A Windows box copies large files (over 50 GB) off-site over a relatively slow link. I needed a quick estimate, from the copying machine, of whether any files were obviously not copied completely.

A quick warning on comparing files by size

Allow me a warning for anyone who assumes this is the proper way to compare files that are supposed to be identical. Every single bit could have been flipped during the copy, resulting in a file of the same size that is different. Whether that happened intentionally or not doesn’t matter if you need an exact replica.

Comparing file sizes is not a proper way of comparing files. For courtroom-proof comparisons, use a cryptographically secure checksum algorithm like SHA-256 or stronger.

In this case, I also copy a SHA-256 checksum for each file in a separate .sha256 sidecar file. But validating those from the copying machine means copying all the data back, so validation is a task better suited to the destination. The 7z archives in this case also carry a CRC32 checksum for each file they contain, so a single file can be validated without copying the whole archive back. Alas, CRC32 is not cryptographically secure and can be fooled by a malicious actor, and validating an archive file by file is of course much slower than copying the whole archive back for comparison.

Establishing the copied amount

Windows “lies” about the true file size and reports the allocated size, so I can’t just compare sizes in a listing. However, right-clicking a file shows both the Size and the “Size on disk” in Explorer, and the latter is the part that was actually copied.

Comparing in Explorer wasn’t practical given the number of files, so PowerShell comes to the rescue.

A file whose transfer was interrupted after 48.7 of 55.8 GB.

I also noticed that the file’s date was left at 1980. I’m not quite sure why that date: the UNIX epoch starts in 1970 (the target system runs Linux) and, if I’m not mistaken, the Windows epoch starts in 1601. Checking for a date too far in the past might have been another option, but it didn’t feel as reliable, and it doesn’t say how much of the file is missing.

The scripted solution

PowerShell’s Get-Item only exposes Length (“Size” in Explorer) and no LengthOnDisk, so I needed another way to read that property. Fortunately, someone on Stack Overflow had posted the code needed to get at it (credit at the bottom).

The function now boldly checks that the “Size on disk” is at least the file size, which must be the case for a complete file (for example because of the cluster size).

Not tested on compressed disks or folders. There, I suspect the “Size on disk” might be smaller than the file size for files that compress well.

# Needed for Get-IsFileFullyCopied()
Add-Type -TypeDefinition @'
using System;
using System.Runtime.InteropServices;
using System.ComponentModel;
using System.IO;

namespace Win32Functions
{
  public class ExtendedFileInfo
  {
    public static long GetFileSizeOnDisk(string file)
    {
        FileInfo info = new FileInfo(file);
        uint dummy, sectorsPerCluster, bytesPerSector;
        int result = GetDiskFreeSpaceW(info.Directory.Root.FullName, out sectorsPerCluster, out bytesPerSector, out dummy, out dummy);
        if (result == 0) throw new Win32Exception();
        uint clusterSize = sectorsPerCluster * bytesPerSector;
        uint hosize;
        uint losize = GetCompressedFileSizeW(file, out hosize);
        if (losize == 0xFFFFFFFF && Marshal.GetLastWin32Error() != 0) throw new Win32Exception();
        long size;
        size = (long)hosize << 32 | losize;
        return ((size + clusterSize - 1) / clusterSize) * clusterSize;
    }

    [DllImport("kernel32.dll", SetLastError = true)]
    static extern uint GetCompressedFileSizeW([In, MarshalAs(UnmanagedType.LPWStr)] string lpFileName,
       [Out, MarshalAs(UnmanagedType.U4)] out uint lpFileSizeHigh);

    [DllImport("kernel32.dll", SetLastError = true, PreserveSig = true)]
    static extern int GetDiskFreeSpaceW([In, MarshalAs(UnmanagedType.LPWStr)] string lpRootPathName,
       out uint lpSectorsPerCluster, out uint lpBytesPerSector, out uint lpNumberOfFreeClusters,
       out uint lpTotalNumberOfClusters);
  }
}
'@

function Get-IsFileFullyCopied {
    param(
        [string]$FilePath
    )
    $fullSize = (Get-Item -Path $FilePath).Length
    $diskSize = [Win32Functions.ExtendedFileInfo]::GetFileSizeOnDisk($FilePath)
    if (($fullSize - $diskSize) -gt 0) {
        # On-disk size is slightly larger than file-size (e.g. due to cluster size.)
        # The result should thus be <= 0 for files/folders that aren't compressed additionally by the filesystem.
        Write-Host "Not fully copied: ${FilePath}: $([Math]::Round(($fullSize - $diskSize) / 1MB, 3)) MB missing"
        return $false
    }
    return $true
}

Running Get-IsFileFullyCopied -FilePath "R:\Foo\file.7z" against the remote shared drive now gives a quick gauge of how much was actually copied. The function returns a boolean and also prints the files whose sizes differ, so it works in a loop over a whole directory, including subdirectories:

# Define the directory to search
$directoryPath = "R:\Foo"

$files = Get-ChildItem -Path $directoryPath -File -Recurse

foreach ($file in $files) {
    $isFullyCopied = Get-IsFileFullyCopied -FilePath $file.FullName
    if (-not $isFullyCopied) {
        # Handle error on $file
    }
}

Credit

Thanks to Stack Overflow user CB., the original contributor of the code behind [Win32Functions.ExtendedFileInfo]::GetFileSizeOnDisk, in this Stack Overflow answer.