Next post: Starting a process with non-ascii characters in Python 2

Avoiding race conditions when copying files

When writing scripts, I often want to perform a file copy where I do not want to overwrite any existing files. I want to copy from "src" to "dest", and if a file exists at "dest", it should throw an exception. It's straightforward to write something like (Python):

def notActuallySafeCopy(srcfile, destfile):
    if os.path.exists(destfile):
        raise IOError('destination already exists')
    
    shutil.copy(srcfile, destfile)
But there is a race condition here -- there is a brief window of time after the check in which if a file is created at "destfile", it will be replaced. (In the worst case, this type of window leads to a symlink race).

I wish Python solved this problem automatically, but the Python documentation explicitly says that shutil.copy and shutil.copy2 can silently overwrite the destination file if it already exists, and there is no flag to prevent this. Even worse, the behavior for overwriting existing files is different across platforms(!) So, I had to call into the operating system at a lower level. Here is my code for safer moves and copies, in both Linux and Windows:

def copyFilePosixWithoutOverwrite(srcfile, destfile):
    # O_EXCL prevents other files from writing to location.
    # raises OSError on failure.
    flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
    file_handle = os.open(destfile, flags)
    with os.fdopen(file_handle, 'wb') as fdest:
        with open(srcfile, 'rb') as fsrc:
            while True:
                buffer = fsrc.read(64 * 1024)
                if not buffer:
                    break
                fdest.write(buffer)

def copy(srcfile, destfile, overwrite):
    if not exists(srcfile):
        raise IOError('source path does not exist')
        
    if srcfile == destfile:
        pass
    elif sys.platform.startswith('win'):
        from ctypes import windll, c_wchar_p, c_int
        failIfExists = c_int(0) if overwrite else c_int(1)
        res = windll.kernel32.CopyFileW(c_wchar_p(srcfile), c_wchar_p(destfile), failIfExists)
        if not res:
            raise IOError('CopyFileW failed')
    else:
        if overwrite:
            shutil.copy(srcfile, destfile)
        else:
            copyFilePosixWithoutOverwrite(srcfile, destfile)

    assertTrue(exists(destfile))
        
def move(srcfile, destfile, overwrite):
    if not exists(srcfile):
        raise IOError('source path does not exist')
        
    if srcfile == destfile:
        pass
    elif sys.platform.startswith('win'):
        from ctypes import windll, c_wchar_p, c_int
        flags = 1 if overwrite else 0
        flags |= 2 # allow moving across drives
        res = windll.kernel32.MoveFileExW(c_wchar_p(srcfile), c_wchar_p(destfile), c_int(flags))
        if not res:
            raise IOError('MoveFileExW failed')
    else:
        copy(srcfile, destfile, overwrite)
        os.unlink(srcfile)
    
    assertTrue(exists(destfile))
Many tests and more file utilities like this can be found on my GitHub page here.