Python Windows cannot store files with invalid characters

Question

I am trying to make a quick Python script to rename a bunch of files. These files were made on a Linux system on this NTFS drive, but now I'm on Windows. The naming convention is as follows:

Screenshot at 2016-12-11 21:12:56.png 

Symbol : is illegal in Windows file names, so the behavior of this script is a bit strange for me.

 for i in os.listdir("."): print(i) x = i.replace(":", "-") comm = """mv "{}" "{}" """.format(i, x) os.system(comm) 

In the above code, print(i) successfully prints the file names. However, when I try to run os.system(comm) to rename my files, I get this error:

 mv: cannot stat 'Screenshot at 2016-12-24 14:54:57.png': No such file or directory 

Firstly, it seemed a little strange to me that Python under Windows can say that these naughty files exist, but cannot move them. Secondly, what is the best way to get around this?

I also tried shutil.move() and os.rename() with no luck. This question seems to be discussing this issue, but seems to care more about prevention rather than eliminating it. I obviously could go back to Linux and fix it, but I wonder if I can fix it on Windows.

Answer 1

You can find them because they are in the directory. You cannot access them because the colon character is parsed differently along the way. This means that files cannot be reached using common path functions, including MoveFile . You basically have two options: find a method that does not rely on a name, such as OpenFileById , or find an alternative name for a file, such as dir /x . The latter gives you the short name (8.3), which should not contain a colon. I do not know if there is a ready-made function for accessing these names from Python, so the shortest explicit (for me) workaround is to perform dir /x and parse its output.

I think the paths for directory descriptors are as close as the standard Python library comes to the first method, but I don't know if that will be enough. The main functions of FindFirstFile / FindNextFile produce both names in WIN32_FIND_DATA (cFileName and cAlternateFileName), but Python expects the first to be valid. Any method would also make sense in PowerShell, but it looks like it doesn't know short names completely, and also tracks files by name, not identifiers. Otherwise, FileInfo.MoveTo would do the trick neatly.

To prevent this situation in the first place, ntfs-3g supports the windows_names parameter. This causes a crash when trying to create files.

Conclusion: as discussed at https://superuser.com/questions/31587/how-to-force-windows-to-rename-a-file-with-a-special-character , there is no clear solution. All my attempts (and several others) were discussed here. Probably the dirtiest option is to reconnect the drive to Linux and rename it; the file system is technically damaged because the characters are invalid, but the Microsoft repair solution is to delete, not rename.

Cygwin simply emulates a colon using a private unicode character (':' + 0xf000).