On Windows system, repeatedly opening a file and writing to it will get “permission denied”.

[gqw]

libtorrent version (or branch): libtorrent-v2.0.3

platform/architecture: Windows 7 and Windows 10/ x86/x64

compiler and compiler version: Visual studio 2017 && Visual studio 2019

please describe what symptom you see, what you would expect to see instead and how to reproduce it.

1. Run simple_client xxx.torrent
2. Sometimes the download is incomplete

expect to see: download all files in torrent.

I found the reason is:

the function posix_storage::open_file() (posix_storage.cpp:540) will be called repeatly. It will be caused a problem on windows system:

I simplified the problem:

bool fopenerr(const wchar_t* path) {
	auto index = 0, max_errtimes = 20;
	while (++index && max_errtimes > 0)
	{
		FILE* f = nullptr;
		f = ::_wfopen(path, L"ab+");
		//if (f == nullptr) {
		//	for (int i = 0; i < 30; ++i)
		//	{
		//		f = ::_wfopen(path, L"ab+");
		//		std::cout << "reopen: " << index << " times: " << i << std::endl;
		//		if (f)
		//			break;
		//		std::this_thread::sleep_for(30ms);
		//	}
		//}

		if (f == nullptr) {
			std::cerr << index << ": open file failed, errno: " << errno << " " << strerror(errno) << std::endl;
			max_errtimes--;
			continue;
		}
		std::string s("hellooooooooooooooooooooooooooooooooooooooo");
		::fwrite(s.c_str(), 1, s.length(), f);
		fclose(f);
	}
	return max_errtimes == 20;
}

it will print like this:

83508: open file failed, errno: 13 Permission denied
83978: open file failed, errno: 13 Permission denied
83979: open file failed, errno: 13 Permission denied
83980: open file failed, errno: 13 Permission denied
83981: open file failed, errno: 13 Permission denied
84307: open file failed, errno: 13 Permission denied
84715: open file failed, errno: 13 Permission denied
84716: open file failed, errno: 13 Permission denied
84717: open file failed, errno: 13 Permission denied
84719: open file failed, errno: 13 Permission denied
85133: open file failed, errno: 13 Permission denied
85134: open file failed, errno: 13 Permission denied
85135: open file failed, errno: 13 Permission denied
85136: open file failed, errno: 13 Permission denied
85137: open file failed, errno: 13 Permission denied
85139: open file failed, errno: 13 Permission denied
85376: open file failed, errno: 13 Permission denied
85377: open file failed, errno: 13 Permission denied
85378: open file failed, errno: 13 Permission denied
85379: open file failed, errno: 13 Permission denied

When I uncomment the code in above, and retry a few times if the file fails to open, I can fix the problem. But I don't know exactly why.

I see a similar problem on StackOverFlow. https://stackoverflow.com/questions/61823407/repeated-data-table-fread-and-fwrite-causes-permission-denied-error/68310864#68310864

[wrtorrent]

can you try _wfopen_s()?

[arvidn]

I suspect closing the file is asynchronous, and if that hasn't completed (in the background) by the time the file is attempted to be opened again, it will fail because the file is already opened. I expect allowing sharing of the file would help.

[wrtorrent]

_wfopen_s() difference is another process can get 13 Permission denied when accessing file _wfsopen(, _SH_DENYNO); still produce errno: 13 Permission denied, file contents can be viewed. _wfsopen(, _SH_DENYRW); act similar to _wfopen_s() However, on my Windows10 I do get errors only when trying to view file from another process (open with external viewer). If code from first post running while all I do is refreshing folder (to see file size update) no Permission denied. MSVS 16.10.3 Win10 x64. Had tried to disable Windows AV Realt-time protection too.

Looks like this happens when another process hold file handle open ?in between fclose() _wfopen() sequence. Confirmation to this set path variable in example code to "filename.exe", then I get Permission denied every second folder refresh since Windows AV Realt-time protection checking file.

Running with elevated privileges in Admin command prompt still occasionally errors with Permission denied when viewing file from another program.

uncomment the code and retry fopen()

[gqw]

I repeated the test， _wfopen_s and _wfsopen have the same result. Disable Windows AV has the same result. All test only one process is running, no other viewer is open. Same reuslt.

I suggest you wait longer, sometimes it takes 20 minutes to show up, sometimes it’s fast.

Win10 x64

[wrtorrent]

takes 20 minutes

1 hour passed no errors. I have to actively reopen file (while it being written) with file viewer to get access errors from example code.

Another program that can access files, like AV in Windows is viruses... When AV disabled and no virus Windows Search indexing files for ya. Or if you have File Explorer open it can refresh icons.

Anyway, this is some kind of MS Windows problem since running as Administrator program shouldn't get any access violations.

add: Tested with SSD: was very difficult to reproduce, clicked like 5 minutes non stop to see error. Then I tried my non system drive, had to click for a minute or two to get error. Problem was File Indexing in Windows Driver Properties. Uncheck bottom option: https://filestore.community.support.microsoft.com/api/images/7d367be5-7584-448a-a7d2-84511d5c96de Disabling File Indexing not fixes problem completely, but it hard to reproduce now.

[arvidn]

does the file indexing service open files in exclusive mode? i.e. preventing other processes from opening the files

if not, the problem would still be that fopen() does open the files in exclusive mode. iirc, that's the default in the win32 API.

[wrtorrent]

fopen() does open the files in exclusive mode

I think my file viewer do exactly this. I do not get Permission denied errors unless continuously open file with 3rd party viewer.

[arvidn]

I imagine it would be simpler to, instead of using posix_storage on windows, implement an win32_storage. Also single threaded and simple.

[wrtorrent]

By win32_storage do you mean CreateFile()? CreateFile() error 32 (0x00000020) The process cannot access the file because it is being used by another process.

Only change to gqw patch should be moving sleep_for() few lines up.

if (nullptr == f) {
	for (auto i = 0; i < 31; ++i) {
		std::this_thread::sleep_for(std::chrono::milliseconds(1));
		f = ::_wfopen(filename, L"ab+");
		if (f)
			break;
	}
}

[arvidn]

By win32_storage do you mean CreateFile()?

Yes. With that API you can specify share mode as FILE_SHARE_READ and FILE_SHARE_WRITE

[wrtorrent]

The code that produced error was

while (-1) {
	hf = CreateFile(L"somefile.dat",
		GENERIC_READ | GENERIC_WRITE | FILE_APPEND_DATA, // FILE_APPEND_DATA,
		FILE_SHARE_READ | FILE_SHARE_WRITE, // 0
		NULL, // LPSECURITY_ATTRIBUTES
		OPEN_ALWAYS,
		FILE_ATTRIBUTE_NORMAL,
		NULL); // template
	if (INVALID_HANDLE_VALUE == hf) {
		ignore = GetLastError();
		std::cout << "CreateFile() err " << ignore << " ";
		LPVOID buff = 0;
		FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
		nullptr, ignore, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPWSTR)&buff, 0, nullptr);
		wprintf(L"(0x%08x) %s\n", ignore, (LPWSTR)buff);
		LocalFree(buff);
		CloseHandle(hf);
		return 0;
	} else {
		ignore = SetFilePointer(hf, 0, NULL, FILE_END);
		WriteFile(hf, s.data(), s.size(), &ignore, NULL);
		CloseHandle(hf);
	}
}

running in Administrator Command Prompt while continuously opening file from unprivileged process, AV running, file-indexing off. CreateFile() error 32 (0x00000020) The process cannot access the file because it is being used by another process.

[arvidn]

ok, so some other process is opening the file without FILE_SHARE_READ and FILE_SHARE_WRITE then. Do you agree with that conclusion? Or so you think that test program interferes with itself?

[wrtorrent]

No, test program not interfere: can start 3+ copy no problem.

other process is opening the file without FILE_SHARE_READ and FILE_SHARE_WRITE then.

I guess so, closed source program, too big to disassemble for curiosity (Norton Commander clone, Midnight Commander in linux). I posted because CreateFile() behavior no different to other file open functions in Windows (there no fix to gqw issue on libtorrent side).

[arvidn]

I posted because CreateFile() behavior no different to other file open functions in Windows (there no fix to gqw issue on libtorrent side).

I see, I was under the impression that perhaps fclose() is async on windows, and interfering with the next fopen() call. I wouldn't be surprised if fopen() usese the default share mode of 0.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.