notepad4 icon indicating copy to clipboard operation
notepad4 copied to clipboard

Support file larger than 4GiB in 64-bit Notepad2

Open zufuliu opened this issue 6 years ago • 3 comments

Loading large file is very smooth since 1815a8ed2b5ac6b8f6b336485ad60afa961a052c, large file TODOs taken from comment in EditLoadFile() 795fb952fcad443bb0b67e0990bc1810bbbb01aa:

  • [x] [> 2 GiB] use SC_DOCUMENTOPTION_TEXT_LARGE somewhere or hard-coded in EditModel::EditModel().
  • [ ] [> 4 GiB] use SetFilePointerEx() and ReadFile()/WriteFile() to read/write file.
  • [x] [> 2 GiB] fix encoding conversion with MultiByteToWideChar() and WideCharToMultiByte().
  • [x] [> 4 GiB] fix sprintf(), wsprintf() and wcstol() for Sci_Position and Sci_Line, currently UINT is used.
  • [x] [> 2 GiB] ensure Sci_PositionCR is same as Sci_Position, see Sci_Position.h

Supporting for UTF-8 encoded file smaller than 4 GiB can simply implemented by replacing SC_DOCUMENTOPTION_DEFAULT with SC_DOCUMENTOPTION_TEXT_LARGE in EditModel::EditModel():

- pdoc = new Document(SC_DOCUMENTOPTION_DEFAULT);
+ pdoc = new Document(SC_DOCUMENTOPTION_TEXT_LARGE);

All other codes (except encoding conversion) already works for 4GiB file.

zufuliu avatar Jul 26 '19 13:07 zufuliu

Since commit 538a8ca39459254e120da6303724bfb969a032fd, Notepad2 can loading file larger than 2 GiB (less than 4 GiB or 1/3 physical memory). Encoding conversion for file larger than 2 GiB is disabled, it will not be fixed: file larger than 2 GiB for encodings other than UTF-8 is rare.

zufuliu avatar Jul 30 '19 11:07 zufuliu

From https://bugs.python.org/issue36311, it's suggested to use 512 MiB buffer to get better performance when using MultiByteToWideChar() for large input.

zufuliu avatar Aug 22 '19 14:08 zufuliu

Larger than 4GiB file can be loaded easily with file mapping (mapping entire file is faster than reading entire file).

https://docs.microsoft.com/en-us/windows/win32/memory/creating-a-view-within-a-file https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile

zufuliu avatar Nov 21 '19 10:11 zufuliu