You also need to fix devres.pkg to make it not crash

[anzz1]

Buffer overflow

The real reason why Visual Studio 6 crashes on new hardware has actually nothing to do with your OS, it works just fine in later Windows versions too but there are multiple buffer overflows in the resource editor module (DEVRES.PKG) which have seemingly gone unnoticed all these years.

The resource editor module has multiple calls to the C++ new[] operator using values that are too small without doing bound checking. But because of the way memory allocation and alignment works, it just always got "lucky" and did not crash, as the pointer memory block had more "extra" space than requested and then it wasn't overwritten or accessed by anything else before the operation was completed.

What has changed since then, is every part of your computer has become a lot faster; processor, memory, I/O, the works. There were no 16 core 32 thread processors in the early 2000s, nor 5000MB/s SSDs or fast memory. So now when running on a modern processor, what happens is that the memory which resource editor didn't properly allocate, gets overwritten by another thread while it's still being used (yes, VS6 is multithreaded, even if very lightly so).

That's also the reason why all sorts of different bags of tricks work to "fix" it, what they have in common is that they change the execution path (by slowing processing down and/or changing how memory is allocated/aligned) slightly so by luck you pass the buffer overflow without crashing. The compatibility layer slows it down, as does running it in a VM. Running it from a HDD instead of SSD can make the difference, changing folders, restarting the computer, or even something like running a 4K video stream in the background.

I know it's 2022 but I don't think Visual Studio 6 is any less useful today than it ever was. The resource editor is still the best comparing to all the new editions, and even if I code something in VS2019 I still use the editor in VC6 to design the dialogs. Also if you want to just code a quick tool which will work in any Windows version without any redistributables, VC6 is still the pick.

In any case, there are two ways to fix the crash for good.

The Fix

1. Delete the "DEVRES.PKG" file, but you will lose the resource editor.

or

2. Patch the buffer overflow in "DEVRES.PKG" with a hex editor.

// Microsoft Visual Studio 6.0
// Crash Fix (Buffer Overflow)
//
// Module: Resource Editor
// DEVRES.PKG v6.0.8168.0 - 17.6.1998 0:00
//
// Original SHA1 : 59afd55f13310dcdbfff777fe6f4c7d0a8191a82
// Fixed SHA1    : 00bb8497adca2467eaba022a34bf4fdafd3d7c6c
//

--------


0x00004518 / 0x50403518:

FF 25 74 1A 40 50    ; jmp     ds:__imp_??2@YAPAXI@Z ; operator new(uint)
->
E9 8F 0F 10 00 90    ; jmp     0001054AC ; nop

--------


0x001054AC / 0x505044AC:

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
->
55 89 E5 8B 45 08 B9 02 00 00 00 F7 E1 50 FF 15 74 1A 40 50 59 89 EC 5D C3

55                      push   ebp
89 E5                   mov    ebp,esp
8B 45 08                mov    eax,DWORD PTR [ebp+0x8]
B9 02 00 00 00          mov    ecx,0x2
F7 E1                   mul    ecx
50                      push   eax
FF 15 74 1A 40 50       call   DWORD PTR ds:0x50401a74
59                      pop    ecx
89 EC                   mov    esp,ebp
5D                      pop    ebp
C3                      ret

--------


0x000001E8:

AC 34 10 00
->
DC 34 10 00          ; increase .text section virtual size by 30 bytes


--------


0x00000140:

29 8D 19 00
->
77 FD 17 00          ; fix PE checksum


--------

[martinmoene]

Thank you @anzz1 !

I've added section Fix resource editor to How to install Visual Studio 6 on Windows 8.1 which refers the reader to this guide.

[samsonsite1]

Please upload or provide a link to the patched DEVRES.PKG file, for those who don't have the ability to patch it themselves. Thanks.

[anzz1]

devres.zip

Here ya go.

[samsonsite1]

Thanks!

I did a file comparison, and my DEVRES.PKG has some slightly different bytes, however it is the same file version (6.0.8168.0), and file size (1,544,244 bytes).

I didn't fully understand the instructions, but it appears he provided two offsets, a compiled DLL file offset, and a disassembler offset. I don't have a disassembler, but I can use a hex editor, so first offset is best.

I was able to find and replace all byte patterns, except the PE checksum. Mine was (8B53 1800), so it did not match the instructions, so I didn't change it. But it worked anyway and MSDEV.EXE didn't crash anymore. Thanks again!