NES MMC3 Config Breaks Pointers and Stack (?)

[seanwiththebeard]

TL/DR: I almost figured out why the bankswitching demo doesn't work on other emulators or real hardware even though things look fine in the IDE

Something between the nesbanked.cfg linker config and the custom neslib implementation is messing up any use of pointers when working in a mapper-4 MMC3 configuration. I've set up three repos to show the problem and a partial solution. I suggest using Mesen to reproduce the issues

The Problem Illustrated - If you try the bankswitching demo from the NES examples, it works fine in the IDE's JSNES emulator but you get a completely blank screen in Mesen or on an Everdrive. FCEUX seems to work fine, but since Mesen and my real hardware exhibit the same behavior in all the relevant cases I'm going to refer to those as "outside the IDE" for the purposes of this issue.

I'm not sure why these behave differently and I was hitting my head against the wall for a few days, but today I looked at some debugging viewers in Mesen and noticed that code is definitely running. Then I looked at the nametable data and noticed there was a line of character 0x7F where it should have said "Bank 0 @ 8000." So code is definitely running, PPU writes are happening, and the character set is being copied, but the string pointer isn't being passed right. The next three lines don't draw at all because they're passed as parameters to the draw_text function, which I assume should be put on the stack. Are we getting a stack? I looked at the nesbanked.cfg config (which I assume is used when you #define NES_MAPPER 4) and there isn't a stack location defined. Same issues with the palette set function, so the screen remains black. I've also wondered if it's a header issue, since the 8kb PRG-RAM doesn't seem to be enabled in the iNES header. Another shot in the dark is the condes table which ends up in the 0x6000 block where that RAM would be.

The problem circumvented (sort of): https://github.com/seanwiththebeard/NES_MMC3_Example

So after a lot of experimentation, I used a different method for setting the palette and setting text strings and made a little demo. This is where I found that strlen doesn't work either. This demo shows two ways of drawing strings, one direct way works in both the IDE and on real hardware, but passing a parameter to a function only works in the IDE. Also notice the local variable x used in the while loop at the end used to set the background color, outside the IDE it only increments before the loop. I tried compiling locally with CC65 (your neslib version's source copied to the source folder) using and surprisingly got identical output. Here is where I'm starting to think it's a linker configuration issue, and pointers/stack things don't get handled to the right places.

The (almost) solution: https://github.com/seanwiththebeard/ConfigTest_MMC3_NESDougConfig

This example from NESDoug is a sample project that compiles locally, one click compile and you get an MMC3 bankswitched demo. I took this template and shoehorned my demo into it, adapting for some different memory segment layout from it's included linker config. Using the included neslib it compiled fine and worked on everything. Once I tried your custom neslib, something with the startup code wasn't working. But, since the template worked for sure, I tried uploading the whole project into 8bitworkshop to see if I can override the custom neslib. It's a hack because you need to link crt0.s and then comment it out to get it to run, but then you can save the ROM and it works on everything. Inside 8bitworkshop there's a weird graphical glitch that doesn't happen in the exported ROM. It crashes after a few seconds and I'm not sure why, but the exported ROM is stable as expected, music even works from whatever bank it's configured to. What if anything is different about the linker config (aside from the memory segments) isn't apparent to me but I haven't been able to find where the condes table goes to in this one... At any rate, things like pointers and global variables work fine.

So here's what I think might be ideal: https://github.com/seanwiththebeard/33_MMC3_8bitworkshop This is NESDoug's demo shoehorned back into the last project, exact same behavior. Same issue where you have to uncomment the crt0.s and recomment it. So if this proves that it's possible to get a fully compatible MMC3 ROM exported with the right neslib, it would be nice to have an option in 8bitworkshop to use the regular neslib instead of the custom one. Or, ideally, it would be great if the custom neslib (or the startup code?) was fixed for this use scenario.

Sorry if this is a longwinded for a small problem mostly related to real hardware and certain emulators, but I had an interesting time getting here. Technically now I have a template for a larger MMC3 project, but hopefully a more elegant fix can be implemented.