gbs-control

gbs-control copied to clipboard

• platformio compatibility (latest arduino-framework, webSockets)
• AsyncWebServer is replaced with standard solution for the sake of memory saving
• WiFi module has been rewritten (excluded PersWiFiManager)
• overal optimization, fixed backup/restore

"pio" branch still contain the beta version, a few features may not work as expected.

Hey, I'm confused here. Didn't we just have a PIO thing? https://github.com/ramapcsx2/gbs-control/pull/507

Hi @ramapcsx2 . I my case it didn't worked, so I've fixed things a bit and I'd change them even more to order up, clean and prepare it to CI. If you, guys don't mind. However if this is not an objective of this project it's ok, just let me know. :) I'd also add a version to the firmware. Can you give me a hint where we are now in terms of versions? Thanks

@Metaln00b Heya, could you share your thoughts?

I would also dare to:

• move all the web things onto the flash (this is a headache to reflash the complete firmware every time you changing something in UI).
• split the fat main.cpp into various modules
• change the project file structure as per PIOs recommendations, and
• implement one click build for the whole project or even better the CI/CD

@Metaln00b Heya, could you share your thoughts?

Well, it has nothing to do with PlatformIO, at least not directly. But I see a lot of changes that might break the environment for the main developers. That's not to say that the changes are bad :P

If I see it correctly, debug messages were moved to the websocket, spiffs were changed to LittleFS, a few endpoints regarding AccessPoint were adjusted etc.

I would also dare to:

• move all the web things onto the flash (this is a headache to reflash the complete firmware every time you changing something in UI).
• split the fat main.cpp into various modules
• change the project file structure as per PIOs recommendations, and
• implement one click build for the whole project or even better the CI/CD

Would it be okay for you to describe the pull request in more detail, what it changes?

Platformio is not the best headline right now 😅

But first of all, thank you for your support with this project🙏

Most developers are currently busy with other projects and can't look at the whole changes of code, so it would be very important to have a description of the changes, especially changes to folder structures etc.

Right. I am sorry for the vague description. In the beginning this branch was named "littlefs" since my initial objective was just to a bit change the source code in terms of 100% compatibility with PIO (such as silent build with no warnings, low memory-low flash :) , using latest framework maybe, just in one click) and change spiffs to littlefs.

Since my first attempt of the rapid dive into the large gbs-control.ino was not so rapid because of the high project entropy, I've decided to groom it a bit and that is how it all started.
The source code is actually rising the learning curve too much to allow people participating by doing small and rapid changes. Touching one thing pulls needless tests of other features.

No-offensive criticism here btw. :)

Now about the changes were made:

1. gbs-control.ino was converted into main.cpp and three additional modules were spawned:
   • wserver.cpp - with all the webserver's stuffs
   • wserial.cpp - supposedly all the serial and WS related functionality
   • wifiman.cpp - replacement for an overloaded PersWiFiManager.cpp
2. SPIFFS -> LittleFS (since soon it becomes deprecated)
3. all (or at least the most) of the static data moved into flash (PGM)
4. some thing were reorganized in terms to be able to build with the latest arduino-framework
5. AsyncWebServer -> ESP8266WebServer (synchronous) it allowed to spare 3-5% of RAM.
6. platformio.ini been changed a bit.

I believe this is it... The only warning which is left for now with OLEDMenuItem, since sizeof nonexistent object is not allowed.

The only environmental problem I see here is that you cannot use Arduino IDE for building and uploading anymore, which is I believe even better since you're moving onto PIO which encourages you to use VScode instead.

That all sounds reasonable, and I'm impressed someone went and started a cleanup here :p May I suggest that you treat this more like a fork, all depending on whether you want to continue working on it or whether you consider it done?

I'm open to merging, but if this is still in the make and break stages, then I'd prefer waiting a bit, so the way to build this doesn't change every 2 weeks, seemingly :)

• It's really good to split up the big .ino file. Please just try to not segment it all too much :)
• SPIFFS -> LittleFS : sounds good. There is no preference, just whatever works is fine
• Remember that static data in flash will incur some latencies. Sometimes it's good to have frequently used data in RAM
• PIO / Arduino: Some people are used to their toolchains, and it would be great to keep compatibility with the main ones. If a pure Arduino and a pure PIO setup can both build the project, I'm good :)

That all sounds reasonable, and I'm impressed someone went and started a cleanup here :p May I suggest that you treat this more like a fork, all depending on whether you want to continue working on it or whether you consider it done?

That is not done yet definitely. I'd like to make some changes to ease the burden :) for all those who're going through the debris when they join the project. I agreed to leave it as a fork in my personal repo, so your repo would stay unmodified. However that would bring more ambiguity and confusion over there.

I'm open to merging, but if this is still in the make and break stages, then I'd prefer waiting a bit, so the way to build this doesn't change every 2 weeks, seemingly :)

My plan is to finish implementing a few more changes in this branch so it can be merged into the master and spawn a new one to fix the project structure there.

Let me know what do you think on that.

PS: taking into account your next comment on the arch and especially the two IDEs-type building possibility. I believe there is no problem with it especially because Arduino IDE is less demanding than PIO.

I also expect collaboration on that from other members of the project. If someone do not agree with something, I'd like to hear that before the big changes, and maybe return to the "yet another fork" version. ;)

I'm a bit rusty in my understanding of the project, but:

• There is currently no versioning system that I'm aware of.
• Porting away from AsyncWebServer is a good idea (as it causes memory corruption and stack overflow from handling Wi-Fi operations in a deeply nested yield()), if you've made sure that handling one request at a time doesn't break any operations in the web UI (the more concurrent operations include saving/loading presets with/without per-slot filters, preset upload/download), and that the new server doesn't have the same issues.
• If you switch build systems it's nice to add docs for the new system. You probably should fix the docs if you remove the old build system.

What's been said. Feel free to go ahead and work on it, and as long as it still works at the end, there will be no problem merging it :)

I've been going through all the previous commits and I think there are two main milestones:

1. throughout the whole project is the video IO thing (adding, fixing, improving)
2. UI update, including web UI and user interaction schema with physical controls, wireless things

Thus in terms of versions I propose to assign version 3 (major) to the code in current branch. However, having in mind "keep it simple" principle I believe that would be more clearly to use the date of release markings for now, ex.: 24.0501 or 24.0502b (beta), etc

Please share your opinion.

I have a few questions related to the architecture:

1. Why there are userOptions and runtimeOptions and could they be merged in one structure of runtimeOptions?
2. What is the difference between rto->presetID and uopt->presetPreference?
3. What is the difference between Resolutions, Presets and Slots?
4. Why resolution change causes selecting custom preset?
5. Why "Presets" as a term can't be merged with Slots, so resolutions could be one of the options for each slot? ...

Concerning WebUI (master version):

1. Does backup/restore functionality works without an issue ?
2. Is there "remove preset" functionality currently in master?
3. Does OTA works without an issue ?
4. Does "reset/restore" functionality works properly? ...

~Thanks

Oh gee, I'll try to remember..

• user and runtime options: as the comments say, user holds the user's preferences, the selectable options. runtime holds (volatile) system variables, and could be better renamed to something like "activeState" or such. You can tell by the variables the 2 structs hold

• rto->presetID / uopt->presetPreference: same thing here. the rto version is the active state and volatile, while the uopt version is the user's preference

• Resolutions, Presets and Slots: relate to the actual resolution (currently active, i suppose), presets being a collection of common settings that are required to run any of the "preset" resolutions and slots.. are presets that get saved to user custom presets (in "slots")

• Why resolution change causes selecting custom preset: Iirc this is a state the scaler is in, where settings are modified and functions need to work differently from the preset / known settings

• Presets are the entire collection of all the register settings that allow one output resolution to work. You can not easily untangle a "preset" from a resolution. Instead, a preset is often used to dial in most settings, and then something custom may be added on top

• webui questions: i'm out of the loop there, no idea :p

@ramapcsx2, so basically the designed process to store/load presets is as follows:

1. SLOTs are saved into slots.bin where each of 72 slots has structure: 1.1. SLOT_ID (char) 1.2. presetID - currently it's output resolution, I guess it is intended to be the first part of the preset file name (see. point 2) ... etc.

2. for each slot there are 9 types of configs created, which names formatted as input_video_format.SLOT_ID (must be presetID.SLOT_ID) are: 3.1. preset_ntsc.SLOT_ID 3.2.preset_pal.SLOT_ID 3.3. preset_ntsc_480p.SLOT_ID ...etc.

3. current/active SLOT_ID is going to be saved in preferencev2.txt file together with other user settings (saveUserPrefs()).

It feels like this is uncompleted intent to define the "many-to-many" dependency of "slots-presets". Also I cannot find "Selecting a resolution also makes it the new startup preset" functionality. So the reason why I got concerned about this is that what you see in UI is not what you get in this case. From the point of view of an ordinary user you want to open UI -> select config or change + save preferences -> load config -> be happy.

Is the preset file name input_video_format.SLOT_ID intended to allow the device automatically switch presets depending on the input video signal?

So I wonder if this would be the working schema.:

Correct me if I am wrong...

@nyanpasu64 Yo, I'm out of my field here. Is this how it's supposed to work? :p

Just in general though, this whole system has clearly evolved over time, people requesting this or that feature, then others doing various consolidations and cleanups, as well as necessary changes for things like the OLED (iirc). It's become a hydra, and I don't think anyone would be too sour if it'd be redone. It all started as that idea of auto detecting one input and mapping to one of maybe 2 or 3 presets that work for that input.. :p

I think at this point we need to decide what is better to keep:

1. Fixed User presets inside of the slots
2. Dynamic preset identification, i.e.: 2.1. The device identifies input signal and creates a new slot with presets. 2.2. A user is able to adjust presets for this slot and change its name. However user is not able to create slots.

The approach 2 is highly limiting user experience since it rely on signal identification which is actually (signal) not tied to the source (ex: game console) itself. On the other hand if the user is able to switch slots itself. This leaves one point of decision making (user) at the time he needs to switch the slots.

Note that people are generally fine with the capabilities of the preset system. It may be convoluted, but it seems to handle most things well enough :)

Alright, so I think we need to describe this functionality once and for all to avoid possible mistakes in functionality in future. The following diagram present one-to-one relation between slots and presets.

1. The device is detecting video input and uses corresponding preset without creating a new configuration file.
2. When a user decides to store the configuration, "slot-preset" pair is created containing all the necessary parameters.

I'll make this available also in README.

Is the preset file name input_video_format.SLOT_ID intended to allow the device automatically switch presets depending on the input video signal?

maybe? note i've been away from gbs-c development for months at this point.

• switching to a custom slot does not load it until you load preset or power cycle. IIRC this is to allow you to save preset instead.
• a custom slot can be associated with 1 or more custom (slot, input resolution) configurations. the intent is to allow a single custom slot to map to a different scaling configuration based on the input resolution. if a given input resolution's preset is missing, IIRC the gbs-c defaults to 960p output (assuming 480i/p input, no clue what happens with 576i/p input or HD).
• the displayed output resolution on the GUI is determined by the most recently saved input resolution's scaling factor, and is cosmetic (the actual output resolution depends on the input resolution).
• if you save a custom slot in passthrough, it "works" (after great effort on my part) but the gbs-c fails to detect input resolution changes and switch output scaling modes (IDK if this can be fixed, I don't know how input resolution detection works and if the necessary hardware registers aren't updated in passthrough mode).

is the current system good? it's probably not intuitive, but it works mostly.

sorry, i don't understand your diagrams or proposal:

• Fixed User presets inside of the slots

• Dynamic preset identification, i.e.: 2.1. The device identifies input signal and creates a new slot with presets. 2.2. A user is able to adjust presets for this slot and change its name. However user is not able to create slots.

• Does backup/restore functionality works without an issue ?

mostly? backup streams individual files to the browser client which saves them to a file, and restore has the browser client separate these files and stream them to the server/ESP. (leftover files on the ESP are not removed when you restore a preset.)

this system most notably breaks when the buggy web server/etc. corrupts flash and creates files with duplicate or invalid filenames (#421).

• Is there "remove preset" functionality currently in master?

i think?

• Does OTA works without an issue ?

updating over USB works, i don't remember if updating over web works.

• Does "reset/restore" functionality works properly?

i think (hope)? is restore the same as "backup/restore" you brought up earlier?

Is the preset file name input_video_format.SLOT_ID intended to allow the device automatically switch presets depending on the input video signal?

maybe? note i've been away from gbs-c development for months at this point.

Let's say "in theory", because it didn't work on my GBSC as I understood it should work.

• switching to a custom slot does not load it until you load preset or power cycle. IIRC this is to allow you to save preset instead.

I've never seen this shared in a user manual or wherever. I was wondering before I put my hands on it why the device doesn't respond to configuration changes right away.

• a custom slot can be associated with 1 or more custom (slot, input resolution) configurations. the intent is to allow a single custom slot to map to a different scaling configuration based on the input resolution. if a given input resolution's preset is missing, IIRC the gbs-c defaults to 960p output (assuming 480i/p input, no clue what happens with 576i/p input or HD).

Gonna try to wrap my head around...

• the displayed output resolution on the GUI is determined by the most recently saved input resolution's scaling factor,

right, that is what I expect it to be.

and is cosmetic (the actual output resolution depends on the input resolution).

what do you mean by "cosmetic"? It does upscaling the input signal of 480p to 1080p when I have chosen 1080p or it depends ...?

• if you save a custom slot in passthrough, it "works" (after great effort on my part) but the gbs-c fails to detect input resolution changes and switch output scaling modes (IDK if this can be fixed, I don't know how input resolution detection works and if the necessary hardware registers aren't updated in passthrough mode).

yes, I've seen that code. The data/manual about registers was ever available somewhere?

is the current system good? it's probably not intuitive, but it works mostly.

mostly is a keyword :)

sorry, i don't understand your diagrams or proposal:

never mind for now. I am trying to find a workable configuration logic that matches the product requirements. This is not just my wish to change something, it is because my device doesn't work as I believe it should be. The last diagram displays that it's maybe better to attach the only preset to a slot, so ONE slot - ONE preset config. However I have a doubt now that it would be a correct logic.

• Fixed User presets inside of the slots
• Dynamic preset identification, i.e.: 2.1. The device identifies input signal and creates a new slot with presets. 2.2. A user is able to adjust presets for this slot and change its name. However user is not able to create slots.

• Does backup/restore functionality works without an issue ?

mostly? backup streams individual files to the browser client which saves them to a file, and restore has the browser client separate these files and stream them to the server/ESP. (leftover files on the ESP are not removed when you restore a preset.)

this system most notably breaks when the buggy web server/etc. corrupts flash and creates files with duplicate or invalid filenames (#421).

Didn't work for me also. That is what I'd like to fix after figuring out slots-presets relations.

• Does OTA works without an issue ?

updating over USB works, i don't remember if updating over web works.

Didn't work either. Upload only via USB...

---

Is there a description of rto->videoStandardInput parameter values? I have identified just a few values for now.

what do you mean by "cosmetic"? It does upscaling the input signal of 480p to 1080p when I have chosen 1080p or it depends ...?

If you enable a 480i input, load the 1080p output preset, and save over slot A, then slot A will scale 240p/480i (15khz) to 1080p. If you then enable a 480p (31khz) input, load the 1080p output preset, and save again over slot A, then slot A will scale both 15khz and 31khz to 1080p. If you instead load the passthrough preset and save over slot A, then slot A will scale 15khz to 1080p but passthrough 31khz input signals. The label on the UI button reflects the most recently saved output resolution across all input resolutions, but in that slot, each input resolution has an independent output resolution/scaling mode.

Is this a good design? It's useful to scale 15 and 31 kHz signals differently (eg. on a VGA monitor, 960p with virtual scanlines for 15khz but native 480p), but the current UI was confusing to me at first (and probably new users in general).

I think the design was originally implemented this way because even at the same output resolution, different input resolutions require different register contents that can be further modified using the UI movement/scaling buttons (and saving a preset is mostly a register dump/load, though with some additional work performed upon load to stabilize the chip).

I think the design was originally implemented this way because even at the same output resolution, different input resolutions require different register contents that can be further modified using the UI movement/scaling buttons (and saving a preset is mostly a register dump/load, though with some additional work performed upon load to stabilize the chip).

Exactly :)

can't get it to build (with vscode platformio) to test.

• [ ] on Windows, you must with open(os.getcwd() + '/translation.json', 'r', encoding='utf-8') as data: or else you'll get a character encoding error reading a UTF-8 file as text.
• [ ] the build still fails because it's trying to open some Arial file that isn't in this branch now:

Traceback (most recent call last):
  File "C:\Users\user\code\gbs-c\gbs-control/scripts/generate_translations.py", line 130, in <module>
    raise FileNotFoundError(f"Font does not exist: {font}") from None
FileNotFoundError: Font does not exist: Arial

I don't have a windows setup available for testing at this very moment, but I've added encoding='utf-8' parameter. Python doesn't recognize the font name, however it is in defaults list: https://learn.microsoft.com/en-us/typography/fonts/windows_10_font_list Let's find out some default font which is installed on all systems OR the choice maybe given to a user which is going to compile the project.

1. @nyanpasu64 I wonder why Bypass mode is not actually Passthrough? Out of all the 3 bypass modes OutputBypass, PresetHdBypass, PresetBypassRGBHV no one is transferring video from input to output without changes. In fact there is no video at all on any of these modes. Is there a tricky logic which I didn't see or it is intentional behavior?

2. In addition, is there any info for reference on PLC registers? I'd like to understand a few things.

Bypass is an operating mode where the ADC results get transported to the output unit without passing most digital blocks. It doesn't use the SDRAM, doesn't scale, etc, and the mode is actually called bypass in the various registers for it. What still happens is sync extraction and forwarding to the output, where they become CSync, iirc. There are limitations on what this can do, and if you don't get a picture, most likely your display doesn't like the limitations.

You can find all the documentation that exists on the google drive link in the first post here: https://shmups.system11.org/viewtopic.php?f=6&t=52172

My my, I notice the changes are getting very extensive. You're retooling everything, right @way5 ?

Please try to get some feedback from someone available / willing to test, see where this is going, if you can :)