Sailfish-MightyBoardFirmware icon indicating copy to clipboard operation
Sailfish-MightyBoardFirmware copied to clipboard

Published 20 hours ago • verified publisher icon jetty840

Build from current master exhibits long delays with certain commands

Open DrLex0 opened this issue 7 years ago • 11 comments

(Note: skip or read all the way to my third reply before replying.)

When I create a ff_creatorx-2560 build from the current master, it works, but exhibits very noticeable stuttering every time an M70, M71, or M300 command occurs inside the code. With the official FlashForge firmware and the r1591 build that can be installed through the Sailfish ReplicatorG, those commands also often cause a pause, but much shorter and often unnoticeable.

I build inside the (64-bit) VM available here, using the instructions from the README.markdown file. I have a FlashForge Creator Pro, therefore I build with scons platform=ff_creatorx-2560 (that's basically my workflow, I didn't change anything else). I've tried it with an avr-gcc 7.3.0 toolchain as well, and those builds exhibit the same problem.

Attached are a Gcode and X3G file that demonstrate the problem. Make sure to unload filament before trying this, because it will perform extrusion moves but won't activate any heaters. Stutter-forklift-trunc.zip (a simpler test file that is less useful: StutterTest.zip) The ‘forklift’ file is a cannibalized real print with some LCD display messages and beeps thrown in. On the older firmwares, I notice no delays whatsoever when those commands are activated, on my master build there are very obvious pauses. The other file is a simpler synthetic test and curiously, it exhibits much shorter pauses on both firmwares.

r1519 was from 2015-07-10, and the newest FlashForge build was from 2016/05/23 according to its version screen, but I have no idea what commit it is based on. I have tried to rebuild the r1519 release to check whether the problem has anything to do with my build workflow, or is actually inside the code. Compilation of those older checkouts fails with many errors however. The earliest commit I can build is from 2015-08-07. Some of those builds make jerky movements all the time, but the ones that don't, already exhibit the very noticeable pauses.

Could anything introduced after r1519 have caused the extra delays, or is some additional build magic required to end up with a smoother-running firmware?

DrLex0 avatar Apr 21 '18 16:04 DrLex0

Update: it must be due to a code change. It seems the simple stutter test file I used was not demanding enough, with a modified file from an actual print that has many tiny segments, the difference between older builds and the current one from master is way more obvious. It also shows that my own builds from old commits (I've tried 2015-10-04) do not exhibit the problem, so it has nothing to do with how I build them. It's time for some tedious bisection work to hunt down the commit that caused this…

DrLex0 avatar Apr 21 '18 22:04 DrLex0

I have tracked it down to the ‘hammerfix’ merges by @dbavatar (#162 introduced very weird behavior, remedied by #164 which resulted in the delays). I did some naïve attempts to fix it, but of course these failed due to my lack of familiarity with the code. I'm not even sure whether a fix is possible because as far as I understand it, the changes prioritize handling print moves above everything else. Still, it seems to make no sense that a single command to queue a beep or display a message should cause printing to grind to a halt for almost a whole second, especially because the older code did not have this problem. Maybe the solution is trivial, but I don't think I'll have time anytime soon to dig through the source.

If you wonder why I consider this problematic, the reason is my solution to offload variable fan speed to an external controller. To send out accurately timed commands from within a print, playing tunes on the buzzer seems the only way that wouldn't require any changes to the firmware or slicer programs. All that is required is a post-processing script and some FFT magic on a Raspberry Pi. This works amazingly well in older Sailfish builds where M300 commands can be inserted anywhere inside a string of G1 commands without introducing problematic delays, and they are timed accurately enough to cool even small overhangs or bridges. A current master build however grinds to a halt every time a beep is encountered in the middle of a set of print moves, and playback of the tunes is sloppier (I could work around that by making the detector sloppier as well, but the pauses are unacceptable).

Ways out of this I currently see, are:

  1. (what I'll be doing for the time being) fork Sailfish before those merges, people who want to use this fan system reliably will have to flash a custom build that includes #201;
  2. someone finds a way to get rid of the pauses in the current master code, this would be the best of both worlds;
  3. someone finds another way to send out command strings to an external device from accurate positions within a print. Of course, without causing the same kind of stuttering. Ingenious as the buzzer hack may be, I would rather replace it with something that doesn't have to cross an air gap. Being able to send out commands would be useful for other things than merely fan control.
  4. I throw away all the work I did so far and try to use #183 instead, but I would lose accurate timing for the fan, manual override and monitoring, fan kickstart, and maybe also the ability to run the fan at duty cycles as low as 1%.

DrLex0 avatar Apr 22 '18 13:04 DrLex0

I forgot one: 5. find out why refilling the command buffer is suspiciously slow.

I have ignored my lack of time and started digging in the code anyway, and the delays are caused by two things: refilling the command_buffer while we try to gather as many print commands as possible, and planning recalculation. The latter is inevitable, but the first one is facepalm-worthy. We are reading from the SD card in chunks of 1 byte. Seriously? The whole fat_read_file routine is re-invoked over and over again to read one single byte from a slow medium.

I have implemented a simple read buffer and this has reduced the stuttering considerably. I'm not sure what a reasonable size for the buffer is, but at 32 bytes the pauses are already short enough that they are not worse than before the hammerfix, and going higher doesn't seem to offer any extra benefit. Pull request coming up…

DrLex0 avatar Apr 22 '18 17:04 DrLex0

Yes, hammerfix introduced these issues. Since ~July 2016 I have not had any time to work on Sailfish. (Switched jobs.... left Innosoft/Sun Microsystems/Oracle after 29 years. I've again switched jobs, only this month. The git handle I'm using here might be some clue.)

32 bytes is probably okay. On your printer, after invoking all the menu options and having done a brief print, you can see the state of SRAM by going to Settings > Version Info. It'll show memory high water mark (well, total memory less the high water mark for usage). It should ideally be above 512 bytes to ensure no issues with stack usage during a print. (I honestly don't recall what it's been running at -- you can get a base line using official 7.7 Sailfish.)

tinder-danielnewman avatar Apr 22 '18 18:04 tinder-danielnewman

Enjoy the new job! 😄 I still have about 554 bytes after doing a print with a build from the 4.6.3 toolchain. A build made with 7.3 has more than 1000 bytes free. I seems suspicious that it can squeeze out so much more SRAM, but so far I haven't noticed anything unusual while running those builds.

DrLex0 avatar Apr 22 '18 18:04 DrLex0

just to let you know, I haven't seen any long pause during prints, but smaller ones, very rarely, can occur.

Can you please test my release and tell me if it does the same as before on your side? Consider that I'm using avr 4.9.2, compiled on a Raspberry pi model B.

Also, what GPX conversion plugin are you using? Are you printing over usb or SD?

SaschaKP avatar Jun 21 '18 14:06 SaschaKP

You should read the whole thread, most of the answers are already in there. To summarize:

  • You won't see significant delays unless there are M70, M71, or M300 commands in between G1 print commands. Try the example files 1 or 2 on both a recent build and one made from the source as it was on 2016-06-13. The ‘hammerfix’ changes that were committed after that, cause the delays.
  • Effects of AVR toolchain version or preprocessing tools like GPX, have been ruled out.
  • From studying the source code, it makes sense why the delays occur due to the way in which the ‘hammerfix’ prioritizes filling buffers with consecutive print commands.
  • This is a big deal for me because of this, which works so well that I wouldn't want to replace it with anything else, not even the recent improvements for fan PWM from within Sailfish (which is great but still has unreliable timings and no manual override).
  • The delays are worsened considerably by the inefficient way in which data is read from the SD card. My pull request #202 reduces the delays to a degree that I would consider this issue closed if that pull request is merged.

As for your other questions, I use GPX 2.5.3 invoked in my own post-processing script as gpx -p -m fcp [filename] (but this doesn't matter because the same x3g file will either exhibit the delays or not depending on what firmware it is printed with). And as the above shows, I only print from SD.

I can give your build a try if I find the time, but I am almost certain I will see the same delays. I am now running my own build that includes all commits except the hammerfix, and of course also my two pending pull requests. This build was made with AVR 7.3.0 and has been running perfectly for months. I don't think there is any good reason to stick to 4.9.2. The newer toolchain produces more efficient code, meaning more functionality can be stuffed in the same microcontroller. The issue of incorrect acceleration computations is something from the past.

DrLex0 avatar Jun 22 '18 13:06 DrLex0

I no longer have a mightyboard, but if those commits are having problems you should pull them out. I ultimately gave up on the board, it's simply too small and too slow to allow for effective path planning. Anything you can think of runs into memory and cpu limits. No incremental changes in compute are going to solve that. I upgraded to a DuetWifi which didn't require much more than some crimping to get it installed in my machine, and is vastly vastly more powerful than the mightyboard, but somehow is still an order of magnitude slower than a $5 raspberry pi zero. But it does have some nice quiet stepper drivers.

dbavatar avatar Jun 22 '18 17:06 dbavatar

Alexander,

Would you be willing share a built version of your firmware with the group? I just flattened my PC and havent installed a Virtual Machine to run the compiler yet, and I would like to test your firmware asap. I was the person that started the original stuttering/hammering thread, and actually planned on scrapping the mightyboard idea all together... I would be happy to give it another try.

Thanks so much!

On Fri, Jun 22, 2018, 9:14 AM Alexander Thomas [email protected] wrote:

You should read the whole thread, most of the answers are already in there. To summarize:

As for your other questions, I use GPX 2.5.3 invoked in my own post-processing script as gpx -p -m fcp [filename] (but this doesn't matter because the same x3g file will either exhibit the delays or not depending on what firmware it is printed with). And as the above shows, I only print from SD.

I can give your build a try if I find the time, but I am almost certain I will see the same delays. I am now running my own build that includes all commits except the hammerfix, and of course also my two pending pull requests. This build was made with AVR 7.3.0 and has been running perfectly for months. I don't think there is any good reason to stick to 4.9.2. The newer toolchain produces more efficient code, meaning more functionality can be stuffed in the same microcontroller. The issue of incorrect acceleration computations is something from the past.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jetty840/Sailfish-MightyBoardFirmware/issues/200#issuecomment-399438417, or mute the thread https://github.com/notifications/unsubscribe-auth/AQ4taICki9nacenK6aH2nsOPgipp4xkjks5t_O3NgaJpZM4Teh1F .

caall99 avatar Jun 24 '18 14:06 caall99

@caall99: OK, I have two flavors of builds:

  • My custom cherry-picked build that does not include the hammerfix but does have the SD card reading improvement, it is available here.
  • A build from the current master branch with nothing custom: Sailfish-master-4d8456e-avr7_3_0.zip

I'm very interested in a comparison between these two builds. I wonder how much of the hammering is due to inefficient SD reading and how much is caused by the processor simply being anaemic.

I guess the limitations of the Mightyboard make it unsuitable for printing high detail at high speeds, but I usually print at low to moderate speeds anyway because I prefer quality over speed. For me it works well enough. I've only considered replacing it to get better fan control but then I came up with my somewhat weird solution that works surprisingly well.

DrLex0 avatar Jun 25 '18 17:06 DrLex0

Anyway, as I understand and from my tests (and again, from source), this is only a problem occurring during those commands (specific to certain situations, not during normal prints; the show message, show message waiting for a key press, and beep aren't normally used during a print), I've done some prints using an SD card, with the buffer mods inside, goor results, but in my case it doesn't really matter (that's something I usually don't do, I've always printed via PC). In the end, this is a problem only occurring in this specific case, that weren't tested (by me) by the way, since, again, normally those commands aren't sent out.

Just to tell you why I use old AVR, it's because I'm using what is available on RASPBIAN on Raspberry Pi, that's what I have, since I don't want to install cross-compiler tool or VM elsewhere. It works, it doesn't give any problem, so far, so good.

For what does matter, I do almost 15 hours printing everyday in two different printers, same firmware (mine, and I only recently added some of the public mods, that are specific to who uses auto-skew mods), same hardware...and I prefer the hammerfix at least from my point of view, but I think that's what makes OS software better, or not?

Cheers

SaschaKP avatar Jul 06 '18 15:07 SaschaKP