flashfloppy
flashfloppy copied to clipboard
keirf
Better buffering for ED track formatting
The Sinclair QL + Miracle Systems (Super)Gold Card combination has the ability to use ED disk drives. However, the geometry used is completely different from the IBM-PC one.
Could the following be added as a known IMG size/type?
cyls = 80 heads = 2 secs = 10 bps = 2048
May not be able to keep up with whole track formatting. Is it possible to try normal writes to an existing image?
Unfortunately I don't have an existing image and none of the emulators support ED format floppy images.
I think that I may have to call ED floppy disks unsupported/unsupportable. Obviously the data rate is too high. (On the original drives it was above the manufactuer's specification. PC disks were 80 track, 2 sides, 36 x 512byte sectors per track.)
I've been informed about a utility which will generate ED floppy images, so I'm re-opening and will test this evening.
I've placed a pre-formatted ED image file onto the USB drive and set up IMG.CFG and the drive works correctly for reading and writing normal files, so it does look to be that formatting is a step too far and the Gotek can't keep up.
Would you like to provide updated wiki text for ED support?
Also I am happy to provide you a firmware to test which may support ED formatting too. Perhaps next week?
Actually your stick is really slow at small writes and won't keep up. Strange as it is a decent branded stick. Perhaps try another? You are looking for significantly shorter sector write times in FFLOG.TXT. Note that currently the timing is highly variable and never less than about 12ms,often way more. Whereas the host is pushing out a sector every 5ms.
I've updated the Wiki entry text in the other issue thread.
I think it's probably safer to manage expectations and state that formatting doesn't work. If people find that it does for them then that a bonus but it's not something you'll have to support.
I was wrong in my 5ms per sector. Because these are 2k sectors the host pushes one out about every 20ms. So your stick is close to coping, it just has slow writes every now and then. I'm confident you could find a stick that does keep up.
I wonder if it depends upon whether a block has already been used and hence needs copying, blanking and rewriting. I very much doubt that USB sticks have wear levelling (and definitely no TRIM).
Anyway, if it's not dependable then it's probably best not to advertise that it may work.
Isn't it possible to cache the data which gotek is going to write, in order not to overload flash cards/usb sticks? And actually inform if it gets overload?
From memory, the stm32 cpu only has 32k ram in total. There probably isn’t a lot left to implement a write cache.
That would be a lot of work for an edge case wouldn't it? Still, if there is space in SRAM a read/modify/write track cache would probably be useful. i.e. on read or write read a whole track (from the USB image file) into cache and flush with the next track move, writing back if there has been a modification. Still, a lot of work I guess.
@penfold42 Doesn't the Gotek use a PIC18F4620 like the HxC devices? If so then that has 3,968KB of SRAM on-board. Of course, that memory would also hold the Flashfloppy system.
64k sram. Currently 32k is used to buffer incoming raw bitcells. Enough for an HD track but not a whole ED track (50k). Sectors are written to usb as they are decoded.
There is also a buffer cache on top of USB which can cache approx 16k. This is underutilised on the write path (data doesn't pass through it at all). On the write path we should probably greedily decode into the buffer cache and then write back dirty usb buffers. We should be greedy because that 50k raw bitcells decodes to 18-20k of sector data. So it's much more efficient to cache that.
This is what I would aim to do if I work further on this ticket.
From memory, the stm32 cpu only has 32k ram in total. There probably isn’t a lot left to implement a write cache.
Well. The question consisted of two parts - caching (as much as possible) and informing about writing problems. F-F suffers from that infamous silent “didn’t work out”, when you find writing gone bad only after everything else checked. It’s not how it expected to be. Proper feedback about writing failure is really needed.
The issue with this SanDisk stick however is that I see a worst case sector write back time of 140ms. During that time we cannot decode more sectors (as that is done in the main loop same as write back). We would instead buffer in the 32k ring which can buffer 64% of an ED track. But in 140ms we receive 70% of an ED track. That's no good we need a less choppy usb stick. And if we have that, the 32k ring may suffice alone.
Is it possible to add some sort of easy to use benchmarking, to test writing time and inform if its not sufficient for proper writing? I still don't know if i am using good enough (though didn’t really needed to write, yet).
Further thought on this: if I allowed dirty buffers to accumulate in the cache and write back a bit more lazily. I could do this only for tracks that don't fit in the 32k ring buffer and only when I detect a whole track write.
I think this would make ED formatting reliable.
@Volutar At HD rates and lower, whole track writes are already reliable as a whole track can be cached in the ring buffer. Furthermore write to the next track is delayed until writeback is complete and FF starts generating index pulses again, so the host cannot get ahead of us. In short I wouldn't worry about it. This ED rate formatting problem is fairly specific.
Blue sky idea: If the ring buffer can handle the overhead of some very simple RLE-compression that should cover the ED formatting without problems, right?
Compression of the ring is possible but probably more hassle than the dirty buffer scheme.
Then again, seeing the biggest problem is during formatting where most of the data being written will either be zeros or a long run of another value this should compress massively even with a simple RLE encoding.
@keirf i see. Random thought about compression. Ring buffer is MFM based (ie 16 bits per 8bit). But it’s not entirely aligned. Thought I think compression via MFM encoding-decoding is possible. ED disks if i’m not mistaken have 25000 bytes per track (or 50000 bytes in MFM form). There is an UDI image format used in zx emulators. It’s pretty rawish, but it stores aligned byte stream (not MFM) with supporting bitmask of special sync bits (for C2 and A1). So its like 8+1bit. Maybe its not bad idea to arrange memory buffer similar way (to avoid MFM 2x sized stream).
Sometimes formatting uses weird bytes (though allowed), to perform some copy protection.
Actually, if you're buffering/caching a whole track then maybe you can support some silly tricks drivers try to do, such as formatting more sectors than exist (assuming that it'll wrap around to the beginning of the track again) by just resetting the pointer in the ring buffer to the beginning if there's an overflow of data, so simulating what the dumb hardware would do? Just a thought.
Those tricks already work fine with HFE image format. The problem is that IMG is in a sense pre-formatted. The sector arrangement and numbering is implicit (because IMG images are headerless and don't hold such information explicitly). Once the track wrote gets written back into the IMG and reading starts up again, everything goes back to the same fixed sector numbering...
In short if you need tricks you must use HFE.
Yeah, I guess it would only work if you recorded the raw MFM data stream and reset the recording to the beginning of the buffer every 0.2 of a second after the "index mark" and then interpreted the data after the write had finished. Then you'd have to worry about the overwritten "track" not being on proper sector boundaries. Fair enough.
From memory, the stm32 cpu only has 32k ram in total. There probably isn’t a lot left to implement a write cache.
Datasheet says stm32f105 has 64 kbytes of sram.
If there is anyone still watching this ticket, I'd like to say I'm hopeful that ED formatting will work as of FF v3.25. The original issue was not one of buffering (ring overflow) but a bug in write ring handling which meant that ED track formatting could never have worked in earlier versions, even with a sufficiently fast USB stick.
I am interested if anyone would care to test ED formatting on v3.25?