flashfloppy
flashfloppy copied to clipboard
keirf
HFEv3 hard sector support
After reading this thread I got the understanding that HFEv3 should support hard-sectored formats, seems that allowing to control directly the index signal. Is this feature supported currently in flashfloppy? I think it would be interesting since I've seen there are other ongoing efforts related to hard-sectored formats.
Regards
If you need a "clean" HFE file with hard sectors, you can use this script. It is configured for Micropolis, but would be fairly easy to adapt to other formats. The result is unformatted, so it really only cares about track size, bitrate, number of tracks, and number of hard sectors.
#!/bin/env python3
import itertools
import math
import sys
def pad(bs):
return bs + b'\xFF' * (512-len(bs))
track_data_len = 100000//8 # bitcell data, in bytes
sector_count = 16
track_count = 77
f = sys.stdout.buffer
header = (
b'HXCHFEV3' # magic
b'\x00' # formatrevision
+ bytes([track_count]) + # nr_tracks
b'\x02' # nr_sides
b'\x00' # track_encoding
b'\xFA\x00' # bitrate in kB/s
b'\x00\x00' # rpm (unused)
b'\x07' # interface_mode
b'\x01' # rsvd
b'\x01\x00' # track_list_offset
)
f.write(pad(header))
tracklist = bytearray()
offset = 2
length = track_data_len + sector_count + int(sector_count > 1)
for track in range(track_count):
tracklist.append(offset & 0xFF)
tracklist.append((offset >> 8) & 0xFF)
tracklist.append(math.ceil(length*2) & 0xFF)
tracklist.append((math.ceil(length*2) >> 8) & 0xFF)
offset += math.ceil(math.ceil(length) / 256)
f.write(pad(tracklist))
# Single side of track
track_side = bytearray(itertools.repeat(0x01, track_data_len))
sector_len = track_data_len/sector_count
track_side.insert(track_data_len - int(sector_len)//2, 0x8F)
for sector in range(sector_count-1, -1, -1):
track_side.insert(int(sector_len * sector), 0x8F)
assert len(track_side) == length
track_side.extend(b'\x0F' * (256 - (len(track_side) % 256)))
# Interleave sides
track = bytearray()
for pos in range(0, len(track_side), 256):
track.extend(track_side[pos:pos+256])
track.extend(track_side[pos:pos+256])
for track_num in range(track_count):
f.write(track)
f.flush()
I've rebased #412 on top of master with the async I/O changes: https://github.com/ejona86/FlashFloppy/compare/master...ejona86:hfe-hard-sectors?expand=1
@keirf, given the below, should I move forward with the index_pulses array approach? Do you want to review the changes, or the approach sounds fine and I just push directly? I'll need to resolve the FIXMEs for fake_fired; probably will disable fake_fired based on track-change = realtime/write-drain = realtime (instant as well?).
I now have access to a Vector 4 and normal access is reliable with the branch (I didn't bother testing it with #412). It appears the Vector's behavior matches the suggested behavior of the FDD board manual and it does verify every write, so there is guaranteed to be 200 ms of slack time after writes. Larger writes only write every 5th sector, requiring 5 rotations to write a full track, and the verification takes the same amount of time so large writes have over a second of slack for writes to drain.
IDX cutting out is the Vector swapping to read from the HDD. It uses a combined HDD+FDD circuit board and many of the signals are shared, so the first STP in the screenshot is for the HDD (this was a file copy).
Formatting is not so easy. It writes every other sector with no pause between sides and provides 50 ms head settle time. It writes the entire disk (outside in) and then verifies the entire disk (inside out). The 4k write_bc provides 8 ms buffer, so writes to flash have 58 ms to complete when changing tracks. Still, I had reasonable success with a (newish) SD card and also the Raspberry Pi Zero.
Luckily, writes write the entire sector, starting at the sector pulse, so we only really need the directory blocks to be written correctly which is just two tracks. We can just ignore most formatting errors. (One manual suggested reading the sector id in the sector prior to the one you want to write; I don't know if that simply isn't being done, but I've written large portions of the disk afterward without issue.)
I've rebased #412 on top of master with the async I/O changes: https://github.com/ejona86/FlashFloppy/compare/master...ejona86:hfe-hard-sectors?expand=1
@keirf, given the below, should I move forward with the index_pulses array approach? Do you want to review the changes, or the approach sounds fine and I just push directly? I'll need to resolve the FIXMEs for fake_fired; probably will disable fake_fired based on
track-change = realtime/write-drain = realtime(instantas well?).
You can go ahead. Is it worth doing 4.0a beforehand or wait until afterwards? Some kind of benchmark pre-release is needed I think, for gathering broader testing feedback.
Is it worth doing 4.0a beforehand or wait until afterwards?
Seems fine either way. DSK is still TODO for #426; I have looked into it and have a good idea of what's necessary but had limited time for a period and now I have access to the Vector. If you're fine with lack of DSK for the pre-release, I'd just assume doing it sooner. If we think hard sectors is neat enough that we'd want to do a pre-release, we can wait for it. I sort of expect those of us doing hard sector would be fine for the while using builds from GitHub Actions.
Is it worth doing 4.0a beforehand or wait until afterwards?
Seems fine either way. DSK is still TODO for #426; I have looked into it and have a good idea of what's necessary but had limited time for a period and now I have access to the Vector. If you're fine with lack of DSK for the pre-release, I'd just assume doing it sooner. If we think hard sectors is neat enough that we'd want to do a pre-release, we can wait for it. I sort of expect those of us doing hard sector would be fine for the while using builds from GitHub Actions.
Maybe we do it sooner then. It really is just a selected point pre-release for testing, not a release candidate in any way, and we can pick another after hard sectoring is in. It just makes sense to have a few of these at chosen points imo so we can point users at them for testing, get feedback, and track the effects of the big changes.
HFEv3 hard sector support is now on master. For the while you can use this recent build: https://github.com/keirf/FlashFloppy/suites/3098062873/artifacts/70800000
The HFE image really needs its tracks to be aligned, although limited skew should still work.
@ejona86 It's been a while since I checked in here. Are we ready for a v4.0a pre-release would you say?
I have some ED fixes to do for #496 it will make sense to pull through into master, so I will do that first as ED support is something that may be obviously improved in v4.
@ejona86 I have pushed out v4.0a experimental pre-release. I hope that the very brief release notes are acceptable. Please feel free to edit RELEASE_NOTES if not!
I just pushed an update to scripts/mk_hfe.py for hard-sector support. When using hard-sectors, you should include in your FF.CFG:
index-suppression = false
track-change = realtime
write-drain = realtime
I looked to add that to the Wiki, but it seemed likely to just confuse most users. This will be good until there are dozens of us using hard sectors.
Hard sector support should be pretty complete at this point. If hard sectors don't work for you, let's work through it on a new issue as it could be other things like flash timing and HFE track misalignment.
Not clear what the status and/or availability is for hard-sector floppy support. Does it support only HFEv3 or can I emulate from a dsk image? I have both Heath and Northstar systems that use hard-sector (FM and MFM respectively) floppy formats. Jeff (HxC) started working on support for this a few years ago, but ended up being unable to support Northstar at all due to timing constraints and implemented read-only on Heath.
@snhirsch, HFEv3 is required for hard sectors with the current support. The other existing image formats are only applicable for soft sectors. The support is only present in the v4 release series (v4.0a currently). Make sure to use the FF.CFG snippet I mention in my last comment.
The work has only been demonstrated to work on my Vector 4 using a blank image and formatting it with the Vector 4. You can use master's scripts/mk_hfe.py to generate an image of your own with the --hard-sectotrs N flag. There were some errors during formatting, but ignoring them worked fine.
There are some test images at https://hxc2001.com/vrac/HS/ and https://github.com/keirf/FlashFloppy/pull/412#issuecomment-753545763, but I have noticed some images have poor inter-track alignment which may or may not cause problems. There is some code in place to handle small levels of misalignment. But based on an earlier comment of mine, I think there's little hope for the heath_kf.hfe image without modifying the file to manually align the tracks.
Northstar was tested in #412 and the experience drove me to the async work (#426; done except for DSK and QuickDisk, and hopefully seeing wider testing with the 4.0a release). I'd love further testing, and the current code has much stronger pulse timings than in #412 so I am hopeful. I'd suggest trying to boot with some of the existing Northstar images and seeing where it goes.
Thanks for getting back. Yes, all those images that Jeff posted were produced during the period of time when he was trying to get HS support working. He took raw sector images I sent and generated those HFE files from them.
DREM is able to fully emulate both NS and Heath formats - even supporting low-level format! But, it's an expensive solution.
I'll try your code and see how it goes.
@snhirsch, also, when you report back, it'd be helpful to know if your Gotek is STM or Artery (https://github.com/keirf/FlashFloppy/wiki/Gotek-Compatibility). Based on #351, it seems you may have a STM which would be preferred for this because of its greater RAM.
It would be possible to support image formats other than HFE, but each hard sectored system is its own special snowflake, so it is more convenient to leverage a general-purpose format like HFE and rely on PC software to convert as necessary.
I'll let you know what I find. I have only STM based hardware at this point. Genuine Gotek and a white Simulant board.
Sorry I didn't have much time lately to test. I had the chance to upgrade the Gotek clon (STM32 based) on my Northstar Advantage to 4.0a and I think I'm still experimenting similar problems with writes as I used to have with @ejona86's fork: I created a blank HFE with the updated mk_hfe.py script (35 tracks, 10 hard sectors). Tried to format it with the Advantage CP/M utilities and got lots of low level errors from the advantage. Similar results using one of the HFE images we used during previous testing (also with the ones that were cleaned up with the script prepared by @ejona86) Just mention that I included a FF.CFG file in the pendrive with the recommended options.
I will try to wire analyzer and serial port to check logs and the relevant signals, in case it can be helpful.
The Northstar has extremely tight timing requirements that have been hashed out in some depth on the HxC forums. I'm also able to capture activity on a logic analyzer (Hantek 4032 with Sigrok software) if that helps.
Poking around, it seems you saw Heath working a while ago. https://torlus.com/floppy/forum/viewtopic.php?p=18213#p18213 . In that case, maybe that is a better starting point if the Northstar is known to be particular. I'll continue thumbing through the thread and see what useful details y'all discovered in the past. Seems some of that previous work was with NSI, not HFE, which might have made it a bit more particular.
Seems we're all discussing here now, so reopening.
Looking with a logic analyzer would be helpful. Sharing a recording is maybe best, but also screenshots of particular parts you notice something fishy is great too. I typically use Sigrok myself. I may have a read through the Heath and Northstar manuals; I think I've already found some useful manuals in the past for them.
Tried to format it with the Advantage CP/M utilities and got lots of low level errors from the advantage
@teiram, you might see if the disk still works okay, even if there are formatting errors. You might also formatting twice. I expect there'd still be formatting errors the second time, but the disk may be more usable. (I think on my device it may only look at the sector number in the sector header when writing, and wouldn't care about wrong checksums from lost writes during the format process.)
That said, I am disappointed to hear it only sorta works for reads from those images. Flash read latency may be coming into play. That could maybe be improved by delaying reading from flash for a few ms after swapping tracks to make sure another step isn't coming. We may be able to notice this being a problem via the logic analyzer.
@ejona86 does Github allow me to PM a dropbox link to you? I have a bunch of information collected on both Northstar and Heath that you may find useful. Would rather not announce to the world just on general principles.
From https://torlus.com/floppy/forum/viewtopic.php?p=18258#p18258:
The "fast" step rate is 12 msec. The slow step rate is about 50 msec (!).
Ooph. Nope, adding a delay before reading from flash wouldn't help. When first looking at this stuff I saw a datasheet for the ST506 with its batched steps and keep forgetting how slow steps are for a floppy. Slow step rate may be useful just for more slack for writes and maybe reduce a rarer read error. And it seems you did something like that in the past and saw errors decrease.
@snhirsch, I'll try to read this stuff carefully just to reduce the likelihood of deja vu for you. But do expect some :)
You can email me at [email protected]
Sorry if I'm missing the obvious, but I downloaded flashfloppy-v4.0a and don't see any mention of hard sector support anywhere. If it hasn't been merged yet, which branch do I pull from?
It is mentioned in the release notes: https://github.com/keirf/FlashFloppy/releases/tag/v4.0a . There's nothing all that special in the firmware itself that would scream "HARD SECTORS!!!1". Maybe a dedicated page for hard sectors in the wiki would work, but at the moment with only me having success, it isn't documented too "loudly."
It's just that if you give it an HFEv3 file it will actually observe the IDX pulses within. Then the FF.CFG configuration would keep those IDX pulses steady. The "core" commit would be https://github.com/keirf/FlashFloppy/commit/991a7747e5750a13b3abbadc88a4e6911b7bf271 , but there were many preparatory commits and some follow-up commits as well. The "async" work in particular was important and allows RDATA to resume soon after WGATE ends; previously RDATA would be delayed while writes drained to flash.
Thank you for the explanation! Much appreciated. I'm always wrestling with horizontal surface theory in my shop, but should be able to have the Heath setup in a day or two.