Micropolis decoder struggling

[cube1us]

The Micropolis hard sector decoder is only correctly decoding about 1/3 of my Mod I disks (35 tracks, 48 TPI). I haven't tried the Mod II's much yet, but from what I did try, I suspect it will struggle there as well.

If you'd like me to capture some flux files etc. let me know what parameters. I have flux files for them all, but they may not have quite what you want.

[ejona86]

I would be interested in some flux files to improve the support. For parameters, if you are using greaseweazle, then at least 20 revolutions to get 1.25 "real" revolutions. If you already have flux files, then they are likely fine as-is. 1-2 flux files is probably enough, but obviously I can verify more of them if I have more.

One thing that would be helpful is to know what machine you are using the Mod I with. The host has a lot of say in how they were written. I have come across some (well, at least one) VGI files that don't seem to use the normal checksum, so I suspect there may have been some variation present. It is interesting that 1/3 of your disks don't work well; I would have suspected it to be more consistent than that. Do you think there is anything special about those 1/3 of the disks that make them different from the other 2/3?

[cube1us]

See issue https://github.com/davidgiven/fluxengine/issues/322 for a 40 revolution SCP file for a Mod II. [Or, do you mean something else by "flux file"??]. In my case, ALL of both the Mod I and Mod II disks were written using a Micropolis controller on a 2MHz 8080 (Altair), mostly under CP/M. I doubt that there is anything special about the different disks. (The only difference between MOD I and MOD II is TPI - density and recording mode is the same.)

[ejona86]

I had the same problem as you getting fluxengine to do much with that SCP file. Since you had some that are working, I thought you had found an alternative import path and were now using .flux files (the fluxengine-native format). But if it is just that some SCP files cause fluxengine to fall over and others don't, then it seems it may not be micropolis related and that helps guide me where to look.

[ejona86]

@cube1us, how does this issue differ from #322? Since I demonstrated that once the SCP import is working properly the Micropolis decoder seemed to work fine, is there anything more to do here and instead just let #322 track the SCP issues?

[cube1us]

#322 was for a .scp file created using greaseweazle to read the floppy - in that case it would not process a .scp file AT ALL. THIS issue was after I got a fluxengine hardware and tried to read Micropolis floppies with it -- no .scp files were involved. The results were/are suspicious because so many floppies had errors and would not successfully decode. I had successfully read almost all of these floppies 16 years ago just fine using my Altair. And I have had similar success with non-Micropolis floppies with my Greaseweazle this year, and all of these floppies were stored similarly, which led me to conclude that the decoder was not working as well as one might hope, since this issue relates to .flux files read with fluxengine hardware.

[ejona86]

Could you share some .flux files for floppies that have issues being read? Using --copy-flux-to to store the .flux when doing a read with fluxenigen is enough to create a useful flux file. If you have an SCP file from greaseweazle for a disk that had trouble decoding, I could use that just as easily (now that I've figured out what was broken with SCP).

[cube1us]

Sure, here are some, both Mod I and Mod II along with my logs.

Once you have downloaded these, please let me know, and I will remove them from the comment, and please do not share them.

[ejona86]

I've downloaded the files.

[ejona86]

Those two files mostly decode, but it seems the inner tracks weren't in good condition and see corruption. I don't see anything unexpected going on. The configuration could maybe be tweaked to improve decode quality, but the last track in particular looks like it'll have a hard time (based on fluxengine inspect's histogram that shows the flux peaks are touching).

[cube1us]

Interesting, as they read just fine in 2005, using these same drives, using the Micropolis controller. Mostly it seems to be struggling on the inner tracks, as you noted. I doubt that they are "corrupted" per se - it seems more likely that the hardware decoding in the Micropolis controller just works better for them. I am unlikely to spend the time to set up my old machine and try and read them again, though, so feel free to close this out.

[hharte]

I’ve observed a similar issue with many of my Micropolis disks, and also need to go back and re-read them with the original machine to see if the disks are still good. They were written with the Micropolis controller, not the FD/HD controller, and are single-sided 70 track 100TPI disks.

[davidgiven]

Belatedly: I've recently applied a major fix to the decoder which makes reads much more reliable. Could you have another try and see if anything's improved, please? It just needs a decode and not a physical read, so the old flux files should work.

[hharte]

Hello David,

Thank you for all the updates to FluxEngine lately. I had a chance to try the new decoder with the latest version of FluxEngine. Unfortunately, the latest version cannot detect the Micropolis sectors properly. I bisected the issue and it looks directly related to the new decoder:

As of git checkout 1d32a4d98499ffc6915f2f590c638406a4b7a905, fluxengine is not able to find any sectors in the Micropolis .flux files.

The FluxEngine read of micropolis .flux files works up to:

git checkout 28aff784695ee58dd7e25da9a5f0fddb682e440d

This link has a .zip containing two .flux files: one that decodes correctly (before the decoder change) and one that has trouble on the inner tracks (the reported issue in this bug.) Both of the attached .flux files are obtained from single-sided 330K disks read using a Micropolis 1015-II drive using FluxEngine hardware.

https://www.dropbox.com/s/o7gglusxhvz0rja/micropolis_flux.zip?dl=0

Read of flux file from good disk (with old decoder:) ./fluxengine read micropolis --heads 0 -s MDOS_System_Disk_Vs_8_2_MASTER.flux

0.0: 857.537 ms in 218048 bytes 68 records, 68 sectors; 2.03us clock (491kHz); 16 distinct sectors; logical track 0.0; 17408 bytes decoded.

With the new decoder, I get: 0.0: 857.537 ms in 218048 bytes 68 records, 0 sectors; 0 distinct sectors; 0 bytes decoded.

Thanks, Howard

[davidgiven]

Oops, the Micropolis decoder got broken during the change... I have fixed it. The good news is that both flux files now read perfectly, almost. I say almost because the Vector disk appears to have track 26 duplicated on track 0, which is peculiar --- it's the same data, but there are no sectors labelled as being from track 0. I don't think that's my fault.

Can I keep the MDOS flux file for use as a Micropolis sample?

[hharte]

Thank you! I’ll give it a try. I saw the track 0 issue as well, I have a feeling this is how the CP/M disks were written originally. But now that reading is working much better, I have a bunch more Micropolis disks to read. I’ll see if this is consistent. Sure you can keep the .flux files as a reference/test case. Unfortunately I don’t have any double-sided Micropolis disks or drives to get you a double-sided .flux file.

Thanks again, Howard

[hharte]

@ejona86 I found a clue as to what may be going on here. The sector SYNC pattern (00's followed by 0xFF) may be detected too "early" after the sector pulse. This is probably due to Micropolis' weak sync pattern combined with FluxEngine only matching up to 64 bits. Detecting the erronious "early" SYNC pattern, and seeking forward looking for another SYNC seems to have resolved this issue for several disks I've tried.

 62.0: 200 ms in 64768 bytes
SYNC found too early for Physical Sector 3, sectorPos 36.80033333333333ms sync 36.814166666666665ms delta: 0.013833333333336206ms
SYNC found too early for Physical Sector 14, sectorPos 171.59416666666667ms sync 171.62775ms delta: 0.03358333333333352ms
SYNC found too early for Physical Sector 15, sectorPos 232.92083333333332ms sync 232.9343333333333ms delta: 0.013500000000006621ms
       19 records, 19 sectors; 2.01us clock (498kHz); 16 distinct sectors;
       logical track 62.0; 5282 bytes decoded.

...

 76.0: 200 ms in 64896 bytes
       18 records, 18 sectors; 1.94us clock (515kHz); 16 distinct sectors;
       logical track 76.0; 10008 bytes decoded.
     Tracks -> 1         2         3         4         5         6         7
H.SS 01234567890123456789012345678901234567890123456789012345678901234567890123456
0. 0 .............................................................................
0. 1 .............................................................................
0. 2 .............................................................................
0. 3 .............................................................................
0. 4 .............................................................................
0. 5 .............................................................................
0. 6 .............................................................................
0. 7 .............................................................................
0. 8 .............................................................................
0. 9 .............................................................................
0.10 .............................................................................
0.11 .............................................................................
0.12 .............................................................................
0.13 .............................................................................
0.14 .............................................................................
0.15 .............................................................................
Good sectors: 1232/1232 (100%)
Missing sectors: 0/1232 (0%)
Bad sectors: 0/1232 (0%)
IMG: wrote 77 tracks, 1 sides, 330 kB total

The change I have currently is a gross hack, but perhaps we can find a better way to do this:

		auto syncDelta = _fmr->tell().ns() - sectorPos.ns();
		if (( syncDelta > 0) && (syncDelta < 1e6)) {
			std::cout << fmt::format(
				"SYNC found too early for Physical Sector {}, sectorPos {}ms sync {}ms delta: {}ms",
				_hardSectorId, sectorPos.ns()/1e6, _fmr->tell().ns()/1e6, (_fmr->tell().ns() - sectorPos.ns())/1e6) << std::endl;
			_fmr->seek(_fmr->tell().ns() + 100);
			_sector->clock = _fmr->seekToPattern(SECTOR_SYNC_PATTERN, matcher);
		}

Before making this change, the same disk read with these results:

     Tracks -> 1         2         3         4         5         6         7
H.SS 01234567890123456789012345678901234567890123456789012345678901234567890123456
0. 0 .............................................................................
0. 1 ..................................X..........................................
0. 2 ...........................................X.................................
0. 3 .................................................................X...........
0. 4 ...........................X.......................X..................X......
0. 5 .........X..................................................X................
0. 6 .............................................................................
0. 7 .............................................................................
0. 8 ...................B..............X..........................................
0. 9 ..............................................................B..............
0.10 .....................X.......................................................
0.11 .............................................................................
0.12 ....................X...............X........................................
0.13 .............................................................................
0.14 .............................................................................
0.15 ............................................................B................
Good sectors: 1217/1232 (98%)
Missing sectors: 12/1232 (0%)
Bad sectors: 3/1232 (0%)
IMG: wrote 77 tracks, 1 sides, 330 kB total

Here is a .flux file that illustrates the "early" sync issue

I should also mention that this needs some scrutiny to make sure that the decoded contents are in fact correct. The seekToPattern() function can seek past a sector pulse to the next sector, but I believe in this case, since Micropolis encodes the sector number in the sector data, it won't cause incorrect interpretation of the sector. For North Star, I had to be really careful about this, since there is no metadata in the sector indicating which sector the data is for. To handle the North Star case, I simply calculated the _hardSectorId after finding the SYNC pattern.

To verify North Star, I wrote some disks with known patterns to be alsolutely sure the decoder worked properly, but I have not done this with Micropolis yet.

[hharte]

A simpler fix that seems to work well is to seek forward about 500uS from the sector pulse before searching for the sync pattern.

[hharte]

PR #463 partially addresses the decode issues for some of the disks I have.

Better hard sector ID'ing would also help somewhat, in cases where the sector ID encoded in the sector header is corrupt, but matches a valid sector number (0-15.) This would be particularly helpful for MZOS disks, which do not include the header in the sector checksum.

[davidgiven]

Merged.

Are the sectors always in order? If so, the sector ID could be calculated from the number of sectors seen. This will require index-synchronised reads but that's not too much of a problem. The MX decoder does this.

Also, do you have a real flux image of a blank (or otherwise non-copyrightable) disk which I can use to add to the test corpus in github.com/davidgiven/fluxengine?

[hharte]

Hello David,

For the several hundred disks I've tried for all known operating systems on the Vector MZ (CP/M, MDOS, MZOS, OASIS) the sector IDs are in order. I do calculate the sector ID (and sync with index) in the North Star decoder. For North Star, calculation of the hard sector ID is required, as there is no sector header. Only SYNC followed by Payload followed by CRC. Some things I noticed with the calculation approach are:

1. It only works with FluxEngine HW (at least the way I did it for North Star,) since Greaseweazle does not support differentiating the Index vs. Sector pulses. @ejona86 had some WIP commits that I tried, that used seeking through the flux to determine the sector IDs based on the gap timing between pulses. This can work with the GW.
2. @ejona86@'s calculation method works very well for the most part, but cannot work as-is without seeking through the flux to identify the index pulse.
3. One deficiency is around "retries." FluxEngine concatenates all of the flux from retries into a single segment in the .fl2 file. For example, if a 220ms-long track read required five retries, the .fl2 would contain 1100ms worth of flux, with no apparent way to differentiate the retries. This leads to re-interpretation of .flux files not always matching the orignal read. I have a set of several .flux files for which this is the case, and at some point I will work to resolve the issues, but perhaps the best thing might be to have a way to signal the start of a retry in the flux.
4. The first sector pulse is missing in the flux stream for hard-sectored disks. It is assumed to be at offset 0. I believe this leads to possibly missing a sector on retries when reprocessing the .flux files. For example, when at offset 0, seekToIndexMark() will advance to the second sector.
5. seekToPattern() will advance to the next sector if a SYNC is not found, regardless of the sector pulse.

Since we don't want to seek through the flux anymore, one solution may be to "map out" the sector IDs at the beginning of the track, to avoid seeking through the flux while decoding sector records.

I do have many Micropolis .flux files and I will search for some that do not have any copyrighted data on them. Most are stored in the older .flux format, but I can convert them or re-read them as .fl2. It may be better to re-read with the latest version of FluxEngine, as it identifies some previously mis-decoded sectors. For example, if the track ID did not match the physical track ID.

Thanks for your help, Howard

[davidgiven]

Re seeking through the flux: the problem was that any seek would require resetting the flux decoder, and the old code would do a several seeks for every records, so making it impossible to properly decode gapless formats --- the MX in particular. (The Amiga continued to work, but only by chance.) So seeking in advanceToNextRecord() is fine if the format's going to require a flux decoder reset anyway. You just need to make sure to call resetFluxDecoder() after doing it.

Re (3): there is actually a special event you can search for, F_DESYNC, which indicates the interval butween successive reads.

Re (4): good point. It's probably worth artificially adding an index pulse at the beginning of synchronised reads.

[hharte]

Thanks David for the tips above. I'll give those a try when I have a chance.