skool2ctl.py Looses line byte span characteristics after conversion

[Rui1973Martins]

If we have this content is CTL file

@ $4000 start
@ $4000 org
b $4000
. Screen Pixels data
B $4000,$1800,32
b $5800
. Screen Attributes data
B $5800,$300,32
b $5B00

it will generate a skool file with 32 bytes per line, as expected Example:

@start
@org
; Screen Pixels data
b$4000 DEFB $00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00 ; { #SCR(LoadingScr.png)
 $4020 DEFB $00,$00,$0E,$8F,$F3,$FF,$E9,$F8,$08,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$08,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00 ;

However, when we run skool2ctl.py, the formatting is NOT preserved in the CTL file, which means that when regenerating the skool file, the formatting is also gone, reverting back to the default of 8 bytes per line.

When parsing the skool file it should acknowledge the fact that there are N bytes per line, and keep that "context" in the CTL file, so that no formatting is lost after regeneration.

WARNING: For some odd reason, the github website does not use a PRE tag or "white-space: pre" on the code tags, so it's mangling the lines into a single one.

[skoolkid]

I am not able to reproduce this behaviour. Can you list the exact commands you're running to do the ctl -> skool -> ctl -> skool conversions?

[Rui1973Martins]

RefreshCtl.bat skool2ctl.py --hex --keep-lines FormulaOne.Skool > "FormulaOne.ctl"

generate.bat sna2skool.py --hex --ctl FormulaOne.ctl FormulaOne.z80 > FormulaOne.skool

[skoolkid]

I'm still not able to reproduce the behaviour you describe. I start with this control file:

@ $4000 start
@ $4000 org
b $4000 Screen Pixels data
B $4000,$1800,32
b $5800 Screen Attributes data
B $5800,$300,32
i $5B00

Then after running sna2skool.py and skool2ctl.py (without the --keep-lines option, which should be irrelevant), I have this control file, which is essentially equivalent to the original, and produces an identical skool file:

@ $4000 start
@ $4000 org
b $4000 Screen Pixels data
B $4000,6144,32
b $5800 Screen Attributes data
B $5800,768,32
i $5B00

[Rui1973Martins]

Did you noticed, that I edited the Skool file ?

There is an extra: ; { #SCR(LoadingScr.png)

I'm not sure if that is the cause, but it's not just run the BAT files.

[Rui1973Martins]

I just tried it myself, and apparently it's working now. Weird!

I can tell you, that yesterday I had to reverse a generation using git, because it was all messed up.

[Rui1973Martins]

But I'm still getting some funky differences.

Some are obvious like whites space differences or generation of Title descriptions for data blocks that I have edited and separated on the skool file, which is fine.

But others, like like that "LAP No" being broken up into 2 lines, are just weird.

git diff attached for reference. formulaOne.diff.txt

[Rui1973Martins]

I can send you the skool file for you to test. But I don't want to include it here.

Here is an extract of the current problematic area. LapNo.problem.txt

And the CTL contents relating to that area (that were generated by refreshCtl.bat):

t $6BF5
. Message "LAP No"
T $6BF5,6,6
t $6BF6
. Message "Pos Car  Diff" (needs to be split ?)
T $6BF6,13,13
b $6C08
. Data block at 6C08

[Rui1973Martins]

Notice that somewhere between the CTL refresh

C $99A5,1
. Result addr = String to write
C $99A6,8
.  Write Extras name string (Length = 10) 010 1 0 000 001 0 0000 =>   1000 1000
. at ( U=0, V=? )
C $99AE,11
.  Write string #R$73BE : "for" (Length = 3)
. at Char ( U=, V= )
C $99B9,11
.  Write string #R$73C1 : "car" (Length = 3)
. at ( U=$19=25, V=? )
C $99C4,1
. Restore AF

and the Skool re-generation:

  $99A6 LD HL,$5020   ; { Write Extras name string (Length = 10) 010 1 0 000 001 0 0000 =>   1000 1000
- $99A9 LD B,$0A      ;  at ( U=0, V=? )
+ $99A9 LD B,$0A      ; at ( U=0, V=? )
  $99AB CALL $9EC5    ; }

We are losing some formatted white space, even though the CTL refresh was done with the option --keep-lines.

NOTE: As far as I know, there is no option --keep-white-space or similar.

But the culprit seems to be the Skool2ctl.py since the CTL file is missing an extra space at the beginning in . at Char ( U=, V= ).

NOTE: This was one of the white-space differences that is visible in the git diff that I sent earlier.

[skoolkid]

The "LAP No" issue is caused by a faulty address in the original skool file:

; Message "LAP No"
t$6BF5 DEFM "LAP No"

; Message "Pos Car  Diff" (needs to be split ?)
t$6BF6 DEFM "Pos Car  Diff"

If the text at $6BF5 is "LAP No" (6 characters), then the next block should be at $6BFB, not $6BF6.

[Rui1973Martins]

OK I probably messed that up. I'll fix that.

NOTE: I had seen Warnings before, when something was being overwritten. But I'm not getting any WARNING on this one. Is that a bug ?

[skoolkid]

sna2skool.py should print a warning if your control file erroneously truncates an instruction (i.e. a machine code instruction, not a DEF* directive), but it doesn't if one DEF* directive overlaps another.

You should take care when modifying addresses in a skool file. The control file ensures that the addresses are correct to begin with, but obviously it can't prevent erroneous modifications, and skool2ctl.py cannot fix them.

[Rui1973Martins]

sna2skool.py should print a warning if your control file erroneously truncates an instruction (i.e. a machine code instruction, not a DEF* directive), but it doesn't if one DEF* directive overlaps another.

But it could. And it probably should.

You should take care when modifying addresses in a skool file. The control file ensures that the addresses are correct to begin with, but obviously it can't prevent erroneous modifications, and skool2ctl.py cannot fix them.

True, but after changing hundreds of DEFM strings bytes back into bytes and regenerating the values and addresses, after splitting them, I'm extremely happy of having botched just a few :)

[skoolkid]

True, but after changing hundreds of DEFM strings bytes back into bytes and regenerating the values and addresses, after splitting them, I'm extremely happy of having botched just a few :)

Changing strings into bytes (or code in any type of data or vice versa) is something I would recommend doing only via the control file. It's much easier to do it that way, and you are asking for trouble if you do it in the skool file.

[Rui1973Martins]

I checked and this happened because there where still lines in the file that overwrite or redefine lines that already had a type. I.e. there where definitions (, c,b ,etc...) that overlapped the same memory area.

This is particularly noticeable with the uppercase version of the commands, since these allow to define a range or count of bytes. So I would suggest to detected these overlapping memory areas and output an error message, to avoid unexpected side-effects.

[skoolkid]

With 3b143a8837aec3d53f4406110992e84ebcf8fd4a, the control file parser now prints a warning when it truncates a sub-block to prevent it from overlapping with the next one.