digpak
digpak copied to clipboard
jratcliff63367
Future direction / assembler plans
Recently, there has been interest in creating digpak drivers for new devices, and possibly resurrecting digpak usage for new homebrew/retroprogamming. @volkertb , as one of the stewards of these sources, are there any plans to translate them to NASM?
I'm asking because I would like to strongly recommend NOT translating them to NASM, and instead leaving them as their original TASM "IDEAL mode" dialect. The main justification for this is that there is greatly added value in preserving their ability to be assembled in actual DOS, on actual old hardware, so that new development can be done against the actual new sound devices. There is no NASM DOS build that works on 286 or lower processors, so developing speed-sensitive code that must run on those processors must be assembled on another system and then copied over -- an inefficient workflow.
Additionally, IDEAL mode provides typing and other safeguards to ensure consistent code generation. Removing it might introduce side effects or other behavior that manifests itself as subtle or inconsistent bugs that may be difficult to troubleshoot.
"One of the stewards of these sources". Wow, I'm honored. :blush:
You make a completely fair point, @MobyGamer. In my occasional n00b attempts to port DOS assembly sources such as TEMU/VSB, ich2player, SoftMPU and DIGPAK to open source assembly dialects (whether FASM, WASM v2 or NASM), I have also come to realize that TASM's advanced features such as type safety and even some forward-thinking object-oriented features are actually quite powerful and helpful. Also, quite a few popular DOS programs and TSRs, including open source ones, are written in TASM/IDEAL dialect.
My big gripe with TASM is that it's closed source and a bit tedious to obtain legally these days. You can get a recent version for free by registering at Embarcadero, but the license terms are quite strict, allow only non-commercial use, etc.
Also, although some open source assemblers (notably NASM and WASM) advertise some level of TASM compatibility, it doesn't come even close to being fully compatible, and in my experience, most TASM sources fail to build with them.
I know there is an old Perl script floating around the Internet called nomyso that supposedly converts TASM code to NASM code, but it's far complete, hasn't been updated in over a decade, and doesn't seem to support many advanced IDEAL mode features. Both NASM and WASM have command-line parameters that supposedly enable some level of TASM compatibility, but I haven't had much success with that either. Although I would like to give @jmalak a shoutout for maintaining the Open Watcom v2 fork for almost a decade now, and having done a lot of effort to improve TASM and IDEAL mode compatibility in WASM. Thanks, Jiří! :slightly_smiling_face:
I think the best way forward for the FreeDOS ecosystem might be to either develop a translator tool like nomyso but with (near) 100% TASM compatibility including IDEAL mode, or to organize a crowdfunding campaign to collectively purchase the rights to the TASM source code from Embarcadero and release it under an open source license.
Heck, to make it interesting, we could negotiate with them to use a smart contract for this. The smart contract would be linked to a wallet that people could send their contributions to, Embarcadero could send the source code to it, the smart contract would then somehow verify that the source code works (perhaps through an oracle that tells a public CI environment to assemble the sources and build a "Hello World" assembly project with the resulting build). Once both the minimum funding amount has been met and the uploaded source code has been verified to be legit, the smart contract would then transfer the contents of the wallet to Embarcadero.
Okay, I'm letting my creativity run a bit amok here. :sweat_smile: It would obviously be completely outside the scope of DIGPAK.
But I'm curious about your thoughts on this. Should we "liberate" TASM, or otherwise reimplement it or develop a highly compatible translation utility to allow the building of TASM sources with existing open source assemblers?
We can also take this discussion elsewhere, if you prefer. :slightly_smiling_face:
Back to your plea not to migrate DIGPAK to another assembly dialect: I'm all for it, as long as we can figure out some way to successfully build and debug it with a completely open source toolchain. I know a lot of other retro coders aren't as hung-up on being independent from proprietary software, but it's important to me, and I'm pretty sure I'm not the only one who thinks that way. Also, there is simply the practical aspect of not being able to include these sources in a distro like FreeDOS as long as they depend on non-Free software.
My opinions on archival and preservation are hopefully well-known by now :-) While I'm not a complete Internet Archive anarchist, I firmly believe that DOS coders shouldn't worry about the legality of distributing software that is well beyond economic recovery by the current rights holder. I believe the practical choice is, to use your terms, "liberate" TASM. Going through a ton of effort to port to NASM has no benefit other than adhering to the ideal of a license that may not be provably valid, and only has downsides (can't develop on older hardware, might introduce bugs, etc.)
That said, I was almost certain there was a version of TASM that was distributed for free; Turbo Pascal 7 was famously made completely free as an updated French version 7.03, although I haven't checked to see if it included TASM like Borland Pascal did.
Another option: Nothing says you must provide TASM, or tell people where to download it. You can simply state that these sources require Turbo Assembler to assemble, and leave it at that. Whether others purchase second-hand copies of Turbo Assembler, or acquire it via some other means, is neither your concern nor your liability. And if bothers other people, maybe create a TASM branch and move all of the code into it as a sort of archive, and leave the main branch for NASM (or FASM) ports.
As for the discussion, I think it's ok, even appropriate, to have it in public, so that others may comment and contribute.
Oh, I didn't mean to take this discussion private. It's just that it's kind of off-topic (or at least not limited to just DIGPAK), and would justify setting up a project or repository of its own.
As a practical argument, dependency on TASM (as long as it isn't released as open source) would make projects like DIGPAK a non-starter to include in a distro like FreeDOS. We really need some kind of completely open source toolchain to build these projects. The same goes for an existing open source retro DOS project like SoftMPU, by the way.
Perhaps the freedos-devel mailing list would be a good place for us to propose the idea of buying out TASM from Embarcadero, or otherwise developing a 100% compatible open source alternative to it?
I wasn't aware this repo was meant to be a FreeDOS-exclusive project...? I can respect translating to an open source assembler for the needs of that project, but I don't feel other areas (such as people retroprogramming on vintage hardware) should be penalized. That's why I suggested possibly maintaining two branches.
Buying TASM and then releasing it via an MIT or even public domain license would be awesome, but I've looked into plans like that before and the price is usually prohibitive (the rights holder wants a return on their investment, and usually won't discuss anything under 5 figures). However, it might be worth pursuing anyway, because it's possible Embarcadero might agree to full use and distribution of TASM for any open-source project. Not the same as open-source TASM, but being legally free would certainly be a win.
I wouldn't try to make an open-source TASM from scratch; it would be more practical to simply fix the IDEAL mode support that supposedly exists in NASM (which would benefit others too).
I decided to fix the sources to be understandable by MASM, as TASM IDEAL mode is full of bugs. Another point is that there seems to be no other assembler, besides TASM that understands IDEAL mode, while MASM is widely used, and other free assemblers exist, jwasm notably.
Full of bugs? Citation needed? The whole point of ideal mode was to fix bugs in MASM. If you are making them compatible with MASM, please test with MASM 6.x, as all prior MASM versions have assumption issues and code generation bugs. Also, please do a binary comparison after translation to ensure that the same code is being output. Bugs in TSRs, especially those that interact with hardware, are incredibly hard to debug...
Assuming that the port to MASM dialect is indeed done flawlessly, that would allow the code to be built with an open source toolchain (JWASM), while still allowing it to be built from within DOS. Everybody wins. :slightly_smiling_face:
Would FASM dialect perhaps also be an acceptable option? FASM was in part inspired by TASM's IDEAL mode, with some improvements on top of that. See https://board.flatassembler.net/topic.php?t=3197
And since FASM is open source and also available for DOS, there would be no objections in that regard either, right?
@galazwoj The source code contains comments that explain why John Ratcliff chose TASM IDEAL mode over MASM, which concurs with @MobyGamer's points:
;; This file is in the format for Turbo Assembler's IDEAL mode. The ;; IDEAL mode syntax makes a lot more sense for 8086 than the old ;; MASM format. MASM has recently been updated to provide some of the ;; functions that Turbo Assembler has had for a number of years. I prefer ;; to consider Turbo Assembler the standard for 8086 assemblers. ;; IDEAL mode functionality includes true local labels, real data structures, ;; typecasting, automatic argument passing and local memory. ;; Converting any of this code into MASM format is an excercise left for ;; the student.
Link to these comments in the source code: https://github.com/jratcliff63367/digpak/blob/4bc77cb8bdfacea02ee2763bacb8f5fa0f029ac0/DIGPLAY.ASM#L8
So although porting it to MASM dialect would make it compatible with more assemblers, it could apparently also make the source code less structured and more error-prone. But maybe it's not so bad? And otherwise, maybe we could consider FASM dialect instead?
I am not in favor of porting to FASM because FASM requires a 386+ CPU to run, and so we wouldn't be able to assemble the sources on any sub-386 hardware we would be debugging on (or developing new drivers for new devices on). Sub-386 hardware development is important for testing performance of DAC-like output (covox, DSS, bigmouth, Adlib, Tandy 1000, etc.) as those devices require a high rate of interrupts. This is something I wanted to address, actually; John's original code resamples everything for such devices to 9KHz and then outputs as appropriate, but some of those devices can actually output as high as 22KHz even on a stock 8088 system, so it was something I was going to tackle.
I do see that JWASMR.EXE (part of JWASM) claims to run under 808x, but I've never tried it. If that could be verified, that would satisfy the above condition. In general I'm not in favor of translating to MASM just so an open-source assembler can be used (open-source software doesn't have to require open-source tools), but MASM+JWASM would allow development on real hardware.
Just to be factual: The assertion that "TASM IDEAL mode is full of bugs" is false, unless you can provide some proof. There are some documented bugs in TASM, but they're all in TASM32 (requires 386+ to assemble large programs) or in the 32-bit support of real-mode TASM, both of which don't apply when writing new 16-bit real-mode drivers on vintage gear.
The problems I had with TASM IDEAL mode was a) errors with struct declarations, b)incorrect LEA for complex expressions that referenced structures. I think Google has some info at least about case a. So, after initial fascination I gave up TASM, and never returned later.
Great porting work, @galazwoj!
I tested whether I could assemble your MASM port of digpak using JWasm v2.14 from within Linux, and it worked. Apparently JWasm (at least the Linux build) requires hyphens (-) instead of forward slashes (/) for its command-line arguments. Other than that, the arguments are mostly the same.
For instance, to build the Covox Speech Thing driver, I ran JWasm as follows:
jwasm -Cp -I. -Focvxsnd -Flcvxsnd -DBUILD=2 shell.asm
This assembled the sources with 0 warnings and 0 errors. Nice.
Then I used the Open Watcom v2 linker (wlink) to link the resulting object file to a .COM executable as follows:
wlink sys dos com file cvxsnd.o
And then I ran the .com file with DOSBox:
dosbox cvxsnd.com
...And it appeared to load the TSR successfully. At least the driver welcome message and copyright notice appeared, as it usually does when you load a DIGPAK driver. I haven't actually checked whether the driver works, but it's late, and I'll test that another day.
I'm also impressed with how quickly you managed to port this.
After I've merged @MobyGamer's PR, can you rebase your changes on top of that and perhaps also port the missing drivers that are added in his PR? And then offer that in another PR? I'll be happy to merge it.
Thanks. :slightly_smiling_face:
And @MobyGamer, again, I understand your preference for keeping the TASM IDEAL mode syntax, but having the code in a dialect that allows it to be built with more different assemblers than just TASM (at least with MASM, TASM in MASM compatibility mode and JWasm) means that you'll still be able to build the code on a retro/vintage PC from within DOS, while at the same time I'll be able to build the code from within modern operating systems such as Linux and with a completely open-source toolchain as well, meaning that I will be able to offer DIGPAK for inclusion in FreeDOS. Also, I can use my preferred modern IDE and development tools as I try my hand at writing DIGPAK drivers for newer sound devices.
Although everyone is absolutely free to fork this project, I hope we can avoid a fork, because I prefer as many people as possible to contribute to this, instead of having some people work on the IDEAL mode variant and other people on the MASM mode variant, while having to rely on people like @galazwoj to regularly port any improvements from a TASM/IDEAL fork to the MASM repo.
What are both your thoughts on this?
@MobyGamer If I understand your earlier comment correctly, you will find a MASM port acceptable if it's at least tested with MASM 6.x and a binary comparison is done to verify the same output, correct?
In the README of his fork, @galazwoj mentioned that he tested it with MASM 6.1x.
Also, in his v.01 release notes, he mentions that "the resulting binary files are almost 100% as binaries produced by TASM with nop opcodes excluded".
I guess that's about as good at it gets, right? I'm willing to set up a CI/CD script in the repo, triggered by each commit or merge pushed to it, that will automatically try to build the drivers with JWasm and then perform a binary diff of the resulting .COM files with the ones from John's original distribution and show a clear visual report of any differences. That sounds like a fun subproject, actually. :grin:
I hope this will keep you on board and prevent an unnecessary fork. :slightly_smiling_face:
One more caveat that I ran into while assembling the sources in Linux: by default, Linux uses case-sensitive file systems. This will prevent the build from succeeding, unless I convert all files to lower case with a command like this one:
for f in `find`; do git mv -v "$f" "`echo $f | tr '[A-Z]' '[a-z]'`"; done
Also, there appears to be some inconsistency in the use of case between various include directives throughout the sources. After correcting that as well, the sources will build correctly with the commands I documented in my previous comment.
I'll fix the filename cases in this repo and I'll add a JWasm-specific Makefile after I've merged PRs from both @MobyGamer and @galazwoj.
Alternatively, the build might work as-is on Linux filesystems that have casefolding enabled if the sources are cloned into a directory that has the +F (casefolding) attribute set, which would result in files in that directory being treated case-insensitively, just like in Windows. This is a relatively new feature in Linux however, and would likely require a reinstall of the OS, since casefolding cannot be enabled in directories in existing partitions from older Linux installs without having to reformat them first. I haven't tried any of this yet, though.
you will find a MASM port acceptable if it's at least tested with MASM 6.x and a binary comparison is done to verify the same output, correct? In the README of his fork, @galazwoj mentioned that he tested it with MASM 6.1x. Also, in his v.01 release notes, he mentions that "the resulting binary files are almost 100% as binaries produced by TASM with nop opcodes excluded".
I'm neither a blocker nor gatekeeper for the project/repo, so I don't really need to be consulted :-) But, if they assemble with MASM 6.1 and are binary-exact with the exception of extra NOPs (common for fast one-pass assemblers back in the day, to generate fixups later), then I think that's perfectly fine. People can develop (or debug) drivers directly on old hardware and old cards with this.
One nitpick: MASM 6.1 doesn't run on 808x systems; I believe it requires a 386 or higher. 6.0x does, so I'd prefer if that be the version, or any version 6.x that runs on 808x systems, used for testing. Also, there should be a test harness for drivers that exercises all common DIGPAK API functions as a test that the binary drivers aren't broken right after assembling (MASM might use an alternate encoding than TASM for some instructions, which means they won't match from a binary standpoint but could still match from a functionality standpoint).
BTW I have no requirement that MASM be used for development; I'm sure many/most development will be done with NASM. I just want the option of assembling with MASM on real hardware if necessary.
@MobyGamer Yes, TASM generates some opcodes differently, so I included compat.inc file with macros that address this issue.
@MobyGamer I have given this some more thought, and I've come around to what you said earlier.
Perhaps it would indeed be better to keep this repository on John's account as the original source release, make sure it's complete (so add any original source files still missing), perhaps add some documentation to clarify the original build instructions, and then ask John to archive this repository. It will then serve as a historic reference, as well as formal proof that John as the original creator released it under the MIT license. Any further development on the drivers, including porting work, such as @galazwoj's MASM port, can be cloned from this archived repo and proceed as separate forks. It will still be clear where the original code came from, how the code originally was before any recent changes were made, and what the legal status is.
If you agree, then let's focus on any other DIGPAK sources and official documentation that might still be missing.
At that point, I will gladly work to get @galazwoj's DIGPAK fork admitted to the FreeDOS project.
Do you guys agree?
One improvement that I would still like to make to this repository before it is archived, is to preserve any original file modification times using the git-store-meta plugin. I proposed the same thing for Microsoft's MS-DOS repository. Unfortunately, that repo was archived before this proposal was considered.
(If anyone reading this knows an even better way to preserve original modification dates in a Git repository than git-store-meta, please let me know.)
I'm not a git-store-meta expert, nor even all that familiar with git/fork/github/OSS/licensing policies, so I can't comment on those. But I'm glad the original work will remain available for those who want to inspect, or develop, with period-accurate tooling and hardware.