ELKS 0.7.0 wish list

[Mellvik]

There is no better time than the release party for starting a wishlist for the next release, right?

For some of us the list is virtually endless, but some issues are always closer to heart than others. Here's my shortlist:

• Enhance fsck to handle 64M minix filesystems
• Enable loading of more than one ethernet driver in the kernel in order to be able to switch fast between them in the same way as we can switch between slip and eth today. When developing new driver, being able to easily switch to the one that works, to get new kernels downloaded, makes an immense difference in time consumption.
• Find a way to access the ethernet device outside of ktcp in order to get statistics and other info (such as automatic QEMU detection) (the multiple open problem). Maybe an additional dev-entry with a minor device # modulo (say) 128, recognized in the driver as a read-only-ioctl-only access.

My 'pet projects' - in the works, and hopefully finished by 0.7.0:

• Ethernet driver for Etherlink III/3C509 - first cut ready now
• More ethernet drivers - WD/SMC 8003/8013
• ktcp updates to make it work reliably with slow peers (such as ELKS), and corresponding ftp updates
• raw access to block (storage) devices (aka character drivers)

--M

[ghaerr]

Hello @Mellvik,

Thanks for your shortlist. Here's some comments on them:

• Enhance fsck to handle 64M minix filesystems

This won't be possible unless we move to a disk buffering scheme for reading and writing inode data to/from disk, as the size of the information required to fsck large filesystems is way larger than our limited 64K data. Should we go ahead with that, we will need to deal with where to find a "temp" disk when there is only one HD attached, since we can't use that for writing temp data when it is being checked. Another possibility would be to use far main memory for temp storage, but we need to compute the actual amount of memory needed to ensure there that there will be enough to store the data.

• Enable loading of more than one ethernet driver in the kernel in order to be able to switch fast between them in the same way as we can switch between slip and eth today. When developing new driver, being able to easily switch to the one that works, to get new kernels downloaded, makes an immense difference in time consumption.

Wow, I can see that developing a new driver directly on real hardware is a real pain because of the kernel switch problem! Can't we find an emulator to make this job a lot easier?

That said, creating a struct of function pointers to each NIC driver entry point, and storing them in an array, rather than calling them directly by name, would solve this problem fairly easily. I suppose then we could either specify the driver in /bootopts or config (or at runtime network startup?!), or perhaps add another entry point for NIC identification.

• Find a way to access the ethernet device outside of ktcp in order to get statistics and other info (such as automatic QEMU detection) (the multiple open problem). Maybe an additional dev-entry with a minor device # modulo (say) 128, recognized in the driver as a read-only-ioctl-only access.

Since /dev/tcpdev is already set to allow only one open, thus disallowing the problem of multiple opens on /dev/eth from ktcp, we could probably just allow multiple opens on /dev/eth, period. This would quickly solve that problem, and still prohibit ktcp from running twice.

• More ethernet drivers - WD/SMC 8003/8013

Nice! Can you give an overview of the differences between WD 8000, 8003 and 8013?

• raw access to block (storage) devices (aka character drivers)

We've talked about this before, I'm not sure there's much benefit in skipping the block I/O subsystem. Nonetheless, almost all the code for character device access to block devices without buffers is already written. See https://github.com/jbruchon/elks/blob/master/elks/fs/block_dev.c. You will need to modify this file to use a static buffer, rather than calling readbuf, which will keep the data out of the buffer system. All the other synchronization code between kernel file pointers and block devices will remain the same.

I might repeat again: whenever an application issues a read or write of a non-multiple of 512 bytes, a static buffer will have to be used to complete the I/O, then transferred AGAIN into the application buffer. Repeated calls to the driver, say for 128 bytes each or whatever, will entirely repeat the process, causing the block device to reread the data. This means the whole operation will be SLOW SLOW SLOW! A solution to this entire issue may be simply to continue use this same code, but just mark any kernel buffer "unused" immediately after the read or write. Then, it will be as slow as the other implementation, but never use more than a single buffer for character read/writes. One final derivation might be to use only a single marked kernel buffer for any character device I/O, thus giving automatic speed for reads less than 512 bytes, but never using more than a single system buffer.

Perhaps we should talk about the real underlying need for character device drivers. Is the issue just keeping the system buffers from being used up by lots of one-time copying?

Thank you!

[Mellvik]

Thank you @ghaerr!!

Enhance fsck to handle 64M minix filesystems This won't be possible unless we move to a disk buffering scheme for reading and writing inode data to/from disk, as the size of the information required to fsck large filesystems is way larger than our limited 64K data. Should we go ahead with that, we will need to deal with where to find a "temp" disk when there is only one HD attached, since we can't use that for writing temp data when it is being checked. Another possibility would be to use far main memory for temp storage, but we need to compute the actual amount of memory needed to ensure there that there will be enough to store the data.

I do realize that this is a serious challenge, but I still think we need to do something about it. Having uncheckable/unrepairable file systems is not acceptable in the long run, and it's probably better to restrict the fs size to whatever is checkable than to keep it the way it is. That said I'm hoping some extra attention to the issue might trigger some idea that enable the impossible ...

Enable loading of more than one ethernet driver in the kernel in order to be able to switch fast between them in the same way as we can switch between slip and eth today. When developing new driver, being able to easily switch to the one that works, to get new kernels downloaded, makes an immense difference in time consumption. Wow, I can see that developing a new driver directly on real hardware is a real pain because of the kernel switch problem! Can't we find an emulator to make this job a lot easier?

I've been looking for that like forever,. No deal.

That said, creating a struct of function pointers to each NIC driver entry point, and storing them in an array, rather than calling them directly by name, would solve this problem fairly easily. I suppose then we could either specify the driver in /bootopts or config (or at runtime network startup?!), or perhaps add another entry point for NIC identification.

I was hoping you'd say that. Adding a name to the function pointer struct and create an array (and include slip) sounds elegant. Of course there is the complication of tying NIC params in /bootopts to the right interface, but I guess it's more a space issue than a real complication. Let me know if you'd like me to take a stab at this.

Find a way to access the ethernet device outside of ktcp in order to get statistics and other info (such as automatic QEMU detection) (the multiple open problem). Maybe an additional dev-entry with a minor device # modulo (say) 128, recognized in the driver as a read-only-ioctl-only access. Since /dev/tcpdev is already set to allow only one open, thus disallowing the problem of multiple opens on /dev/eth from ktcp, we could probably just allow multiple opens on /dev/eth, period. This would quickly solve that problem, and still prohibit ktcp from running twice.

Hey that sounds good. Let me check that out, maybe start by allowing two /dev/eth opens - to reduce the chance of running astray.

More ethernet drivers - WD/SMC 8003/8013 Nice! Can you give an overview of the differences between WD 8000, 8003 and 8013?

Docs are hard to come by, and I've never yet used any of these cards. Actually I haven't heard of a WD8000, but from what I can tell (reading other drivers), the 003 and 013 have so much in common they should be able to share the same driver. The 013 is definitely 16 bit, I don't know if the 003 is always 8bit or exists in both incarnations. Anyway, the key component is the 8390 chip, which is a well known beast by now.

raw access to block (storage) devices (aka character drivers) We've talked about this before, I'm not sure there's much benefit in skipping the block I/O subsystem. Nonetheless, almost all the code for character device access to block devices without buffers is already written. See https://github.com/jbruchon/elks/blob/master/elks/fs/block_dev.c https://github.com/jbruchon/elks/blob/master/elks/fs/block_dev.c. You will need to modify this file to use a static buffer, rather than calling readbuf, which will keep the data out of the buffer system. All the other synchronization code between kernel file pointers and block devices will remain the same.

I do realize I haven't done a good job at convincing you that this is indeed important. However, there is a good reason why Linux and Unix always had raw disk (and tape) I/O. Admittedly in modern systems these reasons are almost gone because the technical distance from the CPU/memory to the physical storage devices has increased, but ELKS is more like an old V7 Unix on similarly restricted hardware than a modern Linux system. Anyway, the purpose is to bypass the buffers - in our case as it was back then. There are times when we do want to read what's on the disk - for sure and right now, not what's in the cache. And there are times when we don't want IO to 'pollute' the buffer cache and severely slow down everything else. Such as when dd'ing a floppy image to a physical floppy (which I'm doing quite frequently). Or ftp'ing a disk image in our out. I've been down this road before so it's familiar territory. And yes, the data will have to be read from/written to process space buffers. It seems Linux tried to get rid of the raw devices a while back assuming the O_DIRECT would be sufficient. It wasn't. It would (probably) be in our case, but it would require a rewrite of all applications that use it, and that seems like a very bad idea to me. I'll report back when I have something and we can evaluate its usefulness.

[ghaerr]

Having uncheckable/unrepairable file systems is not acceptable in the long run

Well, you make a pretty good point there for solving this problem.

it's probably better to restrict the fs size to whatever is checkable than to keep it the way it is.

That would mean no MINIX hard disks... I think the solution could be to rewrite fsck to use far pointers, and add a new system call to allocate space from main memory. We could also potentially use XMS memory now that I think of it, providing at least ways for PCs with sufficient resources to fsck a larger HD. I've added this to my list.

Adding a name to the function pointer struct and create an array (and include slip) sounds elegant.

We can do that, but SLIP won't be included, as it's not a NIC driver, and implemented only in ktcp. Before we start creating the function pointers, lets brainstorm more on how you think we should switch interfaces (including SLIP).

we could probably just allow multiple opens on /dev/eth, period. Let me check that out, maybe start by allowing two /dev/eth opens

Ok, go ahead and do it, but don't count opens, that's a mess; just remove the open count code altogether. You can then test by adding the open and ioctl in netstat.c to get and display the data. (Don't forget we need to change the direct fmemcpy to verified_memcpy in the driver as well, to protect the kernel). Ktcp should not need any modifications to prevent it from running twice - it should fail on the 2nd /dev/tcpdev open.

Actually I haven't heard of a WD8000, but from what I can tell (reading other drivers), the 003 and 013 have so much in common they should be able to share the same driver.

I thought our existing WD NIC driver was a "WD8000" driver? Are you talking about writing a new driver for the 003 and 013 derivations, or adding them to wd.c?

There are times when we do want to read what's on the disk - for sure and right now, not what's in the cache.

Well, that's the first real good reason I've heard for this feature.

And there are times when we don't want IO to 'pollute' the buffer cache

Ok, another decent reason.

I'll report back when I have something and we can evaluate its usefulness.

As I look at the source for what you want, it's a little complicated but there is unused code which is almost ready to go for what you want - please take a look at the fs/block_dev.c code mentioned above. What is happening is that ELKS is taking advantage of the USE_GETBLK functionality for block device access underneath the filesystems, and that is causing CONFIG_BLK_DEV_CHAR (which implements char devices) to use the block_read and block_write routines, which use the buffered I/O system. The unused routine, block_rw is just what you need, but will need to replace the calls to getblk with ll_rw_blk. After you're studied this code, lets talk more about how to proceed. This will allow implementation of char block devices with a minimum of kernel space used.

[Mellvik]

Thanks @ghaerr, this is fast progress!

Having uncheckable/unrepairable file systems is not acceptable in the long run

Well, you make a pretty good point there for solving this problem.

it's probably better to restrict the fs size to whatever is checkable than to keep it the way it is.

That would mean no MINIX hard disks... I think the solution could be to rewrite fsck to use far pointers, and add a new system call to allocate space from main memory. We could also potentially use XMS memory now that I think of it, providing at least ways for PCs with sufficient resources to fsck a larger HD. I've added this to my list.

I was under the impression that reducing the fs size to 32M would be sufficient to have fsck work OK?

Adding a name to the function pointer struct and create an array (and include slip) sounds elegant.

We can do that, but SLIP won't be included, as it's not a NIC driver, and implemented only in ktcp. Before we start creating the function pointers, lets brainstorm more on how you think we should switch interfaces (including SLIP).

I was thinking net start ne2k … net stop; net start el3 … etc … which obviously would require the config parameters to be pulled from somewhere. Or we could have the default ethernet interface configured in bootopts and have setup manipulate /etc/net.cfg. I agree, more thinking is good on this one.

we could probably just allow multiple opens on /dev/eth, period. Let me check that out, maybe start by allowing two /dev/eth opens

Ok, go ahead and do it, but don't count opens, that's a mess; just remove the open count code altogether. You can then test by adding the open and ioctl in netstat.c to get and display the data. (Don't forget we need to change the direct fmemcpy to verified_memcpy in the driver as well, to protect the kernel).

Yes, I've actually pushed the one already, but it seemed to end up on top of the el3 PR (like last time) so I decided to wait till the el3 has been merged. This time I'm sure it wasn't a branch off a branch, so I don't know why git is doing this.

Ktcp should not need any modifications to prevent it from running twice - it should fail on the 2nd /dev/tcpdev open.

Actually I haven't heard of a WD8000, but from what I can tell (reading other drivers), the 003 and 013 have so much in common they should be able to share the same driver.

I thought our existing WD NIC driver was a "WD8000" driver? Are you talking about writing a new driver for the 003 and 013 derivations, or adding them to wd.c?

AFIK the current driver is a for the wd8003 - and yes, I do expect to start off with what we have and get it to the level of the ne2k driver eventually.

There are times when we do want to read what's on the disk - for sure and right now, not what's in the cache.

Well, that's the first real good reason I've heard for this feature.

And there are times when we don't want IO to 'pollute' the buffer cache

Ok, another decent reason.

I'll report back when I have something and we can evaluate its usefulness.

As I look at the source for what you want, it's a little complicated but there is unused code which is almost ready to go for what you want - please take a look at the fs/block_dev.c code mentioned above. What is happening is that ELKS is taking advantage of the USE_GETBLK functionality for block device access underneath the filesystems, and that is causing CONFIG_BLK_DEV_CHAR (which implements char devices) to use the block_read and block_write routines, which use the buffered I/O system. The unused routine, block_rw is just what you need, but will need to replace the calls to getblk with ll_rw_blk. After you're studied this code, lets talk more about how to proceed. This will allow implementation of char block devices with a minimum of kernel space used.

thanks - that is a really decent starting point. I just pulled that into my 'raw disk' notes. :-)

—M

[ghaerr]

I was under the impression that reducing the fs size to 32M would be sufficient to have fsck work OK?

We should be able to check that quickly: run mkfs 32000 /dev/hda, then run fsck on it... if you have time, play with that, I'll add that to my list for testing also. Been a bit busy and my list is growing quicker than I thought after 0.6.0 already...

net start ne2k … net stop; net start el3 …

Or we could have the default ethernet interface configured in bootopts and have setup manipulate /etc/net.cfg.

We probably want a full dynamic switch between NICs using net start/stop; and having the default in net.cfg also sounds good. I'll think about how that could be made to work.

[Mellvik]

I was under the impression that reducing the fs size to 32M would be sufficient to have fsck work OK?

We should be able to check that quickly: run mkfs 32000 /dev/hda, then run fsck on it... if you have time, play with that, I'll add that to my list for testing also. Been a bit busy and my list is growing quicker than I thought after 0.6.0 already...

No such luck:

mkfs /dev/hda4 32000

10666 inodes 32000 blocks Firstdatazone=342 (342) Zonesize=1024 Maxsize=32768000

fsck /dev/hda4

sys_brk(37) fail: brk f996 over by 2580 bytes sys_brk(37) fail: brk f196 over by 532 bytes sys_brk(37) fail: brk f198 over by 534 bytes sys_brk(37) fail: brk ef98 over by 22 bytes fsck: Unable to allocate buffer for zone count

chmem /bin/fsck

TEXT FTEXT DATA BSS HEAP STACK TOTDATA TOTAL 9424 0 2512 21440 65535 0 65520 74944 /bin/fsck

However, 26M works …

mkfs /dev/hda4 26000

8666 inodes 26000 blocks Firstdatazone=279 (279) Zonesize=1024 Maxsize=26624000

fsck /dev/hda4

Which is a start … maybe yank it to 32M as a first step.

—M

[ghaerr]

No such luck: mkfs /dev/hda4 32000

Good news: I have a faked-up version of fsck running that works for 65M disks. The max data allocation required turns out to be exactly 64K, so all will fit in a single external far segment. I hacked a version of fmalloc which uses the ram disk area at 0xD000:0000 which is otherwise unused, and it works :) For production, a way of allocating far memory from main memory is needed, along with the ability to deallocate it later. I'm working on how to implement that and will then submit the fsck enhancement.

[tyama501]

Hello,

Here is my list for PC-98.

• Detecting numbers of FD drives that actually connected to the PC. This can be done using system memory information.

• Probing 1440/1232 disk in bioshd switching the physical device address to support both format in the same kernel.

• support bootoptions for PC-98

• support PEEK/POKE for BASIC. Maybe this is not just for PC-98

• Try nano-X for PC-98!

Maybe future?

• Getting network card work in C-bus?
• Getting the serial port working to support the serial console?

[ghaerr]

Hello @tyama501,

Nice wish list! Here's my comments:

Detecting numbers of FD drives that actually connected to the PC. This can be done using system memory information.

It sounds like you've got the information required to make this enhancement now.

Probing 1440/1232 disk in bioshd switching the physical device address to support both format in the same kernel.

Will physical probing be the only way for this to work, or can the BIOS know somehow what format disk is in a drive?

support bootoptions for PC-98

I would like to see this done, I have it ready for you. All that is needed is an INT 1Bh sector read function in setup.S. See the bootopts: label and compare what is needed for PC-98 versus the already-coded IBM PC INT 13h function.

support PEEK/POKE for BASIC. Maybe this is not just for PC-98

Great, this would work for all systems and could likely be placed in host.c. I notice that the IN/OUT functions are currently in each seperate host-*.c files, which are duplicated. These can likely be moved to host.c along with the PEEK/POKE functions.

Try nano-X for PC-98!

Cool! You will need to write a new screen driver, to replace drivers/scr_bios.c, for PC-98. Let me know when you more information on how to do that. After doing so, all the nx demo programs should run on PC-98 without modification.

Getting the serial port working to support the serial console?

Is the PC-98 serial UART chip different than the IBM PC? With chip information, we should be able to get this working quickly.

Thank you!

[tyama501]

Hello @ghaerr ,

Thank you for the comments.

Will physical probing be the only way for this to work, or can the BIOS know somehow what format disk is in a drive?

I'm not sure it is the only way but I think physical probing is needed.

You will need to write a new screen driver, to replace drivers/scr_bios.c, for PC-98.

I thought the driver is vgaplan4.c and the ega part of that can be rewritten since PC-98 also uses 4planes (A800, B000, B800, E000) and 16colors (out of 4096) with pallet.

Is the PC-98 serial UART chip different than the IBM PC? With chip information, we should be able to get this working quickly.

It uses Intel 8251 compatible chip.

[ghaerr]

I thought the driver is vgaplan4.c and the ega part of that can be rewritten

Yes, that will work. vgaplan4.c is a sub-driver of scr_bios.c.

It uses Intel 8251 compatible chip.

Great, we should be able to get the serial function working with little changes. You might take a quick look at serial-8250.c, I think with CONFIG_HW_SERIAL_FIFO turned off and rs_probe rewritten, it should work.

[toncho11]

I think it will be interesting to have BASIC with graphics on PC. This will allow for some games and "sell" ELKS better. Games are always something exciting.

[Mellvik]

More thoughts on multiple opens of /dev/eth:

we could probably just allow multiple opens on /dev/eth, period. Let me check that out, maybe start by allowing two /dev/eth opens

Ok, go ahead and do it, but don't count opens, that's a mess; just remove the open count code altogether. You can then test by adding the open and ioctl in netstat.c to get and display the data. (Don't forget we need to change the direct fmemcpy to verified_memcpy in the driver as well, to protect the kernel). Ktcp should not need any modifications to prevent it from running twice - it should fail on the 2nd /dev/tcpdev open.

We need to rethink this because open initializes the device and close (virtually) disconnects it (closes it down). So we're either back to counting the opens or adding another status flag which is set on the second open and reset on the next close. The latter may be dangerous if there are more concurrent opens. The former may be messy but should be safe.

--M

[ghaerr]

We need to rethink this because open initializes the device and close (virtually) disconnects it

Yes, I see.

So we're either back to counting the opens

Ok, that will be no problem - this is done in the TTY drivers, see ntty.c. It is done as follows:

int ttystd_open(struct tty *tty)
{
    /* increment use count, don't init if already open*/
    if (tty->usecount++)
        return 0;
    return tty_allocq(tty, INQ_SIZE, OUTQ_SIZE);
}

void ttystd_release(struct tty *tty)
{
    if (--tty->usecount == 0)
        tty_freeq(tty);
}

The above-style code allows for multiple opens, and only inits/disconnecets when the use count is 0 (which is for the first open). When the final process executes a close, the NIC is shut down.

[Mellvik]

OK; thanks @ghaerr -

I'll follow this model, test it with the ne2k driver and add it to the others when it seems solid.

—M

14. jun. 2022 kl. 17:13 skrev Gregory Haerr @.***>:

We need to rethink this because open initializes the device and close (virtually) disconnects it

Yes, I see.

So we're either back to counting the opens

Ok, that will be no problem - this is done in the TTY drivers, see ntty.c. It is done as follows:

int ttystd_open(struct tty tty) { / increment use count, don't init if already open*/ if (tty->usecount++) return 0; return tty_allocq(tty, INQ_SIZE, OUTQ_SIZE); }

void ttystd_release(struct tty *tty) { if (--tty->usecount == 0) tty_freeq(tty); } The above-style code allows for multiple opens, and only inits/disconnecets when the use count is 0 (which is for the first open). When the final process executes a close, the NIC is shut down.

— Reply to this email directly, view it on GitHub https://github.com/jbruchon/elks/issues/1312#issuecomment-1155336573, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3WGOFFTBRI52MAD5VHQO3VPCOSRANCNFSM5YFRDU3A. You are receiving this because you were mentioned.

[cocus]

• Ethernet driver for Etherlink III/3C509 - first cut ready now

I'd like to see more of this, since I have a 3c509b-tpo which I would also like to use on the 8bit ISA interface I have. I know it's possible, it's been already done for 8088/8086 CPUs (as shown here)

• support PEEK/POKE for BASIC. Maybe this is not just for PC-98

That would be nice to have, although that's quite dangerous for the OS stability, since you could (in theory) modify any part of the memory. But count me in on this request as well.

Now, from my side, I think I have just one wish item:

• support MBR partitions on ssd devices (i.e. SD cards for my SBC) so the SD card contains a partition table and not the FS directly on it

[ghaerr]

although that's quite dangerous for the OS stability, since you could (in theory) modify any part of the memory.

Yes, I suppose so. I wonder what real use POKE would have running from BASIC?

But count me in on this request as well.

Me too. What good's an interpreter if you can't have fun playing with it lol :)

support MBR partitions on ssd devices (i.e. SD cards for my SBC) so the SD card contains a partition table and not the FS directly on it

This would almost work now by just applying the MBR onto the device's first block, I think? Is the problem that we don't have minor device numbering scheme implemented under /dev/ssd[0-3] in order to access each partition (and use its calculated sector offset)?

[cocus]

Yes, I suppose so. I wonder what real use POKE would have running from BASIC?

Well, I think it's good for development and general "mocking around". In the good old days where the computer just ran the basic interpreter, you did want to have that so you can "escape" basic somehow and take control of the machine without the hassle of programming in assembly. I assume in our case it would be to access memory-mapped devices easily, like prototyping a video card driver, or doing a demo.

This would almost work now by just applying the MBR onto the device's first block, I think? Is the problem that we don't have minor device numbering scheme implemented under /dev/ssd[0-3] in order to access each partition (and use its calculated sector offset)?

Probably that was the case, yes. It's been a long time since we begun working on the SD driver!

[ghaerr]

Probably that was the case, yes. It's been a long time since we begun working on the SD driver!

I'll add this to my list of things to look at. Let me know when you're thinking you might want to jump back in to help test or develop this. I'll get some ideas first on how to get it done in the meantime.

[ghaerr]

Hello @Mellvik,

Enable loading of more than one ethernet driver in the kernel in order to be able to switch fast between them

Rather than the way I initially suggested, that of having a set of function pointers to an "active" NIC /dev/eth driver, there's another way that might work better: have each NIC have its own /dev/xxx character device. That is, /dev/3c509, /dev/wd8003, etc. In this way, changing "drivers"/NICs amounts to just having ktcp open the desired character device, and the image build process could link in all configured drivers, and create the appropriate /dev entry points.

Each driver would initialize only if/when it's opened from ktcp. Adding "net start eth 3c509" or something similar could just substitute the appropriate char NIC device instead of /dev/eth, passed as a parameter to ktcp for opening. A new /dev entry would be required to be created for each new NIC, but that shouldn't be a problem.

Currently, each NIC driver attempts to register its IRQ at boot time on the init call, which won't work. Instead, the init routine would just display a driver name as being available, and do nothing else until open is called. The other part of the init routine would be moved into open. When ktcp closed the driver, its close routine would free the requested IRQ, possibly to be used by another NIC (or serial) driver.

Another benefit would be multi-homing (someday).

A disadvantage is that the kernel "eth: ..." message would be delayed until and/or displayed each time ktcp was started, but I'm not sure that's bad.

If you like this idea, the first step would be to move most of the code from the init routine, and instead call it at open time; at close time, the driver would also have call a free_irq routine to free the IRQ (and ensure the device was fully disabled). After getting that working for the NE2K, WD and EL3 drivers, we could combine them into a single build and make the changes to ktcp and /etc/eth character devices. I could take a first pass at this for ne2k.c and QEMU, and you could flesh it out with the remaining drivers. After that, we write a new master ethernet character device handler that hands over the open calls to the appropriate linked-in device driver according to minor device number.

Thoughts?

[ghaerr]

Regarding PR #1338:

the first step would be to move most of the code from the init routine, and instead call it at open time;

Only the request_irq code has been moved from init to open; the rest of the init code needs to be moved for each driver and then tested on real hardware. This shouldn't be a big deal if no code is changed, only moved.

at close time, the driver would also have call a free_irq routine to free the IRQ (and ensure the device was fully disabled).

Done for all three drivers.

After getting that working for the NE2K, WD and EL3 drivers, we could combine them into a single build and make the changes to ktcp and /etc/eth character devices.

On hold until init code moved for all three drivers.

I could take a first pass at this for ne2k.c and QEMU, and you could flesh it out with the remaining drivers.

Done for all drivers, except init code.

After that, we write a new master ethernet character device handler that hands over the open calls to the appropriate linked-in device driver according to minor device number.

Master ethernet driver written in #1338, next step will be to create /dev/3c509 etc on disk and use minor number to pass I/O calls to subdriver. We can use the following names/minor numbers:

mknod /dev/ne2k c 9 0
mknod /dev/wd8003 c 9 1
mknod /dev/3c509 c 9 2

[Mellvik]

Wow @ghaerr - that was fast (again).

Thank you for diving into this - in a way that is significantly more ambitious than I had envisioned, and will deliver a more general solution. Having a device entry per interface is very important in that regard.

I'm having reservations about the suggested initialization scheme though. It is easy to see the advantage of being able to 'share' the IRQ, it being the most precious resource in this setting. Certainly - I didn't envision that, and I can see that the proposed regime would most likely work. The scheme has some disadvantages though — in addition to (possibly) making the change more complicated:

• While the proposed IRQ regime might work, there may be cards that somehow pollute the IRQ even if they aren't initialized. This can probably be prevented in the dev_init code after probing, but still seems risky to have several cards set to the same IRQ on the (ancient) ISA bus.

• The IRQ can be 'shared' this way, but the IO port and shared memory addresses can not (only the EL3 interfaces have the very desirable ability to set IRQ and IO PORT at init/open time). So there will still have to be careful considerations as to the hardware config when adding more interfaces.

• The setting we're addressing (at this time) is development, not production. It is not unreasonable to assume that such an environment has an extra IRQ available.

• For future multihoming to work, we would have to reverse the regime back to full initialization at boot time. (and if someone wanted to experiment with multihoming, such regime would allow it).

• I may be wrong, but this seems simpler to me.

So my suggestion is to keep the initialization regime as is, and require that interfaces have their unique IRQ. It seems simpler and more future proof.

FWIW - During EL3 driver development, I had 2 interfaces in the machine all the time, and switched between kernels. It's an AT class machine and I don't think network development (or multihoming) on 'lower' class machines is viable. It was slow, but workable.

Thanks again, this is a very welcome improvement.

—M



Regarding PR #1338 https://github.com/jbruchon/elks/pull/1338:

the first step would be to move most of the code from the init routine, and instead call it at open time;

Only the request_irq code has been moved from init to open; the rest of the init code needs to be moved for each driver and then tested on real hardware. This shouldn't be a big deal if no code is changed, only moved.

at close time, the driver would also have call a free_irq routine to free the IRQ (and ensure the device was fully disabled).

Done for all three drivers.

After getting that working for the NE2K, WD and EL3 drivers, we could combine them into a single build and make the changes to ktcp and /etc/eth character devices.

On hold until init code moved for all three drivers.

I could take a first pass at this for ne2k.c and QEMU, and you could flesh it out with the remaining drivers.

Done for all drivers, except init code.

After that, we write a new master ethernet character device handler that hands over the open calls to the appropriate linked-in device driver according to minor device number.

Master ethernet driver written in #1338 https://github.com/jbruchon/elks/pull/1338, next step will be to create /dev/3c509 etc on disk and use minor number to pass I/O calls to subdriver. We can use the following names/minor numbers:

mknod /dev/ne2k c 9 0 mknod /dev/wd8003 c 9 1 mknod /dev/3c509 c 9 2 — Reply to this email directly, view it on GitHub https://github.com/jbruchon/elks/issues/1312#issuecomment-1159936979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3WGODU7FHSZGHCHY5XW2TVP7S2HANCNFSM5YFRDU3A. You are receiving this because you were mentioned.

[ghaerr]

Hello @Mellvik,

I'm having reservations about the suggested initialization scheme though.

I think I miscommunicated the issue with IRQs - they won't need to be shared between NICs. The subdriver will request it's IRQ on open, and free the IRQ on close, using the dynamic IRQ allocation mechanism added a while back. This would allow for other NIC drivers to (re)use the same or different IRQ. In the future, this mechanism could allow for serial drivers to also request their IRQ on open, allowing a serial IRQ to be used by a NIC driver when the serial port isn't been used.

With regards to setting netport=, netirq= and netram= in /bootopts, in order to support our current three NICs compiled in together, I think we're going to have to move to your suggestion that the Linux kernel uses - placing all three parameters on the same line. This will take the least space in /bootopts and allow for all to be included at once. I'm thinking of using the same names for the variables as the new /dev names for the devices, something like:

ne2k=12,0x300
wd8003=2,0x240,0xCE00
3c509=9,0x330

This would entirely replace netport=, netirq= and netram=. The NIC itself would be selected using net=, but instead of net=eth, one would use the NIC name:

net=ne2k
net=wd8003 (etc)

Likewise net start eth would be replaced with net start NICname:

net start 3c509
net stop
net start wd8003 (etc)

I would like to move forward with this scheme using multiple successive PRs to make all this work, but only for a singly-compiled-in NIC driver for the moment (until the init code is moved over to open, which you should probably do, since moving it could break networking, which I'm trying not to do).

So my suggestion is to keep the initialization regime as is, and require that interfaces have their unique IRQ.

Yes on both - except we move the initialization code from the drv_init boot time function to the open function. We don't really need to execute init code at boot, for all the compiled-in drivers; we just execute the driver init code when the network is (re)started. For NIC IRQs, most cards will have their own IRQ, but it would be possible to use the same IRQ, if the hardware permits it, since the init code only occurs at open time, and there will only be one NIC opened at a time, ever.

[Mellvik]

Thank you @ghaerr, I like the speed of this!

I'm having reservations about the suggested initialization scheme though.

I think I miscommunicated the issue with IRQs - they won't need to be shared between NICs. The subdriver will request it's IRQ on open, and free the IRQ on close, using the dynamic IRQ allocation mechanism added a while back. This would allow for other NIC drivers to (re)use the same or different IRQ. In the future, this mechanism could allow for serial drivers to also request their IRQ on open, allowing a serial IRQ to be used by a NIC driver when the serial port isn't been used.

yes, maybe I misunderstood the intention here. Still, one of my points was that the ISA bus doesn't lend itself well to having two cards installed that respond to the same interrupt, even if only one is (software-)enabled at a given time.

With regards to setting netport=, netirq= and netram= in /bootopts, in order to support our current three NICs compiled in together, I think we're going to have to move to your suggestion that the Linux kernel uses - placing all three parameters on the same line. This will take the least space in /bootopts and allow for all to be included at once. I'm thinking of using the same names for the variables as the new /dev names for the devices, something like:

ne2k=12,0x300 wd8003=2,0x240,0xCE00 3c509=9,0x330 This would entirely replace netport=, netirq= and netram=.

Yes, I very much agree with this!! :-)

I would like to move forward with this scheme on this outstanding PR #1338 https://github.com/jbruchon/elks/pull/1338, so that on its commit, all this would work, but only for a singly-compiled-in NIC driver for the moment (until the init code is moved over to open, which you should probably do, since moving it could break networking, which I'm trying not to do).

So my suggestion is to keep the initialization regime as is, and require that interfaces have their unique IRQ.

Yes on both - except we move the initialization code from the drv_init boot time function to the open function. We don't really need to execute init code at boot, for all the compiled-in drivers; we just execute the driver init code when the network is (re)started. For NIC IRQs, most cards will have their own IRQ, but it would be possible to use the same IRQ, if the hardware permits it, since the init code only occurs at open time, and there will only be one NIC opened at a time, ever.

As I pointed out in my previous post, I don't think this is beneficial. If we conclude that the ability to assign/deassign the IRQ isn't useful, because a particular IRQ would only be used by one piece of hardware anyway, I don't see that this is gaining anything. I guess what I'm also saying is I like to see what we have in terms of working hardware at boot time, whether it's enabled (network wise) or not. Then the dev_open wiould be akin to a ifconfig /dev/nek2 down ifconfig /dev/3C509 up

Having all devices initialized at boot will save time during development by revealing immediately which ones probe/initialize correctly instead of having to manually check each one using ktcp.

—M

[ghaerr]

I don't think this is beneficial. If we conclude that the ability to assign/deassign the IRQ isn't useful, because a particular IRQ would only be used by one piece of hardware anyway

Ok, that's fine, you can experiment with it. Having the request_irq/free_irq in open doesn't hurt anything, as interrupts are never enabled at init time anyways. I have the next PR almost ready to go, where we will use the NICname to allow you to test. I am thinking then of adding multiple NICs to be allowed to be compiled into the kernel immediately after that, and you'll be able to switch between NICs and experiment.

In v0.7.0, we will distribute a kernel with all NIC drivers included. That kernel won't grab the IRQs for each NIC at boot time, since there may be other devices that use the same IRQ.

Having all devices initialized at boot will save time during development by revealing immediately which ones probe/initialize correctly instead of having to manually check each one using ktcp.

Ok - again, you can experiment with it and determine whether we need to move code out of init. I just though it was cleaner, but you make a valid point. If you determine that the init code should stay in at boot time, I'm ok with it.

Then the dev_open wiould be akin to a ifconfig /dev/nek2 down ifconfig /dev/3C509 up

That's pretty much what I have working now in the next PR, except we'll just use net start ne2k, net stop, net start 3c509. The default startup NIC will be set using net=ne2k in /bootopts.

[Mellvik]

OK, sounds good. I'm quite looking forward to it.

I was pondering if expanding the eth struct would be good for placing the NIC params, but given the speed at which you're moving, I'll be back with comments after the next round :-)

20. jun. 2022 kl. 18:52 skrev Gregory Haerr @.***>:

I don't think this is beneficial. If we conclude that the ability to assign/deassign the IRQ isn't useful, because a particular IRQ would only be used by one piece of hardware anyway

Ok, that's fine, you can experiment with it. Having the request_irq/free_irq in open doesn't hurt anything, as interrupts are never enabled at init time anyways. I have the next PR almost ready to go, where we will use the NICname to allow you to test. I am thinking then of adding multiple NICs to be allowed to be compiled into the kernel immediately after that, and you'll be able to switch between NICs and experiment.

In v0.7.0, we will distribute a kernel with all NIC drivers included. That kernel won't grab the IRQs for each NIC at boot time, since there may be other devices that use the same IRQ.

Having all devices initialized at boot will save time during development by revealing immediately which ones probe/initialize correctly instead of having to manually check each one using ktcp.

Ok - again, you can experiment with it and determine whether we need to move code out of init. I just though it was cleaner, but you make a valid point. If you determine that the init code should stay in at boot time, I'm ok with it.

Then the dev_open wiould be akin to a ifconfig /dev/nek2 down ifconfig /dev/3C509 up

That's pretty much what I have working now in the next PR, except we'll just use net start ne2k, net stop, net start 3c509. The default startup NIC will be set using net=ne2k in /bootopts.

Sounds excellent!

—m

[ghaerr]

I was pondering if expanding the eth struct would be good for placing the NIC params

You mean netif_stat? Good idea, except the kernel init code will need three static versions of them in order to parse /boototps, and the struct is large. I'll create another struct for these but keep them in netstat.h.

Thanks!

[Mellvik]

NO, I was referring to the struct in drivers/char/eth.c where you keep the file ops, one per driver.

Then again I don't have the big picture at this time.

-M

20. jun. 2022 kl. 19:02 skrev Gregory Haerr @.***>:

I was pondering if expanding the eth struct would be good for placing the NIC params

You mean netif_stat? Good idea, except the kernel init code will need three static versions of them in order to parse /boototps, and the struct is large. I'll create another struct for these but keep them in netstat.h.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/jbruchon/elks/issues/1312#issuecomment-1160671310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3WGOHPG7M25TA776ZYNU3VQCP3BANCNFSM5YFRDU3A. You are receiving this because you were mentioned.

[ghaerr]

I don't have the big picture at this time.

The bigger picture is that instead of having each NIC driver register itself to be /dev/eth (at 9, 0), a new "master" character device eth.c was created that allows for seperate /dev/ne2k, wd8003 etc devices (at 9, 0 for ne2k, 9, 1 for wd8003 etc). When one of those devices is opened, it just passes all open/read/write/select calls to the original driver through a file_operations table, that's all there is to it. The drivers themselves act exactly the same.

Then, net start and ktcp were modified to open the appropriate device, rather than always /dev/eth, which no longer exists. Finally, /bootopts parsing was changed to store the config values in an array, based on the same numbering as the eth driver minor character number.

[tyama501]

Hello @ghaerr , @cocus ,

I have done 3 entries of my list https://github.com/jbruchon/elks/issues/1312#issuecomment-1152870664 The next one is to add PEEK/POKE in basic.

Do you think some protection is needed for POKE to system memory? It seems there is "SETUP_DATA" definition in elks/include/linuxmt/config.h all for CONFIG_ROM_CODE, IBM_PC, 8018X, PC98, so one idea is protecting under this segment. (There is no complete protection on real mode but...)

I would like to have this PEEK/POKE feature before starting to modify nano-X driver since it can be used to debug the driver.

Thank you!

[ghaerr]

Hello @tyama501,

Do you think some protection is needed for POKE to system memory?

No, I think its usefulness would be greatly decreased by adding such a thing, and also would be hard to know when protection kicked in. In addition to SETUP_DATA, there would also have be KERNEL_CODE, KERNEL_DATA, and all of low memory.

Maybe this is not just for PC-98

You should be able to write just a single portable implementation that will work for IBM PC, PC-98 and 8018x. It could be written in C, use __far pointers and _MK_FP, and reside in host.c (ifdef'd with __ia16 __ so that can still be compiled on Linux/macOS host).

Thank you!

[tyama501]

I tried code like this but it seems print shows word. Do you think peek should be word?

unsigned char host_peekb(int offset, int segment) { unsigned char __far *peek = 0; #if ia16 peek = _MK_FP(segment,offset); #endif return *peek; }

[ghaerr]

Hello @tyama501,

Is it possible that "unsigned char host_peekb(...)" is not declared in basic.c? I am guessing perhaps result is being sign-extended somewhere.

I think it is better to have functions (and arguments) use int, rather than unsigned char to avoid that potential problem. Perhaps try that to see if that fixes this problem. Also try just PRINT A to see more.

Also, "#if ia16" should be "#ifdef __ia16 __" (no space after 6), and no need to initialize peek. The non-ELKS (host) code cannot contain any __far pointers, i.e. entire function interior needs ifdef.

Thank you!

[ghaerr]

Do you think peek should be word?

No, it seems that Sinclair BASIC uses 8-bit value for peek and poke; we should probably stay compatible.

[tyama501]

Thank you @ghaerr

I will try using int. BTW line 163 of host.c already has "if \ _ia16_" not ifdef.

#if __ia16__ /* replacement fread to fix fgets not returning ferror/errno properly on SIGINT*/ size_t fread(void *buf, size_t size, size_t nelm, FILE *fp)

[ghaerr]

#if __ia16 __

Ok, I see host.c has it both ways, ifdef and if. Let's use #if, and you're welcome to change the ifdef __ia16 __ in host.c to #if in your PR.

[Mellvik]

A lot of heavy ELKS usage brings up an old 'friend' - mv, or rather, the rename system call, so I'm adding it to my wishlist.

An old friend because we've discussed it so many times before - and @ghaerr fixing problems as they've appeared. Still, there's this basic problem that renaming directories locally on Minix file systems effectively is copying, for a number of technical reasons. Which takes a long time with large directories - on floppies.

Even on fast hard disks this is becoming painful. Here's my scenario:

• A running ELKS system
• FTP over a new copy of /bin, say - tar-extracted to /tmp/bin
• The easy way would be mv /bin /bin.old; mv /tmp/bin /bin; reboot (and remove the old copy after reboot) which should be very fast indeed, but 1) fails, and 2) takes a long time if it doesn't fail.
• The failure is caused by symlink and is fixable, but the fundamental problem - copying files instead of renaming a- is still there.

What do you say @ghaerr, do you think this a feasible project?

--M

[ghaerr]

Hello @Mellvik,

do you think this a feasible project?

I'd say that even though it's not very "feasible", it still needs to be done, given the negative usability of not having a working directory rename function in ELKS.

The failure is caused by symlink and is fixable

You're talking about the current failure of mv trying to follow symlinks when forced to perform the copy workaround? We should fix that, so that by default symlinks are not followed. (There's a standard option to turn that on or off, I think).

While a full, proper solution is a lot of work, since it needs to be implemented on all filesystem types, perhaps a first pass on MINIX would be a step in the right direction. I am wondering out loud whether there might be significant issues with a directory rename for any open files underneath or within that directory, especially the current directory itself and whether getcwd would still work or not if the current directory was renamed. Other than that, renaming any directory or file never changes the inode number, so should not produce a side effect the kernel needs to worry about in other places.

I've added this to my list, and will consider how we might get this working.

Thank you!

[Mellvik]

Thank you @ghaerr, and I agree with you, this has to start with (and possibly be limited to) minix fs. Actually - I'm not sure whether the DOS MOVE command will copy or just rename the dir if possible.

The failure is caused by symlink and is fixable

You're talking about the current failure of mv trying to follow symlinks when forced to perform the copy workaround? We should fix that, so that by default symlinks are not followed. (There's a standard option to turn that on or off, I think).

Yes, this is the one:

elks15# mv bin bin.x bin/httpget: can't move directory or symlink elks15#

I am wondering out loud whether there might be significant issues with a directory rename for any open files underneath or within that directory, especially the current directory itself and whether getcwd would still work or not if the current directory was renamed. Other than that, renaming any directory or file never changes the inode number, so should not produce a side effect the kernel needs to worry about in other places.

I think this is an unavoidable side effect that has always been there in Unix like filesystems. Occasionally we will get a message like 'Invalid current directory, enter new >'.

I just checked - Linux bash caches the original name and uses it until you get out of there. BSD 2.11 csh and Bourne shell reports the new name immediately:

pwd

/

cd tmp

mkdir xx

cd xx

touch 1 2 3 4 5 6

(cd ..; mv xx yy; pwd)

/tmp

pwd

/tmp/yy

I'm guessing that's what would happen in the ELKS case too. I think it's a go, and handle problems when/if they arrive.

—M