archinstall

archinstall copied to clipboard

Updated 21th of July Code is ready to be tested and bugs and new features/checks are kindly accepted. There are slight changes at the UI from what is shown the diskmanager script now fully interfaces with guided and all its features

Updated 8th of april This is an ongoing effort to change the user interface of the disk management part of Archinstall. We present it as a PR so you can check it, and propose as many changes/enhancements you feel like. For the time being it is a standalone program (a script called diskmanager.py). Expected to run at master ~with no additional changes.~ Current version still has a lot of limitations (see below)

executing

sudo python -m archinstall --script diskmanager (--disk_layout json_file) (--long_form) The arguments:

• --disk_layouts reads an existing disk layout file
• --dm_long_form generates screens with more data
• --dm_no_add_menu when adding a partition if set, the user will be asked a series of questions, otherwise a menu will be shown

and all the arguments relevant to archinstall, at least regarding disk management

If a layout is given it will go directly to the main screen

otherwise

If options 2 or 3 are selected, yo will be prompted for disk selection

And

if you select a disk

if you select a partition

if you select empty space, you will be asked for adding a partition, item by item, ending at the general partition screen

Development Phases

We have planned an iterative process, with five phases

• [X] I User Interface with full navigation. Complete in the sense that all UI elements are in place and are navigable
• [X] II Full UI implementation and dependencies in the workflow
• [X] III Generation of json files which work. Only basic functionality (no -very little- "backoffice changes") Due to debugging needs, we have made this step earlier than expected. Now we generate files in the LOG_PATH directory under the name layout_diskmanager.json
• [x] IV Full standalone implementation. (8th april) Without "backoffice" enhacements (a new only_hd )
• [ ] V General integration in the the tool.

Some new functionality may be left for a second iteration. The target is to be at least as functional as today's implementation Up to the 3rd phase, there is no guarantee that any result (json file) from this program can be feed to Archinstall. Even at this level known incompatibilities and peculiarities may be present. See below Up to the 4th phase, the development will be independent from the main application. Some bug solving and enhacements to the general code may be upstreamed, as long they're compatible with the actual implementation (f.i. subvolume handling has already been merged)

The order is merely a planning affair. Since the 5th, i'm working into real execution against disks, so it is the IV phase. And things go well ... Phase II, due to the need of collecting outside documentation has been put a bit on the backburner

As of 30 of March we feel the I phase is up to public display and critique. As of 5 of April we steeped forward to Phase III. Now usable disk_layout json files are generated, and work (albeit idiosincratically). ~No support for delete partitions yet~

Testing

As usual you can test it from my own repository or better yet, using this PR as a local branch via the commands (those strings between <> are names you can/should update to your need)

git remote add <upstream> git://github.com/archlinux/archinstall.git
git fetch <upstream> pull/1053/head:<BRANCH_NAME>

Known incompatibilities

• [x] Until there is a successful solution to the problems initially addressed in PRs #794 and #854 (that size in disk_layouts really means end sector) our implementations are incompatible, as size has for us its semantic value, and we -still- don't use any end reference Updated We now generate a json which should be compatible to the current ones, only space is strictly in sector units
• [ ] We aren't still able to process a preexisting file, which doesn't imply a full disk wipe. We expect that it will be an algoritmically complex situation, and suspect it will be the last issue standing
• [x] We are ready for/need real partition deletions, but it probably needs changes at Archinstall
• [ ] All our geometrical assumptions are for GPT drives. We need to gather more info for other cases
• [x ] All our physical variables are measured in sectors. The output needs to be more flexible (as the input is now).

• Updated* Everything is in place for this. Only the json is still not created with size other than sectors to simplify debugging

• [x] We still struggle with entries with % at some places Updated We have managed to use percentages in the most important place. There are two kinds of percentages, one relative from the start of the partition till the end of the disk (which is the one will be written in the json file), and another strictly for the add partition task, which is form the start of the partition till the end of the free gap where it will be located. Translation between both is managed internally. Why? Only the first is transportable between different disks, but is unwieldly for actual space allocation.

For lack of better words, this would be really amazing and a welcome change/upgrade!

We have been working in the definition of space of the partitions. This is how it looks now

Note the different percentages. It's intentional

Help needed ! I need a list of all possible restrictions you can think about partition management (size constrains for GPT/MBR disks, constrains due to UEFI/BIOS, what can and can not be encrypted, how many boot partitions in a disk, minimum partition size, .... Let's see what can be implemented here.

I've added the possibility of generating disk_layout files -before i had planned- Was needed in order to check compatibility. It is generated at the usual directory with the name layout_diskmanager.json. To be used (beware, wildy experimental, can eat your pets) you need a --harddrives argument in the usual scripts (only_hd is at your service) Test have shown some minor anomalies, already changed. It looks promising

I've integrated only_hd into diskmanager, so we can now generate a REAL disk structure directly from this module (i.e we are now in phase IV). Testing was otherwise too tedious, and i need real interaction. Is very rough, still WARNING this is highly experimental. I you use it in something resembling reality, do at your own risk. And beware, it can eat your kids ;-)

@torxed If I get it correctly, once you put an encryption password ALL the non boot partitions are encrypted. This could be a time for changing the behaviour if desired, or to be able to have different passwords for different partitions. Think about it, and I can add it to my list of things to do. BTW. I found why archinstall generates keyfiles where it shouldn't. As usual, has to do with btrfs ...

Now you can delete partitions from inside diskmanager.py. Let's hope it doesn't breaks many installations ... I'll be on something like vacations for all of the Holy Week, so don't expect further updates (barring bad weather) during this time. A good moment to download and test

I will attempt to continue development on this for a while after talking with wllacer who will be taking a well deserved break. I'd love to see this go in to v2.4.0 although time is short and will do my best in Wllacer's absence, and would love Wllacer's feedback once back! :)

For lack of better words, this would be really amazing and a welcome change/upgrade!

Seconding, will be a huge improvement! esp for new users who are both more likely to dual boot and less likely to be familiar with traditional arch setup/partitioning steps :heart:

Merged changes from master (two months worth of them) Seems diskmanager has picked a lot of bit rot since them (good for the project, not for me) and it is broken for now. I'll try to put it in a workable state ASAP. If you have the need to test its contents, download and remove commit https://github.com/archlinux/archinstall/pull/1053/commits/ab76909e412b7a213bfeff94fffb71a3578e4bd1, will get you a very obsolete archinstall but a working diskmanager user interface :-(

Welcome back! Unfortunately we had a lot of changes to the BTRFS and disk operations due to some critical bugs. I tried to make the changes as minimal as possible but it was really hard :/ So I felt I had to bump to v2.5.0 instead of v2.4.3 as intended.

I did see. I can make use of some of them ... I'm stuck, though,with one or two of the changes made at list_manager , but once i get again on speed, don't seem too serious

@svartkanin Daniel, in which PR did you introduced and documented the last changes to list_manager and introduced the MenuSelection class ? Two months absence have put me a bit off-track

@wllacer ohh I didn't know this was started already... I'm actually starting to finalize this in another branch https://github.com/archlinux/archinstall/compare/master...svartkanin:rework-partition-management which introduces the ListManager for the partitioning

Apologies, I was not aware that this already in progress otherwise I would have build on this

@wllacer I extracted the latest changes for the list manager in it's own PR https://github.com/archlinux/archinstall/pull/1331. The point of them are that the list manger is more self-explanatory and less error prone now.

The other branch https://github.com/archlinux/archinstall/compare/master...svartkanin:rework-partition-management does partly include the implementation for the new partition management, if you prefer to go ahead with yours code you make sure to utilize the new functions, especially for creating tables FormattedOutput.as_table to not create too much duplicate functionality, it seems that there are quite a few places all over the code where things are transformed, I intend to clean that up in hopefully the near future :)

Those changes are now merged @svartkanin :)

Damn that was quick :D

@svartkanin I'll have a close look at your https://github.com/archlinux/archinstall/compare/master...svartkanin:rework-partition-management branch. A lot less cluttered than mine for a start, everything at its proper place , and less "prototype quality"... Will see how i can reuse it

Coolio, something that I realized and didn't get around to yet was that the returned type of the original partitioning function is a plain Dict. I have already used the actual Partition objects in the implementation though which runs into some conflicts though, hence why I introduced a new type VirtualPartition, which is the user created one.

This should probably be simplified since they can't be handled equally in the ListManager. One idea is to expend the Partition object to be able to handle virtual partitions or to transpose the actual Partition into VirtualPartitions bedore initializing the PartitionManager.

Coolio, something that I realized and didn't get around to yet was that the returned type of the original partitioning function is a plain Dict. I have already used the actual Partition objects in the implementation though which runs into some conflicts though, hence why I introduced a new type VirtualPartition, which is the user created one.

This should probably be simplified since they can't be handled equally in the ListManager. One idea is to expend the Partition object to be able to handle virtual partitions or to transpose the actual Partition into VirtualPartitions bedore initializing the PartitionManager.

That is one of the things I wished we solved some how, so I'm happy to hear we have. Working with user defined partitions as dicts have caused so many issues ^^

Most of UI works now with the new ListManager and Subvolume, but still some functions missing at partition level, and code is still patchy. For the time being is behaves as if dry_run were set

@svartkanin Still haven't had the time to play with your partition handler. Just a note, i can not use Formatted_output because my partition list screen is heterogeneous ( disks and partitions on the same screen. Perhaps even subvolumes one day) and that function is for homogeneous objects.

The diskmanager.py seems to be really big, could we put that into it's own module and split the file into smaller parts?

The last flake8 error seems to be related to code i didn't touch ¿? Will check Otherwise, now diskmanager.py seems to be bug compatible with the status i left it before my pause. I can now have some productive work. I will, though spend some time exploring the use of dataclasses here (@svartkanin usage made me interested). In the meantime i need input of which disk/partition related restrictions you can find interesting to include

@wllacer I probably already saw the cleanup PR here https://github.com/archlinux/archinstall/pull/1333. Does this new disk manager essentially replace this existing Partition class or is it going to live alongside it? The PR was meant to address some bugs, obviously not all of them since you had raised an additional one recently for loop device :)

I was just wondering if your new manager is going to replace all of this, we may want to consider integrating these changes, but it's rather difficult since nothing is merged... Just throwing it out there for now :)

@wllacer I probably already saw the cleanup PR here #1333. Does this new disk manager essentially replace this existing Partition class or is it going to live alongside it? The PR was meant to address some bugs, obviously not all of them since you had raised an additional one recently for loop device :)

My original idea behind this PR was to design an alternative presentation layer for all storage related function, when I realized there are a lot of reported issues which are mostly misunderstandings around the current interface. Once I got the first version of list_manager a rewrite was a low hanging fruit ... till "real life" made me stop. One of the limits I imposed was not to change anything in the "backoffice" layer (more on this later). In practice, I stop once the harddrives and disk_layout arguments are filled, leaving the remainder to current code. Thus I don't use/need the Partition class (well, once to get actual information from an existing partition, i.e. the attributes). You can safely (for me) work on this. Even use the current interface as a starting point, as you do in your rework-partition branch. I also felt the Partition class needs a rethink. My impresion -outside the implementation details- is that the whole class model we have for storage needs a rethink to allow us to grow functionaly. I have somewhere some notes of what I thought was needed. When I find them i'll forward it to the community. There is (from the implementation pov) another hurdle: we depend on the output of existing command line utilities. It's a pity we can not make use of pyparted (at Arch is only an AUR package) and I suspect to get it on the installation CD is a no go (@torxed, what do you think about?)

Yes things are very tightly coupled regarding storage of actual settings which makes changes rather breaking changes or a maintenance nightmare.

My main thought I had was that a move towards classes for all standalone functions would help a lot. The reason being that if things are wrapped into classes we can change functions and interfaces way easier without breaking anything or changing apis.

And yes, it would be amazing if we could use libraries instead of writing everything in vanilla python. It would alleviate a lot of pain points and make things saver. I'm not entirely sure how the packaging of archinstall into the iso works but I remember @Torxed mentioned that bootstrapping with external dependencies is not possibly (?) hence why we are including the simple menu file explicitly. Maybe we could revisit this to see if it's possible to get that to work somehow

TOTALLY NEW CODE I've rewritten the whole thing modularized and using dataclasses. To keep code isolated i moved the code to the archinstall.lib.user_interaction.diskmanager directory. Diskmanager.py is just a variant of guided.py with a call to an entrypoint (called -originally- diskmanager) to the whole thing. @svartkanin have a look at how i ended up defining the dataclasses at the equally named module, and how i had to change some things of your FormattedOutput at the output.py in that directory (i'm not that sure it is generalizable)

Still not able to be merged (documentation, type annotations, and a number of outstanding bugs) but very close now

Thanks for creating dataclasses, it will make the code quite more manageable!

Lots of cleanup. Against my philosophy i've modified guided.py this time (i've introduced an `if name' block and seems to work. I hope this will make my life easier (i import the routines i don't modify to diskmanager.py instead of copying). Seems to work When the normal flow restores at the project i'll send a separate PR for this