The DFHack Control Panel
In order to make our fortress automation functionality more useful and accessible, we could provide a unified "assistant" interface. It can have modules for:
- food (autofarm, seedwatch, nestboxes, ban-cooking)
- stock management (tailor, autoclothing, orders, autochop, autogems)
- animals (autobutcher, autonestbox, zone)
- fixes for things that can be automatically detected and applied
- tweaks
- jobs/labor (manipulator, prioritize, do-job-now, labormanager, autolabor, autohauler)
- hotkeys (hotkey-notes)
We could wrap the existing plugins/scripts or subsume them, depending on whether it makes sense to keep them separate.
Everything would be configurable from the commandline or via gui/assistant (config persistent in save). We can also add hotkey links to bring up the config screen from relevant DF viewscreens.
We can also add new functionality, like stock management based on dynamic calculations, like "ensure we have enough prepared meals and booze for the current population"
Also, we could create a "fortress advisor" screen that indicates things you can do to make your fort better, like:
- what are the top 5 contributors to dwarven stress?
- what workshops/items don't you have that might be needed for moods?
- how long will current food/booze stores last?
Maybe combine the advisor with an overview screen (dfstatus)
the configuration page for "enable-able" tools should have four columns:
- tool name
- whether to start when you load the fort
- whether it is running right now
- start configuration tool
and maybe a hotkey/button to start the ones that are selected for autostart but are not running right now.
user can save profiles (and we should provide some default ones) in dfhack-config/assistant
the current autostart configuration should be saved to the fort save dir (since it is fort-specific), in the same format as a profile. a script can be added to the save dir's init.d/ directory to run the autostart config.
maybe now that the launcher is more of the primary DFHack interface, this script should be called control-panel..
Recording of a discussion on discord about how we can potentially manage state for plugins and scripts so their enablement is persistent.
I've arrived at zone for the documentation pass. I'm splitting it up into three plugins before I do the docs, though. I'm also taking this opportunity to figure out what a "canonical" plugin should look like I think it comes down to this: there are three classes of plugins: 1) provides commands but has no persistent impact on the game, 2) has a persistent impact that is not save-specific, and 3) has a save-specific persistent impact. (2) and (3) implement the enabled API, (1) does not There are some plugins, like reveal, that declare an "enabled" var that is visible in the enable output, but they use it like internal state and they can't be enabled/disabled by the user via the enable and disable commands. I'm classifying those as (1) Class (2) plugins can get enabled anytime, and often get enabled now in dfhack.init Class (3) plugins can only get enabled after a save is loaded So, behavior. Here's the current proposal. myk002 — Today at 11:07 AM Class (1) plugins: no guidance about keeping state. most plugins of this class do not have state to keep, but those that do can persist it if it makes sense to. Note that state would likely need to be kept in dfhack-config since it's not guaranteed that a save will be loaded. Class (2) plugins: similar to class (1). Since there is no save, state (if any) must be kept separate. In particular, class (2) plugins will not remember if they are enabled and must be reenabled in startup scripts. Class (3) plugins must keep their state in the savegame AND remember if they are enabled. This is the labormanager model. I proposed earlier that plugins should not remember if they are enabled to bring them on par with scripts, and then both could get re-enabled on load by a separate mechanism (i.e. the proposed assistant). I'm walking that back, since things should work conveniently even if players don't use the assistant interface. I'm now thinking instead that scripts need to be brought up to the capabilities of plugins and be given the chance to reload their state automatically. I'm trying to put together example code that makes creating class (3) plugins with canonical behavior easy. I wrote the code, but I'm not happy with it. It looks confusing and brittle and unclean. It might be better to inherit this behavior from a superclass instead of providing a copyable "skeleton". Or maybe people here have suggestions for improvement. I wrote the new autonestbox as an example of the canonical structure. The skeleton code would be based on the first 219 lines of https://github.com/myk002/dfhack/blob/myk_rezone/plugins/autonestbox.cpp GitHub dfhack/autonestbox.cpp at myk_rezone · myk002/dfhack Memory hacking library for Dwarf Fortress and a set of tools that use it - dfhack/autonestbox.cpp at myk_rezone · myk002/dfhack dfhack/autonestbox.cpp at myk_rezone · myk002/dfhack I'm not happy with how the enabled state is accessed directly in so many places, and I'm not happy with the raw access to the persistence API and what that means for reacting to invalid or uninitialized state. Lethosor — Today at 1:44 PM regarding the last bit, couldn't they do that with a state change handler? myk002 — Today at 1:45 PM possibly, but how would they register the state change handler? the user isn't running the script from anywhere the act of loading the game needs to trigger something in the script e.g. if scripts could define a global function script_onstatechange() that gets called automatically Lethosor — Today at 1:46 PM that's another reason why "enable" runs the script to enable scripts to install event listeners myk002 — Today at 1:47 PM but also nobody is running enable -- enable would have been called at some point, then the game is saved and DF is closed the script needs a way to reactivate when the DF is opened anew and a game is loaded that previously had the script enabled (without anything specific in an init script) plugins have plugin_init() and plugin_onstatechange() to handle this case. how would it work for scripts? Lethosor — Today at 1:50 PM dfhack.init? maybe in the save folder I kinda see your point... how are plugins enabling themselves per-save? I had assumed there was a "no plugin code runs unless the plugin is enabled" policy... but some plugins do try to enable themselves myk002 — Today at 1:52 PM labormanager, autobutcher, and autonestbox create persistent data in the save and check for that persistent data when a map is loaded. if the persistent state says the plugin was enabled last time, it autoenables itself this kind of behavior does make sense for what I called "class (3)" plugins. "persistent per fort" is what labormanager calls it. Lethosor — Today at 1:53 PM so no command is ever run containing labormanager for it to work? just the existence of the plugin is enough? myk002 — Today at 1:54 PM they do require an initial manual (or init-script-based) call to enable but only once. further game loads do not need the call to enable Lethosor — Today at 1:54 PM then a script could arguably require the same oh, by "game load" you mean "restart DF", nvm myk002 — Today at 1:55 PM well, or save current game and load a different one labormanager and the zone twins do automatically deactivate themselves when a map is unloaded. in plugin_onstatechange myk002 — Today at 1:56 PM I think that's a good idea, but we need to figure out the mechanism scripts adding a stub to savedir/init.d would certainly work, but I wonder if that is the best way we already have the persistence API, so it might be more natural for scripts to store state that way instead of in a stub file and if we're using persistence anyway for other state, then it seems like it would be easier to have an API closer to what plugins support Lethosor — Today at 1:58 PM I think the persistence API is tangential to this the main question I'm seeing is "how do scripts run code on fort load if they were never invoked in this DF session" doesn't matter where their state is stored if they don't run myk002 — Today at 1:59 PM I'm bringing it up because a persistent script has two tasks it has to accomplish when a game is loaded - 1) enable self if previously enabled and 2) reload state if (2) is going to be via the persistence api anyway (probably), then it makes sense that (1) could also be done that way Lethosor — Today at 2:01 PM yes, but invoking code that does (1) is the problem I guess that could be a new core feature or something keep track of some list of enabled scripts in the save, and call enable on each when the save is loaded? myk002 — Today at 2:04 PM that could work. it would involve getting the scripts to expose whether they are enabled (or rewriting them to require enablement via enable instead of what most of them do now, which is take start and stop commands) Lethosor — Today at 2:04 PM would it? I think they could request it themselves dfhack.something.markScriptEnabled(true|false [, dfhack.current_script_name()]) although having enable/disable handle it could reduce potential for error on the script side myk002 — Today at 2:09 PM that would work. at least for "class (3)"-type scripts that require a world in order to be enabled Lethosor — Today at 2:11 PM right, there's no way for enable/disable to know whether a script is (3) or not myk002 — Today at 2:11 PM I suppose "class (2)"-type scripts (if there are any) would need to be enabled from an init script Lethosor — Today at 2:11 PM yeah there are some UI tweaks written as scripts myk002 — Today at 2:12 PM it would be nice if we could distinguish the two programmatically Lethosor — Today at 2:13 PM well, can we with plugins? myk002 — Today at 2:13 PM not that I know of i mean, you can try enabling it without a world loaded and seeing if it fails, I suppose Lethosor — Today at 2:14 PM I guess plugins are a bit of an unfair comparison because they can run code whenever they want Lethosor — Today at 2:14 PM so I'm kind of leaning towards this although it would be specific to "enable on world load" (class 3) scripts, so maybe the name could be improved myk002 — Today at 2:16 PM dfhack.enableOnWorldLoad()? Lethosor — Today at 2:17 PM sure. can't think of a great location for it. Not quite dfhack.internal material, imo (it should also take a enable/disable boolean to allow disabling per-save) myk002 — Today at 2:18 PM it could be a script header thing myk002 — Today at 2:18 PM (ack, I was just eliding the params for brevity) Lethosor — Today at 2:18 PM figured Lethosor — Today at 2:18 PM hm, yeah, it could be. Those are already checked in dfhack.enable_script() --@ enable_per_save = true? myk002 — Today at 2:21 PM and dfhack_flags.enable_per_save_state = true|false to set the state? Lethosor — Today at 2:22 PM I don't think that'd be necessary. if dfhack.enable_script() knows the script wants to be enabled per-save, it also knows the desired state based on its state param so it can handle the update (plus I think dfhack_flags is read-only, at least now) myk002 — Today at 2:23 PM not if the script rejects enablement (because a world isn't loaded, for example) unless we require a qerror so we can catch the refusal Lethosor — Today at 2:24 PM enable_script() could fail if enable_per_save is set and a world isn't loaded myk002 — Today at 2:24 PM true, however, what about scripts that self-enable? Lethosor — Today at 2:24 PM since it would be updating world-specific data if a world is loaded sigh wait, that should work with this ok, let me start from the beginning
- user runs enable myscript with a world loaded
- this goes through dfhack.enable_script(), which checks for the enable_per_save flag
- if enable_per_save == true, and a world is loaded, then enable_script() writes some config somewhere in the save to mark the script as enabled in that save that config is then read by something in core when a world is loaded and enables any listed scripts myk002 — Today at 2:27 PM plus, we can potentially differentiate between class 2 and 3 by whether they use @ enable = true or @ enable_per_save = true in their header Lethosor — Today at 2:28 PM yep, that's the reason I'm suggesting it myk002 — Today at 2:29 PM for the self-enablement path, how would this work. Say, for example, you run prioritize -a CLEAN_SELF. what's the flow? (looks like we just got pre-commit bombed) Lethosor — Today at 2:29 PM lol Lethosor — Today at 2:30 PM ok, so maybe we do want a markEnabledForThisSave() API both enable_script() and scripts themselves could use it myk002 — Today at 2:30 PM could a script just route back through enable_script()? prioritize -a could check am I enabled? if not, then enable_script on self, then go ahead and add to my watchlist Lethosor — Today at 2:34 PM yeah that could work too just have to be careful not to get stuck in a loop :p myk002 — Today at 2:36 PM yeah, enabling should trigger loading state (not including the enable state itself) and nothing else and in Core, on SC_WORLD_LOADED, we load our persistent state from the save on which scripts are enabled_per_save and call enable on those scripts. Similarly, SC_WORLD_UNLOADED would trigger disabling all enabled_per_save scripts now..would this same scheme work for plugins? complement DFHACK_PLUGIN_IS_ENABLED with DFHACK_PLUGIN_IS_ENABLED_PER_SAVE and manage enablement horizontally Lethosor — Today at 2:43 PM it might be nice to abstract some of that logic away into core, but it would live somewhere different from what's been proposed for scripts myk002 — Today at 2:44 PM yes, different implementations, but I like how the properties can be exposed to the user in a more uniform way so users can more easily understand what to expect from each plugin/script also common enable state restoration logic can be shared across plugins instead of each class (3) plugin having to implement it separately. class (2) plugins and scripts could also have their enable state managed, actually we'd just trigger them on SC_CORE_INITIALIZED and store the list in dfhack-config/settings.json (or something)
The DFHack Control Panel design doc
Overview
The DFHack Control Panel is an in-game GUI interface that will allow the user to quickly understand and manage which DFHack tools are enabled, edit their current configurations with GUI config screens, set the configuration they want per-fort tools to be in when starting a new fortress, and select the one-time tools they want to run when starting a new fort. It is intended to offer players a visual management interface for DFHack configuration, and will be an in-game alternative to most current use cases for init scripts (though init scripts can still be used exactly as they are today if the player prefers them).
The DFHack Control Panel will not handle restoring enabled tools and tool configuration when a game is reloaded. That will be managed by a separate enable-tracking subsystem. The control panel will interact with that subsystem to display current tool status and to toggle whether tools are enabled.
Requirements
- Most players must be able to configure new-fort state completely from in-game, with zero editing of text files and zero typing of commands (though see Limitations section below)
- Control panel operations must be accessible from both the CLI and GUI
- Existing
dfhack-config/init/scripts must continue to function, and users who prefer to configure everything in init scripts must continue to be able to do so - 3rd party tools must have a way to integrate with the control panel UI
Critical user journeys
These are primary interactions and workflows we expect the player to go through with this tool.
- Player starts DF for the first time after installing DFHack. The control panel comes up, showing all DFHack tools that can be enabled and already configured with sane defaults. A welcome message is shown, telling the player how to use the control panel and how to get back to it later. The player is given the opportunity to select a more appropriate profile, if they want.
- Player wants to set initial configuration for starting a new fort. Player brings up
gui/control-panelvia the tilde hotkey or shift-clicking on the DFHack button. Player chooses and applies the profile closest to what they want, then optionally chooses named presets for individual tools they would like configured differently. If they want something completely custom, they manually configure the tool via the commandline (or its relevant GUI config screen, if available) and then selects the "Use current configuration" button for the tool in the Control Panel UI. - Player wants to see which DFHack tools are enabled right now and/or wants to modify active configs. Player brings up the control panel and views the Enabled checkboxes and/or clicks on the tools they want to configure, which launches the GUI config screens for those tools.
- Player wants to try a new config profile in a running game (e.g. going from "Light automation" library profile to the "Full automation" library profile). Player brings up the control panel, Selects "Import/export profile", imports the profile from the list, and selects "Reset all tools to new game presets".
Invocation
The DFHack in-game control panel will be accessible via:
- Running
gui/control-panel - Shift-clicking on the DFHack overlay button
- Hitting the tilde (Shift-backquote) hotkey
- Hitting the Ctrl-Shift-C hotkey
- Selecting "DFHack control panel" from the Esc menu
The CLI will also be accessible by running control-panel.
Interface
Overview tab:
|----------------------------------------------------------------------------------------------------------------------|
| DFHack Control Panel |
|-----------------------------------------------------------------------------|----------------------------------------|
| /Overview\New fortress\DF config\ | |
| | Preset details |
| Alt-T Showing: All types | ============== |
| Alt-S Search: _ | |
| | enable autofarm |
| Tool Enabled Type Apply preset New fort preset | autofarm default 30 |
| --------- ------- --------- ------------ ----------------------------- | autofarm threshold 150 GRASS_TAIL_PIG |
| autochop [ x ] Per-fort [ <- ] Disabled ^| |
| autofarm [ x ] Per-fort [ <- ] Enabled with default settings o| |
| autolabor [ ] Per-fort [ <- ] Promote artisans o|----------------------------------------|
| autobutcher [ x ] Per-fort [ <- ] Custom | |
| spectate [ ] Ephemeral | autofarm ^|
| zone [ x ] Global v| ======== o|
| | o|
| Alt+P: Import/export profile Tab/Shift+Tab: Cycle keyboard focus | tool help text |
| Ctrl+Shift+R: Reset all per-fort tools to new game presets | v|
|-----------------------------------------------------------------------------|----------------------------------------|
New fortress tab:
|--------------------------------------------------------------------------------|
| DFHack Control Panel |
|---------------------------------------|----------------------------------------|
| /Overview/New fortress\DF config\ | |
| | Command details |
| Alt-S Search: _ | =============== |
| | |
| Tool Run on new fort | 3Dveins |
| ------------------- --------------- | |
| 3Dveins [ x ] ^|----------------------------------------|
| light-aquifers-only [ x ] o| |
| fix/blood-del [ x ] | 3Dveins ^|
| reveal [ ] v| ======= o|
| | o|
| Alt+P: Import/export profile | tool help text |
| Tab/Shift+Tab: Cycle keyboard focus | v|
|---------------------------------------|----------------------------------------|
Import/export:
|-----------------------------------------------------------------------------------------|
| DFHack Control Panel |
|------------------------------------------------|----------------------------------------|
| Ctrl-S Search: _ | [library] UI improvements and fixes ^|
| | only o|
| Profiles | o|
| -------------------------------------------- | o|
| [Export new profile] ^| Includes basic DFHack improvements, o|
| My awesome profile o| but no gameplay changes or automation o|
| [library] UI improvements and fixes only o| helpers. o|
| [library] Light_automation | |
| [library] Full_automation v| Newly enabled: |
| | autolabor [Promote artisans] |
| Ctrl+E: Export profile Ctrl+I: Import profile | autobutcher [Custom] |
| Esc: Go back | zone v|
|------------------------------------------------|----------------------------------------|
Left panel:
- The "Showing" drop-down filters by tool type (Global, Per-fort, or Ephemeral), or "All types" for no filter
- The search box allows additional filtering by substring
- If there is an associated gui configure tool, the tool name will be shown on a button. otherwise it will be plain text. Launching the tool's gui config will hide this control panel, which automatically comes back up afterwards (with the same view state as when it was hidden).
- The "New fort preset" column drop-downs (only offered for persistent-per-fort tools) will include options for:
- "Enabled with default settings"
- any alternate presets
- "Disabled" and includes a button for "Use current active configuration". If the button is selected, a popup comes up confirming that the player wants to use the current tool configuration as the new game config, and a "Custom" option appears in the drop-down list and is selected.
- Tool name buttons, "enable" checkboxes, "apply preset" buttons, and the "Use current active configuration" dropdown button are disabled if the tool cannot be enabled (or configured) in the current mode (i.e. the tool is fort-specific and a game isn't loaded).
Right panel shows the commands the currently selected preset will run and the help text for the currently selected tool.
When "Import/export profile" is selected, the import/export panel replaces the left panel. The left panel shows the list of saved profiles, including library profiles. A list of changes compared to the active profile will be shown on the right (in the help panel). If the profile has a populated description field, the description will appear above the diff output. Export current settings to the highlighted profile with Ctrl-E. When exporting, the player is prompted to update/create the description. Overwriting by export is disabled for library profiles. Modifications to library profiles must be done by importing them and exporting to a new copy. Apply the selected profile with Ctrl-I.
Keyboard navigation
- Tab cycles focus among panels. focus is visually indicated by a highlighted border
- When the tool list has focus, up/down/left/right/pgup/pgdn moves the highlighted entry in the grid. Enter invokes the widget in the current grid cell.
- When the profile list has focus, up/down selects a profile
- When a help panel has focus, up/down/pgup/pgdn scrolls the text
Mouse navigation
- All hotkey labels/buttons are clickable to invoke the associated action
- Scrollbars are clickable/draggable
- Clicking on the tool name launches the gui config for that tool (if any)
- Clicking on the enabled checkbox enables/disables the tool
- Clicking on any drop-down pops up the drop-down list
- Clicking on a profile name selects that profile
Files
- Selected "New fort preset" options are stored in
dfhack-config/control-panel/on_new_fort.json - player-saved profiles are stored in
dfhack-config/control-panel/profiles/<profile_name>.json - library profiles are stored in
dfhack-config/control-panel/profiles/library/ - dfhack-owned configuration presets for individual tools are stored in
hack/control-panel/presets/<tool_name>.<preset_name>.json - player-owned (and 3rd party) configuration presets for individual tools are stored in
dfhack-config/control-panel/presets/<tool_name>.<preset_name>.json
profiles include a name, description, list of tools and their enabled/disabled state, and the selected preset for new fortresses, if applicable.
Library profiles
DFHack will come with some useful basic profiles.
- "UI improvements and fixes only" (similar to our current defaults, but perhaps with more UI enhancements and bugfixes enabled)
- Light automation (automation that would benefit most people but which don't replace any gameplay elements, like
workNowandprioritize - Full automation (everything that's not super niche is enabled with sane defaults)
Program load flow
Here's the flow of operations from starting DF through having a map loaded and ready to play:
- DF starts
- DFHack loads plugins
- DFHack core re-enables class 2 (global, non-save-specific) tools that were enabled on last exit
- Class 2 tools that have saved state load their state
- dfhack.init runs (can override automatic re-enable logic)
- Player starts/loads savegame
- onLoad.init runs
- Map is loaded
- If this is the first load of this fort, run
control-panel --on-new-fortto apply tool presets and run any registered one-time commands (likelight-aquifers-onlyor3Dveins) - onMapLoad.init runs (can override control panel config)
control-panel --on-new-fort can be invoked from hack/init/onMapLoad.default.init to apply initial fortress config. We need to ensure that it runs exactly once per fortress, which unfortunately neither once-per-save nor on-new-fortress can guarantee. AIUI, once-per-save wouldn't run the command if a fortress is abandoned and a new fortress was started in the same world, and on-new-fortress could run multiple times if you load the save at timestamp 0. We'll have to figure out an appropriate detection mechanism.
Limitations
This interface is not very well suited to managing keybindings and other non-profile-specific config. We wouldn't want a player to lose custom keybindings just because they're trying out a new profile! At this time, this type of config must still be configured in init scripts. We'll have to reach out to the community to figure out whether this is acceptable.
Data structures
Tool entry:
name: string (name of tool; what you'd type to invoke it)
gui_config: string (commandline to invoke to start the GUI config interface for this tool; nil if none)
is_enabled: boolean (whether the tool is currently enabled)
tool_type: enum (classes defined [here](https://docs.google.com/spreadsheets/d/1hiDlo8M_bB_1jE-5HRs2RrrA_VZ4cRu9VXaTctX_nwk/edit#gid=459286236))
presets: list of preset objects (nil if tool has no presets, which means that the tool can be run without params (if class 1) or just enabled/disabled if (class 2, 3, or 4))
cur_preset: preset (a pointer to the selected preset object, nil if tool has no presets)
Preset:
name: string (can contain spaces)
commands: list of strings (the commands to invoke to apply the preset)
A profile (including saved profiles) is a list of tool entries, sorted by name.
The saved list of presets is the active profile, filtered to include only fort-specific tools and only persisting the name and cur_preset fields. This data is applied when a new fortress is begun.
Metadata infrastructure
We need to be able to query the following properties for tools:
- Whether they can be enabled
- Whether they are enabled
- Their tool class
- Whether they have a GUI config interface and how to invoke it
There will be a separate design doc around tracking the enabled state of scripts. It will likely result in an API that we can call to discover which tools are enableable and which are enabled. This section will focus on the other bits of metadata.
In order to support 3rd party tools, we can't just keep hard-coded lists of class 2 vs class 3 tools and which commands we run to get to which config screens. We need to have a system that 3rd party scripts can plug into. Ideally, this would be the same system that the in-tree scripts plug into so that everything works the same way.
Here are three potential ways I can think of for tools to expose their metadata:
- Implement APIs
- Scripts can implement a specific API to return the metadata, e.g. a global
function script_getMetadata()that returns a table that indicates the script class and the command to use to invoke a gui config screen. - Plugins can implement a similar
DFHACK_EXPORT ToolMetadata plugin_getMetadata()API.
- Declare metadata in code
- Scripts can add lines to their headers, similar to the
@enabledmarker that indicates they implement the enabled API. e.g.@script_class=saveand@gui_config=gui/autofarm - Plugins can declare similar exported variables via pre-defined macros.
- Declare metadata in help
- Both scripts and plugins can report their metadata in their help text, which can be parsed and processed by helpdb and made available for querying
I haven't explored all the options yet, but putting the metadata in the help appeals to me. The information is useful for players to help them understand what to expect from the tool, and the gui config command could be rendered as a hyperlink to the associated tool in the online docs.
Required changes to tools
In addition to whatever we do for the metadata (and of course whatever changes we need to make for persisting/reloading configuration), we'll have to ensure that all tools that can be configured supply a way of generating the commands necessary to reproduce the active configuration. autobutcher and some other tools do this already, but all tools will need to support it so we can provide the "Use current configuration" button and create the "Custom" presets. Whether the tool supports this interface could possibly also be included in the metadata the tool exposes. This would allow us to integrate tools into the control panel even if they don't support the "Use custom configuration" feature.
Questions
- Do we need to detect when a new adventure is started? Are there any tools that players would want to enable and apply a preset for in adventure mode? [Ozzatron: "no"]