autofix assigned ammo

[myk002]

when the ammunition assigned to a marksdwarf is forbidden or otherwise not accessible to that dwarf

research log for related behavior here: https://steamcommunity.com/app/2346660/discussions/0/4756452379512383421/

Related issue: #3247

[myk002]

More discussion here: https://steamcommunity.com/app/2346660/discussions/0/596263710430864537/

[myk002]

From the Steam discussion:

Well, I made a few more discoveries and ran into some more issues.

Just to give a bit of background to this, first I'll describe how I identified which stack of bolts were problematic, how I removed the assignment of such stacks from a unit, and how I assigned news ones to replace them.

1. I looked for a unit in the squad equipment sidebar that had a yellow ammo icon and checked to see they're not doing a Pickup equipment job.
2. I selected the unit and ran 'gui/gm-editor', then navigated to 'uniform/uniform_pickup' to confirm they have items that they haven't picked up yet. If all other equipment slots in the sidebar is green, the IDs listed here should all be ammo items.
3. I highlighted one of the item IDs and inspected it. From here I find out where it's located and if it's in a building, in a container, or being held by another unit. Repeated for each item in the list.
4. I then deleted any problematic ammo stacks from the unit's 'uniform/uniforms/REGULAR' and 'uniform/uniform_pickup'. This alone is not enough, as the game will simply re-add the same stacks when the game is unpaused.
5. Next, I navigated to 'military' and inspected the 'squad_id' then navigated to 'ammo/ammunition/0/assigned'. This shows a list of all stacks of bolts that the game assigned to the unit's squad. I deleted all entries that contained the problematic stacks.
6. Then I navigated back to 'ammo/ammo_items'. Stacks listed here are assigned to individual units according to the corresponding entry slots in 'ammo/ammo_units'. So a stack in slot 0 of 'ammo_items' is assigned to the unit in slot 0 of 'ammo_units', and so on and so forth.
7. Since I rather not delete any entries here, I simply edited the existing entries for the problematic stacks and replaced them with the IDs of stockpiled ammo stacks that I've confirmed to be valid stacks for pickup.
8. Finally, to prompt the unit to pick up the newly assigned stacks, I navigate to 'uniform/pickup_flags' and toggled 'update' to TRUE. (this step is not strictly necessary, the game will still update it sooner or later)

And voilà! The unit went to the ammo stockpile to pick up new bolts! Yay!

…but the joy was short-lived. It turned out that messing about with adding bolts can cause assignment conflicts elsewhere.

For reference, here's a spreadsheet with the results obtained from further testing: https://docs.google.com/spreadsheets/d/1IFiKg9Vml0bcESvKe6lBKYFXqNl-VwiV/edit?usp=sharing&ouid=102329348625573417439&rtpof=true&sd=true

The sheet labeled 'Initial' shows the condition of all the units and assigned stacks at the start of the test before any changes were made. The upper table lists the units in two separate marksdwarves squads, while the lower table lists the stacks of ammo in the order they appear in 'ammo/ammunition/0/assigned' for the given squad. Immediately we can see that two of the stacks assigned should have never been valid, as they're held by monster slayer residents in the fortress. Interestingly, one of the stacks is not assigned to any individual unit in the squad. Later on I find out that it isn't uncommon for squads to have "spare" bolts. It might have something to do with the game trying to allocate 250 bolts per squad. One other stack is invalid because it was traded away at the depot, though it might not have been sold yet when the assignment was made, and thus was initially valid. More importantly, 8 stacks could not be picked up because of conflicting assignments. What's also notable is that the conflicts only occur between units of different squads (hence the o/sq in parentheses). Later on I realize that this is a frequent occurrence when you have more than one marksdwarves squad, as I recalled how smugly satisfied I was in the early years of my fortress because my marksdwarves had green ammo icons most of the time. I only had one squad of marksdwarves during those early years. Etur in The Walled Bristles appear to have a partially filled quiver, with only 15 bolts from 3 stacks. Two more stacks were assigned to her but they do not appear in her 'uniforms/REGULAR' or in her 'uniform_pickup', and she does not have a Pickup equipment job. Perhaps quivers can only fit 3 stacks regardless of stack size?

Moving on to the sheet labeled 'Delete', this is what the situation looked like after I deleted all the problematic stacks but did not add any new ones. The game assigned new stacks of bolts; some of which were problem free, while others had assignment conflicts. Note how The Walled Bristles, the newer squad, have more or less no issues picking up their stacks. Only one of their units was (temporarily) unable to pickup their bolts and that's only because it was being hauled from one stockpile to another. Only one unit in The West Town was able to pick up new bolts, and even then one of the stacks assigned to them was a conflicted stack. Etur still only has 3 stacks, but the game unassigned the extra two stacks from her even though she has less than 25 bolts in her quiver. The two stacks instead went to another unit within the squad. A few stacks in the Walled Bristles became conflicted even though they initially weren't, but in practice they do not affect the units they were assigned to because the units were already holding those stacks. However it did prevent the units from The West Town from filling up their quivers. Rather disappointingly, one of the problem stacks I deleted was re-added back to the same unit in The West Town squad. When I let the game run for a bit longer, some of the assignments for stacks that haven't been picked up yet got changed again as the two units practicing archery ran out of ammo. Further observations to changes in the ammo assignment without making any further inputs shows that even if only one unit runs out of ammo, the ammo assignment for any stacks that haven't been picked up yet get reevaluated and reassigned, and the game also appears to have a tendency to discard older stacks and assign newer ones. Stacks already in an assigned unit's quiver are typically left unchanged (exception: see the following tests).

Result: 5 of 7 units with empty quivers were able to pick up new bolts again. 6 new assignment conflicts occured.

Next is the sheet labeled 'Delete and Add'. This time, in addition to deleting the entries I created new ones to add new stacks of what should be problem free bolts. And it turned out to be a mess. Most units in The Walled Bristles including Etur were able to pick up new bolts, but once again only one unit from The West Town was able to do the same. Concerningly, one unit from the Walled Bristles was assigned to pick up a new stack of bolts even though they already had 25 bolts in their quiver. Almost all of the stacks that I manually added and assigned were either unassigned or reassigned to other units. Even worse, it seems that by adding new assignments, I have caused assignment conflicts to occur within the squad itself. And if you look at stack 2322499 assigned to slot 6 (unassigned) in West Town, it also got assigned to unit 4246 in Walled Bristles which used to be assigned to unit 10629 within the same squad and is still holding on to the stack. Plus, a stack from each squad got duplicated for reasons that I cannot figure out.

Result: 4 of 7 units with empty quivers were able to pick up new bolts again. 1 unit with a partial quiver was able to pick up new bolts. 1 unit with a full quiver was also able to pick up new bolts. 8 new assignment conflicts occurred including within the same squad.

In the sheet labeled 'Modify 1 unit', I didn't delete any assignments and simply edited the entries for one of the units. I chose Litast in The Walled Bristles because he had 3 stacks assigned to him, allowing me to see what happens if I try to overload his quiver. I wasn't sure what to expect, but it was a pleasant surprise to see that most assignments that I didn't modify remain unchanged. Only one of the stacks I assigned to Litast was kept, the rest was unassigned, hopefully because the game correctly recognized that one stack of 25 bolts is sufficient. As an added bonus, the game reevaluated the two uncollected stacks assigned to Etur and reassigned them to another unit, allowing them to pick it up. However, one conflicted stack also changed assignment from another unit to Litast, which may cause issues later on.

Result: 2 of 7 units with empty quivers were able to pick up new bolts again. Majority of pre-existing assignments and conflicts unchanged.

Finally, in the sheet labeled 'Modify All', once again I didn't delete any assignments but this time I modified all assignments that were problematic. This time, all of the manual assignments I made were either unassigned or reassigned, but most of the changes were relatively harmless. Only two conflicts occured, one of them intra-squad. No stacks were duplicated. One unit from The West Town with a full quiver was assigned a new stack, and still only one of the units with no bolts from the same squad was able to pick up new bolts. All units with empty quivers in The Walled Bristles plus Etur was able to pick up new bolts.

Result: 5 of 7 units with empty quivers were able to pick up new bolts again. 1 unit with a partial quiver was able to pick up new bolts. 1 unit with a full quiver was also able to pick up new bolts. 2 new assignment conflicts occured.

Conclusions

• Having more than one marksdwarves squad will create many ammo assignment conflicts.
• Equipment update runs itself more frequently than expected and should eventually resolve most invalid ammo assignments on its own, but it may have difficulty completely resolving assignment conflicts between squads.
• Adding stacks whether via adding new entries or modifying existing ones will be treated by the game as uncollected stacks and will be reassigned as it sees fit. In other words, assignments to any particular unit doesn't seem to matter.
• It seems that issues tend to occur more frequently on the older squad, which may have something to do with squad equipment priorities.

DFhack Solution Suggestion

From what I can tell the 'Delete' method might not do much more differently than what the vanilla equipment update already does, but as an automatic fix it could run at a higher frequency and kicking in whenever it detects invalid/conflicting assignments instead of only when someone runs out of ammo or makes new ones. It could also be used as something that players can execute on demand e.g. at the start of a siege, and be able to execute it as many times as necessary until all units have collected their bolts.

The 'Modify All' method looks like it could be a more effective fix, but I'm not sure it wouldn't create more issues than it solves since there was at least one intra-squad assignment conflict when none had occured previously. My observations have been limited, but so far I haven't noticed any intra-squad assignment conflicts occuring "naturally". In any case, due to the tedious nature of doing stuff manually I haven't done more than one or two tests for each method to be more definitive about my findings.

There might be utility for the 'Modify 1' method as well, e.g. if players want to attempt fixing just unit they selected.

Whatever the method, first DFhack must be able to identify which stacks are problematic. I'm not a coder so my logic will have flaws here and there, but taking into account the issues encountered in the tests thus far, it should check if an assigned stack:

• is marked forbid/dump/melt/trade
• has the 'trader' or 'owned' flag set to TRUE
• is in the inventory of any creature that is not the assigned unit (exception: in_job is TRUE)
• is unreachable

If any of the above is true, it should be considered an invalid assignment and rectification should be attempted.

If adding ammo stacks is to be implemented, I suggest limiting the selection pool to stockpiled ammo only. That way players can avoid situations such as having their units run halfway across the map to pick up a loose spent bolt that happened to be unforbidden. Of course, it might be a good idea to also implement options to expand or limit ammo selection, even down to material and quality. Perhaps the player can influence their units to pick up wooden/bone bolts during training, or go for metal bolts when preparing for a siege.

Right, hopefully I've covered everything! I apologize if I rambled a bit.

[myk002]

Thank you very much for the research!

Having more than one marksdwarves squad will create many ammo assignment conflicts

This explains a lot -- I tend to have only one marksdwarf squad and I very rarely have any sort of trouble.

It seems that issues tend to occur more frequently on the older squad

In general older squads get "first pick" at new equipment. However, since it appears to prefer most recently created stacks of arrows, perhaps the issues are coming from the high churn of item creation and acquisition in those items. However, if you only have one marksdwarf squad it's not negatively affected? Not sure what to think here.

is marked forbid/dump/melt/trade

In my experience, squad members tend to ignore dump and melt flags and can choose to use those items (which keep the flags set). I'm not sure if this is desirable or not. trade items will be "in_job" (by definition), so they shouldn't get chosen

is unreachable

This appears to be the primary culprit of issues, from what I've heard in the bug reports. Maybe one thing we can do is make ammo deteriorate away when not in a stockpile or building. Then at least misfired unreachable ammo will get cleaned up over time. There's no guarantee that players would turn such a tool on, though.

[amade-w]

In general older squads get "first pick" at new equipment. However, since it appears to prefer most recently created stacks of arrows, perhaps the issues are coming from the high churn of item creation and acquisition in those items.

Bear in mind in these tests I'm in effect "forcing" old stacks onto the squads and the game simply tries to work with what it's been given; none of the stacks I manually assigned got discarded in any of the tests, even though some ended up as unassigned stacks. So the results may have been skewed because of this.

However, if you only have one marksdwarf squad it's not negatively affected? Not sure what to think here.

I classified the issues into two types: invalid and conflict, but for all practical purposes they're both stacks that cannot be normally obtained and shouldn't have been assigned to the squads. Both types could be said to be invalid, but I'm really bad with names/terms.

Invalid stacks are the simple ones, they don't belong to any squad but because they're forbidden, traded away, unreacheable, etc they ought to have not been assigned to any squad at all. However, it's possible like in the case of the stack of bolts that was traded away in the test that when it was initially assigned the stack was still valid, but became invalid before the unit had a chance to pick them up. These assignments should resolve itself quite easily, especially since the game has a preference for assigning newer stacks.

Though I am puzzled as to why in the initial condition two very old stacks belonging to monster slayers were assigned to the first squad; there was never a point in time when either of these two stacks could have been considered to be valid stacks for squad assignment.

Conflict stacks are stacks that have been assigned to multiple units in the player's squad(s). At the start of the test and in the majority of the results, most of the conflicts are inter-squad conflicts while the intra-squad conflicts seem to only happen when I try to manipulate the data. So having only one squad would massively reduce if not totally eliminate conflict stacks. As to why it happens more for the older squad, well looking back at the results the conflicts occur when the game tried to add new stacks to the older squad first and some of those stacks were already previously assigned to the newer squad. So it kinda makes sense that, although technically the conflict is shared between squads, the older one is more affected because the stacks are already in the possession of the newer squad so the older squad can't pick them up.

In my experience, squad members tend to ignore dump and melt flags and can choose to use those items (which keep the flags set). I'm not sure if this is desirable or not. trade items will be "in_job" (by definition), so they shouldn't get chosen

As mentioned earlier, they might have not had those flags when they were assigned the bolts. But for the purpose of detecting problem stacks that are already assigned to the units, we'd have to use a different set of rules. in_job stacks may still be valid if, for example, the unit is in fact in the process of picking up said bolts. There can be many valid reasons why a stack of bolts may not have been picked up yet despite it being obtainable, so the fix would need to ensure that it does not attempt to remove non-problematic assignments.

Any bolts marked forbidden/melt/dump are most probably ammo the player doesn't want the squad to use, so should be considered as candidates for removal. As for items marked for trading, I recall seeing items hauled to the depot having the job_type BringItemToDepot which should help in discerning them from the generic in_job. Though I have no idea how to see via DFhack if an item already in a depot is something marked for trading, since I can't find any flags or specific_refs that point to the trading status.

Maybe one thing we can do is make ammo deteriorate away when not in a stockpile or building. Then at least misfired unreachable ammo will get cleaned up over time.

I like this only because I hate clutter, but I'm somewhat skeptical that it will reduce the instances of invalid ammo assignments. It would take a long time for them to disappear (at least, at a rate that won't appear unnatural to the player or detrimental to normal operations) and throughout the time it exists it will continue to potentially be available for assignment. But I guess it would still be very useful for players who can't be bothered to clean up the mess left behind from huge battles. In fact, the reason I'm so diligent at cleaning up spent bolts is because I'm actively trying to avoid instances of invalid assignments!

Also, looking back at my much earlier report that spent bolts aren't unassigned from the unit, I just realized that this is because when they fire the last bolt and it survives, that last bolt retains the stack's old ID! Otherwise, spent bolts usually gain a new stack ID if they survive and wouldn't already be in the unit's assignment. But if the game later assigns single bolt stacks to the unit, the unit may fire it and still keep its assignment if the bolt managed to survive again. Though this is probably an edge case and players shouldn't encounter it very often since most bolts get destroyed in use.

[amade-w]

Okay, I think I've solved the issue of assigning new bolts to units. I finally got green ammo icons on all 20 marksdwarves! https://drive.google.com/file/d/1VBZ0tygVLZrNxABnFsfXaA40_Prejm_c/view?usp=drive_link

There are two things I did wrong in the previous tests. The first is that although I added new stacks to the squad in 'ammo/ammunition/0/assigned' and in 'ammo/ammo_items', I neglected to also add the new stacks into the unit's 'uniform/uniforms/REGULAR' and 'uniform/uniform_pickup'. I made the incorrect assumption that pairing the stacks to the units in 'ammo/ammo_items' was all that's needed for the game to handle adding it back to the unit's uniform on its own.

The second thing I did wrong was that I messed up the sorting in 'ammo/ammo_items' by adding new stacks out of order. Again, I thought that as long as the stacks are correctly paired with 'ammo/ammo_units' it should work fine, but it turns out the game doesn't like having the stack IDs being out of order and will re-sort them back in ascending order. Somehow, this is what's causing the weird unassigning/reassigning of stacks.

What this means is that the 'Delete and Add' method can work just fine, provided that:

• Only stacks that are newer than those already assigned to the squad is added
• New stacks must be added in ascending order

Example: https://docs.google.com/spreadsheets/d/1eCO0-2CkQ6LSHE3YMSkI4aTbCqdse8YO/edit?usp=sharing&ouid=102329348625573417439&rtpof=true&sd=true

It's also possible to add stacks to a unit that didn't have any stacks assigned to them yet (red ammo icon).

Also, the 'Modify' method is actually impractical, since there's no way to guarantee the availability of replacement stacks that will preserve the original sorting.

So, to recap, this should be the correct method of replacing problematic stacks:

1. Delete the problematic stacks from the squad's 'ammo/ammunition/0/assigned' and 'ammo/ammo_items' as well as from each unit's 'uniform/uniforms/REGULAR' and 'uniform/uniform_pickup'
2. Add new stacks with higher IDs than the stacks that were not deleted to the squad's 'ammo/ammunition/0/assigned' and 'ammo/ammo_items' in ascending order
3. Pair the newly added stacks to the units with empty quivers in 'ammo/ammo_units'
4. Add the newly paired stacks to each respective unit's 'uniform/uniforms/REGULAR' and 'uniform/uniform_pickup'

[amade-w]

After yet even more testing I've found that replacing bolt assignments for a squad (not individual units) works best when the game hasn't paired any existing bolt assignments to individual units. So when all marksdwarf squads are off duty, it's very easy to just remove all assigned bolts from all squads and insert new ones to each squad that will not conflict with one another. When the squads go on duty, the game will automatically pair the newly assigned bolts to each unit without further intervention.

However, a practical fix should be able to run when squads are on active duty with units that have already picked up bolts. So the solution would be to have all units drop any bolts they are holding before reassigning new ones. It seems the only way to get units to drop bolts is to get them to also drop their quivers. In the same instance, remove all existing bolt assignments from the squads and insert new non-conflicting ones. Finally, run an equipment update to get the game to assign new quivers to each unit; everyone will run around picking up new quivers before they pick up new non-conflicting bolts.

This method should work better as it ignores the game's preference for newer items by completely reseting the list of assigned bolts, allowing the fix to assign any valid bolts that are available.

So here's the revised method for a proposed fix:

1. For each marksdwarf squad, delete all entries in 'ammo.ammunition.0.assigned', 'ammo.ammo_items', and 'ammo.ammo_units'.
2. For each squad position, remove the assigned quiver (maybe do a check and skip empty quivers).
3. For each marksdwarf unit, delete the pre-existing assigned quiver and bolts in each unit's 'uniform.uniforms.REGULAR' and 'uniform.uniform_pickup' (again, skip empty quivers if possible).
4. For each marksdwarf squad, populate 'ammo.ammunition.0.assigned' with at least 10 new entries*.
5. Prompt the game to do an equipment update to assign new quivers and allocate the newly assigned squad bolts to individual units.

*Ideally each stack assigned should have a stack size of at least 25. Otherwise the game will pair more than one stack to a unit to make up for the shortfall and it doesn't do a good job of dividing stacks equally (it tries to fulfill a unit's quota first before moving on the next unit), leading to one or two units getting too few bolts or none at all. This is especially problematic with bone bolts that are usually made in stacks of 5, mixed in with partially used stacks containing less than 25 bolts each. The game natively tries to assign stacks totalling 250 bolts to a squad but it doesn't ensure that each unit gets at least one stack. It's possible to add more than 10 stacks totalling much more than 250 bolts to a squad to get a safety margin. Any extra stack will simply be held in reserve for the next time the game updates the squads' ammo assignments, perhaps even potentially reducing future conflicts.

Optionally, the fix could include arguments to specify ammo type so players can assign metal/wooden/bone bolts. The fix should also warn players if they don't have enough full stacks of bolts to assign to the squads.

This fix probably shouldn't autorun periodically because it would be annoying to have squads dropping quivers all the time, so players should run it on command instead. Since sieges only happen during the first month of a season, players can run it at the start of the month and ignore any ammo assignment conflicts until the start of the next season.

During archery practice, ammo assignments will get updated frequently so it's better to just let the game handle it on its own as usual, even with conflicts. Any units without ammo will do combat training (if it's assigned to a barracks training zone), and switch to archery practice when it gets a chance to pick up new ammo when the game assigns new ones to the squad. Of course, if it seems that some units keep missing out on practice, the player can run the command to give the squad ammo assignments another reset.