Dwarf-Therapist
Dwarf-Therapist copied to clipboard
splintermind
64-bit support
Hopefully this is not actually too hard on the DT side. In theory, the only changes needed are to change VIRTADDR to be typedefed to void * and the memory layout parsers to be changed. However, there are probably a lot of places where VIRTADDRs are stuffed into ints which will end badly.
One question is though whether we should support connecting to 32-bit DF from 64-bit DT and vice versa. I'm inclined to say "no, that's stupid, why would you use 32-bit DF on 64-bit OS", but maybe there are arguments to the contrary.
I've started work on this in the x64 branch. I'm not sure what adjustments need to be made in the pro for the qmake stuff yet though. Testing has been limited without offsets yet, so there will be more work for sure.
There's still going to be the need for a 32-bit DT while DF supports 32-bit so keeping two builds is probably easiest... although there may need to be some tweaking for the updater and/or layout directory structure to split 32 vs 64 layouts.
I've started work on this in the x64 branch. I'm not sure what adjustments need to be made in the pro for the qmake stuff yet though. Testing has been limited without offsets yet, so there will be more work for sure.
I'm strongly in favor of moving to CMake entirely, because qmake sucks (more than CMake anyways). I started a local branch for it but got distracted by proper OS X support.
There's still going to be the need for a 32-bit DT while DF supports 32-bit so keeping two builds is probably easiest... although there may need to be some tweaking for the updater and/or layout directory structure to split 32 vs 64 layouts.
right, I'm saying that there are two ways to handle 32-bit support:
- compile DT in 64-bit mode and select
uint32_torvoid *depending on the bitness of the DF - compile DT in 32-bit mode and always use
void *
I think option 2 is easier.
Related to option 1, would a union like this work?
union df_ptr {
uint32_t ptr32;
uint64_t ptr64;
}
Alternatively, maybe using uint64_ts for all pointers and shoving 32-bit pointers into the first 4 bytes would be possible, although compilers tend to complain about uint64_t-to-uint32_t casts. I'm not really familiar with how DT handles this stuff already, so apologies if any of that would be too hard to implement.
~~No, that would result in a union sized incorrectly for 32 bit.~~
edit: Hello71 is correct, this statement makes no sense.
In hindsight I'm not sure what I meant by that statement. The union would be 8bytes long regardless of architecture, but other than wasting space on 32bit, accessing the 32bit pointer field would work correctly. I'm guessing I had a brain fart and was thinking about accessing the union without specifying which field. Regardless, it still won't work because you need to specify the field.
{
unint32 foo[128];
df_ptr bar = foo;
/*code here*/
}
How do you put the address of "foo" into "bar" in an architecture independent manner using that union? Even if you set the pointer (say by always writing to the 64bit field), how do you access it without knowing what architecture you are building for?
you do it... based on the bitness of the connected process? we are already doing highly arch-specific hacks (including but not limited to injecting mmap via ptrace which requires knowing arch syscall convention), so one more is not such a big difference
you are correct, but that means you are back to where you were before and the union itself doesn't add anything. In fact, just using a plain pointer that isn't of a fixed width (unless needed) would likely be way less of a headache across the board anyhow.
though I am wary to fully commit to that statement. It would make my day job obscenely easier, but I can't remember what edge cases that would cause pain with.
Alternatively, maybe using uint64_ts for all pointers and shoving 32-bit pointers into the first 4 bytes would be possible, although compilers tend to complain about uint64_t-to-uint32_t casts.
narrowing conversions are legal but you would need to be sure in the case of pointers to do (uint32_t)*ptr instead of *(uint32_t*)ptr which is undefined behavior. a union is the way to go in the case of option 1.
Hello71, yes, but my response was initially in reply to lethosor, so I had neglected to read back further up during our conversation
I think I might roll a patch combining all the changes I want to do some time in the next few days, supporting Linux only.
@Hello71, thanks for working on this. I just pulled your forks latest changes and have been testing tonight.
I got it working after starting a new fortress and was able to correctly set some labors, but DT starts crashing after getting my first migration wave.
Here is output of GDB:
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copyi
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from DwarfTherapist...(no debugging symbols found)...do
(gdb) run
Starting program: /df/DwarfTherapist
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.
[New Thread 0x7fffebcc5700 (LWP 431)]
2016-Nov-27 04:36:30.559 INFO core Dwarf Therapist "37.0.0" starti
2016-Nov-27 04:36:30.559 INFO core Runtime QT Version 5.5.1 [/buil
process 427: D-Bus library appears to be incorrectly set up; failed to
not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.
[New Thread 0x7fffea3e2700 (LWP 433)]
2016-Nov-27 04:36:30.916 INFO core beginning to read settings [/bu
2016-Nov-27 04:36:30.925 INFO core finished reading settings [/bui
[New Thread 0x7fffe1417700 (LWP 434)]
2016-Nov-27 04:36:37.562 INFO core attempting connection to running DF game [/build/src/mainwindow.cpp:403] (connect_to_df)
2016-Nov-27 04:36:37.568 INFO core adding valid layout "v0.42.06 linux" checksum: "0x6ffd260f" SHA: "10203b4ffa91ab6cc1e479dd6b809ef214cbb0fe" [/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.571 INFO core adding valid layout "v0.42.05"
checksum: "0xde7a048b" SHA: "9e0f2039f008546517997c08c41f318f98a79d7b"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.573 INFO core adding valid layout "v0.42.04"
checksum: "0xddd87cd8" SHA: "cbce922a4dae18a4e59e5d904da96d01a5be7b2e"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.576 INFO core adding valid layout "v0.42.03"
checksum: "0x1313e22c" SHA: "d81b90aafff9af84b4d6723a600f61099ea32da6"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.578 INFO core adding valid layout "v0.42.02"
checksum: "0x79b0a9a1" SHA: "ec6a3c43d97c096f997bdd22e39921b065326ad9"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.580 INFO core adding valid layout "v0.42.01"
checksum: "0x5feb615a" SHA: "00f2d47192b17c96b0787205b7b241ae15184233"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.583 INFO core adding valid layout "v0.43.03"
checksum: "0xdf4ffde3" SHA: "a79c9a734d3af0631e91a2c1aad5285c65ab200f"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.585 INFO core adding valid layout "v0.43.02"
checksum: "0x3c048bd0" SHA: "efe236ee1921468def7d24f1018afcd912ca5042"
[/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.588 INFO core adding valid layout "v0.43.05 linux64" checksum: "0xaf8cc86c" SHA: "97dc4619028e83d69d60a5d1f13eb70dca2fd800" [/build/src/dfinstance.cpp:104] (DFInstance)
2016-Nov-27 04:36:37.589 INFO core Dwarf fortress path: "/df/df_linux" [/build/src/dfinstancelinux.cpp:291] (find_running_copy)
2016-Nov-27 04:36:37.689 INFO core Setting memory layout for DF checksum "0xaf8cc86c" [/build/src/dfinstance.cpp:1043] (set_memory_layout)
2016-Nov-27 04:36:37.690 INFO core Detected Dwarf Fortress version "v0.43.05 linux64" using MemoryLayout from "/df/share/memory_layouts/linux/0.43.05-linux64-therapist.ini" [/build/src/dfinstance.cpp:1048] (set_memory_layout)
2016-Nov-27 04:36:37.691 INFO core Connection to DF version "v0.43.05 linux64" established. [/build/src/mainwindow.cpp:427] (connect_to_df)
2016-Nov-27 04:36:38.654 INFO core Loaded 12 views from disk [/build/src/viewmanager.cpp:174] (reload_views)
2016-Nov-27 04:36:38.659 INFO core group_by now set to 0 for view "Labors Full" [/build/src/dwarfmodel.cpp:780] (set_group_by)
2016-Nov-27 04:36:38.660 INFO core "redrew views in 4ms" [/build/src/viewmanager.cpp:321] (draw_views)
2016-Nov-27 04:36:41.955 INFO core current year: 153 [/build/src/dfinstance.cpp:746] (refresh_data)
Thread 1 "DwarfTherapist" received signal SIGSEGV, Segmentation fault.
__memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:117
117 ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
(gdb) up
#1 0x00000000007c263b in DFInstanceLinux::read_raw(unsigned long long, unsigned long long, void*) ()
(gdb) up
#2 0x000000000058e3c3 in QVector<unsigned long long> DFInstance::enum_vec<unsigned long long>(unsigned long long) ()
(gdb) up
#3 0x00000000005708c2 in DFInstance::enumerate_vector(unsigned long long) ()
(gdb) up
#4 0x00000000006255be in FortressEntity::read_entity() ()
My build environment is described in my fork's Dockerfile. I've also added the memfile.
You can recreate using my docker newb pack
Let me know where and how best to give you feedback. I might be able to submit patches if you're interested (and i get up to speed on the specific challenges here)...
Thanks!
What do you mean, "added the memfile"? If you mean the save, where? Otherwise, compile with cmake -DCMAKE_BUILD_TYPE=Debug, and do gdb, bt full.
By memfile I meant the 64bit Dwarf Therapist Memory Map you suggested in the bay12forums. Not saved games.
So I rebuilt your latest commit 680274b with debugging flags enabled and seems to be better. I can at least set some labors. However if I wait a while and read again it eventually crashes. backtrace debugging here
I'll get these new dwarves busy filling the quantum stockpiles and keep you posted! Thanks!
PS I updated ubergarm/dwarf-fortress docker image with DwarfFortress build 680274b if any other intrepid folks want to give it a try.
I was feeling intrepid, so I tried the docker image @ubergarm provided. It worked okay until a dwarf died. After this it kept crashing each time it opened. backtrace and crashlog.
I was able to mitigate the problem by saving and continue playing, but when I opened up the job manager DT crashed and couldn't be opened again without crashing.
EDIT: added links to backtrace and log
@telnoratti I appreciate your intrepidness! I just built a new container from the the latest patches to see if it helps anything out.
I'd love to get this going before Toady One does the next release, lol.
Just pushed a new ubergarm/dwarf-fortress docker image. Sorry no automated builds, haven't figured out best way to build/vendor everything in one place.
This update has DwarfTherapist binary built from 49b792b7
Just tested that it starts up and connects. Hopefully it keeps working with migrations and dead dwarves!
Cheers have a good weekend!
@ubergarm how do I get the 64bit patch without Ironhand? Installation instructions at all?
You may be better off with the latest Linux lazy newb pack which caught up to df 0.43.05 recently I believe:
http://www.bay12forums.com/smf/index.php?topic=163211.0
Otherwise, you can try building Hello71's dwarf therapist fork yourself like I did here:
https://github.com/ubergarm/Dwarf-Therapist/blob/DF2016/Dockerfile On Aug 4, 2017 11:31 AM, "SimplyStars" [email protected] wrote:
@ubergarm https://github.com/ubergarm how do I get the 64bit patch without Ironhand? Installation instructions at all?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/splintermind/Dwarf-Therapist/issues/313#issuecomment-320279450, or mute the thread https://github.com/notifications/unsubscribe-auth/AEgWYT5z1Q5wjHFxjjSZA_8fGnQC4j4xks5sUzlcgaJpZM4JM1fa .
@splintermind We have 64 bit memory layout of DF 0.44.02. Meybe we should compile ver 38 of DT with support of 64 bit and relase it as beta. What do you thing?
Hi, I've recently updated the DT derivation in nixos, in order to be able to use it with 43.05. Similar to the LNP, I needed to use @Hello71's branch, to get a working version. I have no stake in the cmake vs qmake question, as the according change in the recipe was rather easy (https://github.com/NixOS/nixpkgs/commit/d6ee81e912c5dbfb7db23387c31996b258d0193f#diff-0ca3a9429c3b2c3144eefa9018953779R20), but either way it would be awesome to get the fixes for https://github.com/splintermind/Dwarf-Therapist/issues/311 and https://github.com/splintermind/Dwarf-Therapist/pull/323 into the main branch, to free people to get 44.02 support going ;-)
