OpenBVE perhaps return to Debian again...!?

[ginga81]

Relative with #261.

Today, via Twitter, a Debian Developer henrich is conversation to me. https://twitter.com/henrich

He say that if you mails to Debian's WNPP, he becomes to the sponser. Please send WNPP to [email protected].

If recoverly to debian, you(Mr.leeser3) can continue to registration to debian directly.

At https://www.clear-code.com/blog/2014/3/7.html, the WNPP's prototype mail foramat is written.

Additionally, he checked your OpenBVE's .deb, and he advice to me. He say that some files are still not ready for registration. For example, debian/copyright. The deficiency files should to attach based from bellow. http://snapshot.debian.org/package/openbve/1.4.0.10-3/

Additionally, perhaps you are not use official deb packaging tool. If you use official deb creating tool, it is so smart.

I will continue to contact to developper henrich. If you sent to email to [email protected], please announce to me. I tweet to him.

[leezer3]

Interesting!

I agree debian/copyright is missing, will add that back again. The last conversation I had with @directhex (Previous package maintainer) was over email about 2 years ago.

I've got no real experience with Debian packaging, and the current package is based upon a combination of the original scripts and some tweaking of my own. IIRC Debian would want the compatibility data separated into a second package.

She was relatively happy to chat, but nothing happened, and I think she had a lot of other stuff on her plate too.

[ginga81]

fetch from the http://snapshot.debian.org/package/openbve/1.4.0.10-3/, some more deficiency files.

He said that he will be a sponsor if you prepares and emails the necessary documents and the appropriate deb file. He said to me that it is busy now to the next debian release, so he said to me that every few days please mention. So please email me after preparation.

[directhex]

The last active package maintainer in Debian was actually @sladen, I just had some oversight as part of my general control of .NET in Debian

[sladen]

@leezer3 @ginga81: would encourage working with the packaging as-is:

• https://github.com/sladen/openbve/

was a careful import of released .zip snapshots into revision control, until Michelle moved on.

Is there a clear new upstream, from which to make new releases?

[leezer3]

This repository is the upstream these days. I don't believe there has been any other fork / continuation released with any working development or new features :)

The release branch and the point releases should provide the snapshots you're after I think.

The history should be contiguous from the last source Michelle released although I'd have to check that; the packaging and other build scripts have been later additions on the top, as well as a lot of (ongoing) reorganisation.

Essentially, I think what this needs is a clear list of points Debian would like us to address?

[leezer3]

OK, so I've had a quick look back into the commits and the linked repo from @sladen

The commit tree for this repo starts here: https://github.com/leezer3/OpenBVE/commit/7416862f3009bf32ad87f407237b48f5872e61d3

I appear to have started from a base of Michelle's last released ZIP source (1.4.3.0), and done some work before importing this into the GIT repo. This contained the full set of developer tools, data etc. After that, the history is contiguous.

Sladen has a set of repositories which are split by program, so that each of the tools lives in it's own repository. I'm not necessarily sure this is the 'correct' method of doing things in a game of this nature, especially as there is a massive amount of commonality between the different tools, and none of them have any real use outside the game. (N.B. One of the ongoing jobs is merging the duplicate code between the tools), either way it's two different / disparate approaches. Either way, Debian appears never to have distributed the tools, so I'm not sure how relevant this is.

---

Thoughts / TODO List:

• Sort out the missing files from the Debian package. Should just be able to pull these from the main Debian tree.
• Create a separate package for the menu / language / compatibility data? I'm not entirely clear what the original Debian motivation for this was, but presumably something to do with reducing download sizes on an upgrade? Separating the default route / train I can absolutely understand, but I wouldn't have done this personally :)
• We distribute / require various C# libraries / wrappers ( CS-Script, NAudio, NUniversalCharDet. OpenTK, SharpCompress, Ionic.Zlib), which are not available as default Debian imports. All of these have a suitably permissive licence, but I'm not entirely clear how exactly Debian would want these presented?

[sladen]

@leezer3, it's 10 years ago, so things may be a little hazy.

• openbve-data is cross architecture: consisting of images (which mostly never changed), documentation, and translations (which tended(?) to change quite rapidly, because translations would also be contributed directly, outside of the main source releases)
• openbve is the executable and immediately required configuration programs. ie. everything that has to be compiled at-the-same-time for things to work. Being CLR/CLI this is theoretically cross architecture; but I did have loading of i386 plugin DLLs working, and that code it was not cross-architecture.

For the content side, we had three examples (all thanks to Anthony):

• bve-plugin-uktrainsys, a plugin usable across various routes
• bve-route-cross-city-south, an example route
• bve-train-br-class-323, an example train
• bve-train-br-class-323-3dcab, an example 3D train interior, depending/extending bve-train-br-class-323

The route/train/cab/plugins are not OpenBVE-specific per-se, they are in BVE-CSV fileformat and could also be used with eg. the original BVE Trainsim non-free freeware program + tools. Hence the bve- prefix.

It would be great to bring the tools into the main repository, and I'm sure Debian/Ubuntu would be very happy to follow whatever Upstream does! ;-)

[leezer3]

No trouble.

I've been in this community since 1997 or there abouts, and have seen the entire gamut of BVE..... Started developing this rather by accident, but now things have rather started to get a lot more complex!

Main Program:

I've actually done some playing with cross-architecture proxying for i386 plugins: https://github.com/leezer3/OpenBVE/pull/230

This works acceptably(ish), but it's not code I was ever happy with. Being honest, it's something which needs someone with considerably more understanding of cross-architecture would need to look into properly again. We're also still limited to openGL 1.1 & Display Lists, which somewhat rules out most non x86 architectures.

Data:

The data included in the current releases has grown somewhat since the original days. At present, it's this:

• Windows dependancy stub installers (irrelevant to Debian packaging)
• Menu images / data etc- These get added to occasionally to support new features, but are generally largely static.
• Translations- A lot of these are out of date unfortunately; Where new strings have been added, these have been inserted in English. Getting contributions to this is hard, but might change if we get back into Debian....
• An open-source copy of various objects from the Uchibo route by Mackoy- These are called by a lot of early BVE2 / BVE4 Japanese content, and the original is not easily available / installable for users of modern non-Japanese Windows.
• Documentation (?) - A copy of this is included in the main source repo, but not with the downloads.

At 5mb compressed, whether a separate package is worth the trouble is something Debian would have to decide. Distributing a deb from our site, I made the decision that supplying a single file was just easier, although obviously being in a repo means that it just gets pulled in as a dependancy :)

Tools:

My instinct would be to include with the main program, and use executable names prefixed with openBVE as follows:

openbve-objectviewer
openbve-routeviewer
openbve-traineditor
openbve-carxmlconvertor

etc.

An alternative would be to add a Developer Tools button / dialog to the main menu, which would avoid adding anything to the path?

Content:

I'm happy to provide 3 more fully cross-platform locos & assorted consists ( GWR 81xx, GWR Manor, BR Class 52 ) in public domain / BSD-3 if we feel that is a good idea. None of these work correctly on the original BVE trainsim however. (n.b. openBVE introduced both the 3D cab and train exteriors, so neither of these work per-se with the original BVE Trainsim)

Route-wise is somewhat more complex, and I don't think there's anything easily available that could be added.

As opposed to Michelle's original managed content system. a relatively early implementation was a basic package manager. This isn't anything particularly special, just reads a metadata XML from a zip, and allows installation / removal etc. I don't know quite how this will relate to the Debian content packages, but it was considered important to have a format which automatically packaged / installed content, and was cross-platform.

Other:

The issue of the licence mess Michelle left us with pops up every once in a while: https://github.com/leezer3/OpenBVE/issues/305

We have agreement from all the non-trivial contributors to this repository that a move to BSD is acceptable in principle, and TBQH the current licence probably would allow us to do this anyways. I'd be interested if you have comments or preferences in this regards.

[leezer3]

OK, have done rather a lot of messing around with the generated deb.

This is now clean of all red lintian warnings, other than the lack of a changelog. I haven't currently got a good way to pipe this into the requisite file, so it would probably have to be created manually by whoever builds the debian package.

However, there are still some flaws:

• A bunch of the stuff in the assets directory has permissions which Lintian thinks is incorrect. This can probably be sorted easily enough with a permissions commit in GIT. (Just how important is this anyway?)
• Some of the built code seems to be getting again 'incorrect' permissions. I'm tempted to just recursively chmod all exe files to 0755 and anything else to 0644 as a post-build step, but it'd probably be 'better' to fix this in the makefile.
• The small usertool we use to add the LBA header is getting packaged. Harmless, but should be squashed after it's used.

This really needs someone familiar with Debian packaging to take a look now though. I can read Lintian warnings just fine, but I don't know enough about the fine details of what's happening.

[directhex]

Okay, so, taking a look at the packaging metadata, I feel it would be much easier to maintain in the longer term by using some newer conventions - right now it's using format version 5 from 2005 (https://github.com/sladen/openbve/blob/debian/debian/compat) and 75% of that file would go away with version 7 from 2008. https://salsa.debian.org/dotnet-team/xsddiagram/blob/master/debian/rules is an example of a Debhelper 7 rules file for a project built entirely from sln/csproj - and half of that file is a custom rule for downloading the .tar.gz.

Some of the automagic things in dh7 would likely help you

[directhex]

Sorry, I should have given a bit more context as to what's happening.

%:
	dh $@ --with cli

Means "for every Debian packaging rule, just do whatever the dh command wants to do, but add the .NET-specific commands defined in /usr/share/perl5/Debian/Debhelper/Sequence/cli.pm"

Then for every command dh wants to shell out to, you can choose to override it by adding a override_dh_foo: target, e.g. override_dh_auto_build (which detects things like autotools or cmake or meson) to instead call xbuild on the sln. You'd throw away everything in the existing rules and leave something like:

#!/usr/bin/make -f
export DH_VERBOSE=15
BUILDDIRS = openBVE/OpenBve openBVE/OpenBveApi \
 openBVE/OpenBveAts \
 openBVE/Sound.Flac openBVE/Sound.RiffWave \
 openBVE/Texture.Ace openBVE/Texture.BmpGifJpegPngTiff
TARGET = $(CURDIR)/debian/openbve
EXEC_TARGET = $(TARGET)/usr/lib/openbve
EXEC_PLUGIN_TARGET = $(EXEC_TARGET)/Plugins
#DEBUG_CONFIGURATION=Debug
DEBUG_CONFIGURATION=Release

override_dh_auto_build:
	for builddir in $(BUILDDIRS); do \
	  (cd $$builddir && xbuild /p:Configuration=$(DEBUG_CONFIGURATION) *.csproj) || exit 1; \
	done

override_dh_auto_clean:
	for builddir in $(BUILDDIRS); do \
	  (cd $$builddir && xbuild /t:Clean *.csproj); \
	  rm -rf $$builddir/bin $$builddir/obj; \
	done

override_dh_auto_install
	#do most of this stuff in debian/install, maybe with dh-exec
	lynx -dump -nolist $(CURDIR)/debian/changelog.html > $(TARGET)/usr/share/doc/openbve/changelog

override_dh_clideps:
	dh_clideps -d -r --exclude-moduleref=AtsPluginProxy.dll

%:
	dh $@ --with cli

[leezer3]

Thanks!

The current in-tree script we've got (and which I've been modifiying today) actually builds the solution (Uses mcs , not xbuild) before copying it into the Debian package structure and calling dpkg-deb to build the thing. IIRC the only thing that got used from the original package was a stripped down metadata file so that we'd install over an existing install gracefully.

After today's work, it declares format V7, which if I understand correctly isn't actually doing much in this case, as I'm essentially just packing binary files.

If I understand what's going on with the rules file correctly, the only real advantage would be that we could just call the dpkg creation command in the root directory, and dpkg would sort everything else out? Not entirely clear whether it's a Debian requirement for things to work in this way, or just whether the deb must conform to standards?

[directhex]

For a "use Debian packaging on a tarball of binaries not source" your example is https://github.com/mono/linux-packaging-nuget

But bear in mind this strategy isn't ok for inclusion in packaging repositories

[leezer3]

OK, so I've got some progress, but I'll admit this is giving me a real headache.

Due to a lot of changes to the structure of things (specifically intended to reduce the amount of duplicate code between the main program and the viewers), the build order is rather important. Attempting to do this manually is a PITA and requires manual work every time something changes.

As it looks likely that the tools may be included with the main program, I think this makes the easiest method of doing things to be to simply build the main solution file directly, and let this handle the build order itself.

I think we'll need a separate target, or a call to delete the data, but keep that back until the deb is actually generating.....

---

Current state of play (n.b. Please ignore the version number etc, I just needed something to stuff in there to keep the scripts happy) https://github.com/leezer3/OpenBVE/commit/afd9ea90bcaf937d73fc7a3024314a45089422d9

This is the current test commit. At the minute, dpkg-buildpkg is failing with the following (truncated for brevity) output:

	Could not read /mnt/downloads/SourceCode/debian/openbve.substvars
	Could not read /mnt/downloads/SourceCode/debian/openbve.substvars
	Could not read /mnt/downloads/SourceCode/debian/openbve.substvars
	(grep -a -s -v cli:Depends debian/openbve.substvars; echo "cli:Depends=libc6 (>= 2.24) | libc6.1 (>= 2.24) | libc0.1 (>= 2.24), libgl1-mesa-glx, libmono-corlib4.5-cil (>= 4.6.1.3), libmono-system-core4.0-cil (>= 4.6.1.3), libmono-system-drawing4.0-cil (>= 4.6.1.3), libmono-system-windows-forms4.0-cil (>= 1.0), libmono-system-xml-linq4.0-cil (>= 4.2.0), libmono-system-xml4.0-cil (>= 4.6.1.3), libmono-system4.0-cil (>= 4.6.1.3), libopenal1, libx11-6 (>= 2:1.6.0), libxcursor1 (>> 1.1.2), libxi6 (>= 2:1.6.99.1), libxinerama1, libxrandr2 (>= 2:1.5)") > debian/openbve.substvars.new
	mv debian/openbve.substvars.new debian/openbve.substvars
	grep -a -s -v '^cli:Recommends=' debian/openbve.substvars > debian/openbve.substvars.new || true
	mv debian/openbve.substvars.new debian/openbve.substvars
	grep -a -s -v '^cli:Suggests=' debian/openbve.substvars > debian/openbve.substvars.new || true
	mv debian/openbve.substvars.new debian/openbve.substvars
dh_clideps: Error: unresolvable module references or missing shlibs entries, please check above errors!
debian/rules:20: recipe for target 'override_dh_clideps' failed
make[1]: *** [override_dh_clideps] Error 255
make[1]: Leaving directory '/mnt/downloads/SourceCode'
debian/rules:23: recipe for target 'binary' failed
make: *** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

[directhex]

Can you stick the full build log in gist? Some is missing

Also, it looks like you included some temporary files in your commit - you can delete all files with substvars or debhelper in the filename

[leezer3]

https://gist.github.com/leezer3/bf8a4e5fd3ab1485cb35b209cb6d3650

Full build log.

I'm sure there are plenty of other flaws in my build logic, so please don't spend too much time on this. Happy enough to keep fiddling myself once I understand it a little better :)

[directhex]

dllmap issues - the installed assemblies dllimport native libs whose packages can't be determined - e.g. ole32.dll (unsurprising, since that's Windows only)

You can either exclude the named things by adding --exclude-moduleref calls to the override_dh_clideps target in debian/rules, or add them to a dllmap , or ideally a combination of both. i.e. dllmap for libXxf86vm -> libXxf86vm.so.1, exclude for /System/Library/Frameworks/ApplicationServices.framework/Versions/Current/ApplicationServices

You also have one shlibs issue for libSDL2-2.0.so.0 - change the Depends: line in debian/control to Depends: ${cli:Depends}, ${shlibs:Depends}, ${misc:Depends} for full automatic dependency resolution, including a fix for the missing libSDL2 dependency

[leezer3]

Thanks.

I've got it building now, although I had to disable EGL, KVM and some other stuff from checking. I don't think that matters, as we don't use these features, they're just present in the openTK dll we import?

TODO:

• Exclude the data assets from the built deb. I thought excluding them from the source tarball would work, but nope. Complicates things marginally that we store our plugins in this directory, else I'd just delete as a post-build step.
• Related- lintian is still complaining about the permissions of the data. I think the entire folder needs a chmod to 644 in GIT.
• ~~Add the launch binary. I think this probably wants to be created as a script somewhere and then called from the debian/install script.~~
• ~~Create a changelog. How much do Debian want here; There are a lot of large changes since the last build they had. Possibly just use the new upstream version?~~
• Who should be packaging / signing?
• Related, we need to figure out who the maintainer actually is. We have an offer from henrich, @sladen is still I think the current maintainer, or what?

[leezer3]

openbve_1.6.0.0-1_all.zip

Test generated deb & the two other generated files. Note that it's unsigned, and the version number is temporary.

[ginga81]

・Who should be packaging / signing? I think that this is may be yourself(Mr.leeser3).

・Related, we need to figure out who the maintainer actually is. We have an offer from henrich, @sladen is still I think the current maintainer, or what? I think that if the build is completely new, someone who candidates choose our project. I think may be that we cannot choose the new maintainer directly. Mr. henrich is only become the sponser of WNPP, not a maintainer.

[ginga81]

To create deb package more easily, may be helps this tool? https://lintian.debian.org/ This tool is written here. https://www.clear-code.com/blog/2014/4/3.html

[sladen]

For the record, the Maintainer: is a shared team:

• Maintainer: Debian CLI Applications Team <[email protected]>

[ginga81]

Mr.leeser3, if you are ready to deb package, please send the email to [email protected] with WNPP template, and report me. I tweet to Mr.henrich.

[leezer3]

I'm still working on it in the linked branch :)

We now have basic wrappers for the Object Viewer / Route Viewer binaries, but for some reason the menu entry isn't triggering correctly. Other than that, the data needs sorting and packaging correctly too (assuming that is Debian want to retain the two package structure)

When I'm done, the linked branch will get merged into mainline, and hopefully added to the CI. At that point, we should be ready to release the 'real' 1.6.0.0, which is what will want submitting to Debian :)

[ginga81]

I heard that this marge is ready to come back to debian from @s520 . Are you ready for WNPP and send to [email protected]? If so, I will tweet to henrich.

[leezer3]

It seems to build the deb OK on Stretch, but not on Jessie. Haven't had a chance to investigate yet.....

[s520]

Jessie's support is LTS, so I don't think we need to worry too much. As you know, LTS only makes security updates. So I think the problem is whether there is a problem with Buster.

[leezer3]

Jessie wasn't building due to a lack of SDL2, so that's minor :)

RFP bug now filed here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=927278

I think the next step is for me to register on Debian mentors, & see if I can figure out how to get the built deb up there: https://mentors.debian.net/

[ginga81]

I tweeted to Mr.henrich to get in touch with. Please wait.

[ginga81]

From Mr.henrich, he said that he will check this to be a sponsor at least weekend. Please wait.