crate-digger
crate-digger copied to clipboard
Java library for fetching and parsing rekordbox exports and track analysis files.
= Crate Digger James Elliott [email protected] :icons: font :experimental:
// Set up support for relative links on GitHub, and give it // usable icons for admonitions, w00t! Add more conditions // if you need to support other environments and extensions. ifdef::env-github[] :outfilesuffix: .adoc :tip-caption: :bulb: :note-caption: :information_source: :important-caption: :heavy_exclamation_mark: :caution-caption: :fire: :warning-caption: :warning: endif::env-github[]
+++
+++
image:https://img.shields.io/badge/chat-on%20zulip-brightgreen[Chat on Zulip,link="https://deep-symmetry.zulipchat.com/#narrow/stream/275855-dysentery-.26-crate-digger"]
A Java library for fetching and parsing rekordbox media exports and track analysis files.
image:https://img.shields.io/badge/License-Eclipse%20Public%20License%202.0-blue.svg[link="#licenses"] or secondary options image:https://img.shields.io/badge/License-Mozilla%20Public%20License%202.0-blue.svg[link="#licenses"] image:https://img.shields.io/badge/License-LGPL%203-blue.svg[link="#licenses"]
This project uses the http://kaitai.io[Kaitai Struct] compiler with the help of a https://github.com/valery1707/kaitai-maven-plugin[Maven plugin] to create classes that can parse and output binary data structures in a convenient and efficient way. It uses them to create Java classes, because it was created to support https://github.com/Deep-Symmetry/beat-link[Beat Link], but other projects can use them to output code for other languages.
It also uses the jrpcgen
tool (which is part of the
https://sourceforge.net/projects/remotetea/[Remote Tea project]),
via another plugin, to generate classes that know how to talk to the
non-standard NFSv2 file servers running in Link-capable players, so
the rekordbox data can be reliably obtained even during big shows
where all four players are in use.
== PDB Database
The file link:src/main/kaitai/rekordbox_pdb.ksy[rekordbox_pdb.ksy
] contains
the structure definitions needed to parse exported rekordbox databases
(export.pdb
files).
NOTE: Huge thanks to https://github.com/flesniak[Fabian Lesniak] for figuring out the details of how to interpret these files in his https://github.com/flesniak/python-prodj-link[python-prodj-link] project and https://github.com/GreyCat[Mikhail Yakshin] for helping me quickly learn the more subtle aspects of Kaitai Struct. And this was all started by a question https://reverseengineering.stackexchange.com/users/4599/evan-purkhiser[Evan Purkhiser] posted on https://reverseengineering.stackexchange.com/questions/4311/help-reversing-a-edb-database-file-for-pioneers-rekordbox-software[Stack Exchange].
There is an https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export Structure Analysis] site describing the details of what we have learned about these file formats. Reading that will help make sense of the exploration tools and the objects returned by this library.
[[exploring-analysis]] === Exploring the Analysis
One of the amazingly cool things about Kaitai Struct is that you can
use its https://ide.kaitai.io/#[Web IDE] to see how the structure
definitions work and visually explore the contents of files you are
analyzing. This also means you can look inside your own .pdb
files
and check my work, or get a better understanding of how to use the
generated parsers. To do that, simply upload the .pdb
file you want
to examine to the Web IDE (it doesn't actually go to the web, it just
gets put in your local browser storage), then also upload my
link:src/main/kaitai/rekordbox_pdb.ksy[rekordbox_pdb.ksy
] file, and
the Web IDE will parse the exported database, letting you explore the
structures in the tree view, and see the corresponding raw bytes in
the hex viewer.
TIP: You can find the export.pdb
file on a media stick prepared by
rekordbox inside the PIONEER
folder, which may be invisible in the
Finder, but you can open it using Terminal commands if you have to. To
download my
link:src/main/kaitai/rekordbox_pdb.ksy[rekordbox_pdb.ksy
], click on
its link, then click on the Raw
button in the header above the first
line of the listing, then tell your browser to save it to disk. Be
sure to keep the .ksy
extension. Then you can upload it to the
Kaitai Struct Web IDE.
== ANLZ Data
Each track in a rekordbox database also has ANLZnnnn.DAT
and
ANLZnnnn.EXT
files associated with it, containing the beat grid, an
index allowing rapid seeking to any time in variable-bit rate audio
files, the waveforms, memory cues and loop points. The paths to these
files are found inside the corresponding track record.
The structure definitions for these files are in
link:src/main/kaitai/rekordbox_anlz.ksy[rekordbox_anlz.ksy
]. You can
use it with the Kaitai Struct Web IDE as described
<<exploring-analysis,above>> to explore analysis files found in
your own exported media.
== Using the Library
Crate Digger is available through Maven Central, so to use it in your Maven project, all you need is to include the appropriate dependency.
Click the maven central badge above to view the repository entry for crate-digger. The proper format for including the latest release as a dependency in a variety of tools, including Leiningen if you are using beat-link from Clojure, can be found in the Dependency Information section.
There are two halves to what Crate Digger offers. The first is an ability to talk to the nonstandard Network File System servers that are running in Pioneer players, and ask them to deliver the rekordbox data export and track analysis files (programs like Beat Link need these files to provide smooth integrations with the music being performed). The second half is the ability to parse the contents of those files, as described above.
=== Retrieving Files
The class org.deepsymmetry.cratedigger.FileFetcher
is a singleton,
so to work with it you will start by calling getInstance()
, as is
customary. Retrieving a file is then as simple as this:
[source,java]
FileFetcher fetcher = FileFetcher.getInstance(); fetcher.fetch(playerAddress, mountPath, sourcePath, destination);
playerAddress
is an InetAddress
object holding the address of the
player from which you want to download a file. mountPath
identifies
the media slot you want to get information from, as shown in the table
below. sourcePath
is the path to the specific file you want within
the mounted media, and destination
is a File
object identifying
where you want the downloaded data to be stored.
.Media Slot Mount Paths
|===
|Media Slot | Mount Path
|SD
|/B/
|USB
|/C/
|===
[IMPORTANT]
The FileFetcher
caches information about players to make
requests more efficient, so it is important for you to tell it when a
player goes away, or unmounts one of its media slots, by calling:
[source,java]
fetcher.removePlayer(playerAddress);
====
=== Parsing Structures
The class org.deepsymmetry.cratedigger.Database
provides support for
accessing the contents of rekordbox database export files. You can create
an instance to wrap a File
instance that contains such an export (for
example one that you downloaded using the fetch
method above). Then
you can query it for track and other information:
[source,java]
Database database = new Database(downloadedFile); RekordboxPdb.TrackRow track = database.findTrack(1); System.out.println(database.getText(track.title()));
Strings (like titles, artist names, etc.) are represented by a variety
of structures with different encodings, so a getText()
method is
provided to convert them into ordinary Java strings.
See the http://deepsymmetry.org/cratedigger/apidocs/[API documentation] for more details about these classes, and the https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html[Export Structure Analysis] for more details about the file formats.
== Logging
Crate Digger uses http://www.slf4j.org/manual.html[slf4j] to allow you to integrate it with whatever Java logging framework your project is using, so you will need to include the appropriate slf4j binding on your class path.
== Unfinished Tasks
-
There are still more tables to be figured out.
Columns
looks like the list of things that can be searched by, so perhaps it will hold some clues for how to find and use the index tables, which must exist because it would be horribly slow for the players to do a linear scan through the main sparse tables whenever they wanted a record. -
If we could figure out how to use the indices ourselves, we could avoid having to load the whole file and index it ourselves.
== Building the source
As noted above, he Maven project uses a plugin to run the the
jrpcgen
tool (which is part of the
https://sourceforge.net/projects/remotetea/[Remote Tea
project]) to generate Java classes to implement the
ONC RPC specifications found in src/main/rpc
.
(These are used for communicating with the NFS servers
in CDJs.) It also uses the http://kaitai.io[Kaitai Struct Compiler]
through another plugin to generate Java classes that can parse the
rekordbox databases it downloads from the players, based on the
specifications found in src/main/kaitai
.
These things happen for you automatically during the code generation phase of the Maven build. If you want to use something other than Maven, you will need to figure out how to configure and run the tools yourself.
== Building the Structure Analysis
I started out using pdfLaTeX to write and format the document, but then, at the recommendation of one of the Kaitai Struct developers, switched to XeLaTeX in order to take advantage of newer features. But over time some of the packages I was using, especially for tables, became unsupported and started having issues. So this and the dysentery project's protocol analysis document have been ported to more modern Asciidoc source in the form of https://antora.org[Antora] sites.
To re-create (and even improve on) the byte field diagrams I was able to achieve in LaTeX, I ended up writing my own diagram generator, https://github.com/Deep-Symmetry/bytefield-svg[bytefied-svg], which runs as an Antora plugin with the help of David Jencks' https://gitlab.com/djencks/asciidoctor-generic-svg-extension.js[generic-svg-extension].
This documentation site can be built alongside the dysentery project's protocol analysis, by following the directions in https://github.com/Deep-Symmetry/dysentery/blob/master/doc/#readme[that project].
== Contributing
If you have ideas, discoveries, or even code you’d like to share, that’s fantastic! Please take a look at the link:CONTRIBUTING.md[guidelines] and get in touch!
== Licenses
++++++
Copyright © 2018–2022 http://deepsymmetry.org[Deep Symmetry, LLC]
Distributed under the https://opensource.org/licenses/EPL-2.0[Eclipse Public License 2.0]. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software. A copy of the license can be found in link:LICENSE[LICENSE] within this project.
Secondary Licenses: This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: https://www.mozilla.org/en-US/MPL/2.0/[Mozilla Public License 2.0], or https://opensource.org/licenses/LGPL-3.0[GNU Lesser General Public License v. 3].
=== Library and Build Tool Licenses
https://sourceforge.net/projects/remotetea/[Remote Tea] is licensed under the https://opensource.org/licenses/LGPL-2.0[GNU Library General Public License, version 2].
The https://github.com/kaitai-io/kaitai_struct_compiler[Kaitai Struct Compiler] is licensed under the https://opensource.org/licenses/GPL-3.0[GNU General Public License, version 3] and the Kaitai Java runtime embedded in crate-digger is licensed under the https://opensource.org/licenses/MIT[MIT License].