Export/view all files per zone/instance

[Symphoneum]

Hi Kruithne,

I think it would be really useful to add a feature that would let you search for and maybe even batch export all files called within a certain instance or zone. For example, if I wanted to take all the sound files that are used in the classic Deadmines map and export them, it would be really useful to have a feature like this. It would also be useful to quickly figure out what goes in each zone in terms of creatures and buildings without spending tons of time researching on wow wiki or fandom. Anyway, hope this sounds interesting and that I put it in the right place!

[Marlamin]

Can you expand on this? What type of sounds? Music, ambience and the sound any models in the map make would be theoretically possible for example, but additional stuff like NPC dialog isn't referenced in the client.

[Symphoneum]

I guess I was hoping for anything and everything that might be possible to parse together. I didn't realize dialogue wasn't referenced in the client, that certainly is disappointing but not surprising. Personally, I would like to take any voice lines that might exist from certain instances like the deadmines or Onyxia's lair. I know there aren't many, if any at all, but I'm also pretty sure if there are they are not labeled well and I might miss some so a feature like this would guarantee that I don't miss any surprise lines. But if dialogue is not referenced then at least I could get music and ambiance files as well as a list of models. I'm sure there are other uses for this too, right now I'm just hunting for every line of dialogue and sound made by Onyxia's voice actress and I know this would be helpful not to miss any stray files.

[Kruithne]

Adding some kind of export feature where you can take a map and have all of the associated sound files exported is something we could potentially add, although we'll need to put some thought into how it's done.

As Marlamin said, we're probably not going to be able to add something that will gather up all of the NPCs within the map and then list/export their sound files, at least not without jumping through some extra hoops.

I'm going to assume based on what you said that you're gathering voice lines to train an AI synthesizer. A solution does spring to mind for being able to gather up all of the necessary files though.

1. Given the ID of an NPC, such as Onyxia being 10184. Make a GET request to https://www.wowhead.com/npc=X where X is the NPC ID.
2. Inside one of the script blocks in the returned HTML, there's a new Listview constructor which takes an object with the ID sound. Extract the object and parse it as a JS object.
3. Iterate over the object.data array. Each of those is an entry on the "Sounds" tab. Each one has a files property which contains each of the sound files (since a sound can have multiple).
4. For each one, either take the ID (object.data[x].files[x].id) and export it via wow.export; you can automate this using our RCP API if you want. Alternatively, you can use the .url from the object to download the file directly from Wowhead (be mindful of this and don't abuse their bandwidth). The URLs are extensionless, but the MIME type is included as .type, so you can either deduce it from that or just assume they're all .ogg.

Here's a quick prototype written in NodeJS.

// IMPORTS
import https from 'node:https';
import fs from 'node:fs';
import path from 'node:path';

// CONFIG
const npcID = 10184; // Onyxia.
const whHost = 'https://www.wowhead.com';
const whURL = `${whHost}/npc=${npcID}`;

// CODE
const fetchHTML = async (resURL) => {
	  return new Promise((resolve, reject) => {
		https.get(resURL, (res) => {
			if (res.statusCode === 301) {
				// Follow 301 redirects since Wowhead decorate URLs.
				fetchHTML(whHost + res.headers.location).then(resolve).catch(reject);
			} else {
				let data = '';
				res.on('data', chunk => data += chunk);
				res.on('end', () => resolve(data));
			}
		}).on('error', (err) => {
			reject(err);
		});
	});
};

// Fetch HTML from Wowhead.
const html = await fetchHTML(whURL);

// Find the sound Listview in the source code.
const regex = /new Listview\({template: 'sound',.*?}\);/g;
const matches = html.match(regex);
let soundData = matches[0];

// Naive way to remove the function call just to get the raw object.
soundData = soundData.substring(13, soundData.length - 2);

// Parse the sound data as a raw JS object (not JSON).
// We need to fudge tabsRelated, WH.Types.getUpperPlural and WH.Types.Sound to get it to parse.
soundData = eval(`const tabsRelated = ''; const WH = { Types: { getUpperPlural: () => {}, Sound: '' } }; (${soundData})`);

// Create a folder using the NPC ID to store the sounds in.
await fs.promises.mkdir(npcID.toString());

// Iterate over the soundData.data, each entry is a sound from the sounds tab.
for (const sound of soundData.data) {
	// sound.name is the name of the sound, such as DragonPreAggro.
	// sound.id is the ID of the sound, not the file ID.

	for (const file of sound.files) {
		// file.id is the fileDataID of the sound file.
		// file.title is the name of the sound file.
		// file.type is the mime type (audio/ogg; codecs="vorbis").
		// file.url is the URL to the sound file on Wowhead.

		// Get the file extension from the mime type.
		// Extend this if you want to support more file types.
		let fileType = '';
		if (file.type.startsWith('audio/ogg'))
			fileType = '.ogg';

		const storePath = path.join(npcID.toString(), file.title + fileType);
		const fileStream = fs.createWriteStream(storePath);

		console.log(`Downloading ${file.title}...`);

		// Download the file.
		await new Promise(resolve => {
			https.get(file.url, (res) => {
				res.pipe(fileStream);
				res.on('end', resolve);
			});
		});
	}
}

Keep in mind that older NPCs such as Onyxia don't seem to have their actual voice dialog lines on the sounds tab, so this falls apart. However, it does pick them up for more modern NPCs such as Whitemane. My theory here is that Classic raids are using server-side scripts to throw the voice lines out, rather than having an association in the data tables.