DDRCardDraw icon indicating copy to clipboard operation
DDRCardDraw copied to clipboard

refactor(script): remove old DDR importer & merge into one script

Open nogic1008 opened this issue 2 months ago • 4 comments

Song data

  • Update name_translation, artist_translation and genre via zenius-i-vanisher
  • Re-order charts by style and diffClass

Scripting

  • Use TypeScript
    • Use constructor fields
  • Change Song & Chart order on sortSongs()
    • Prevent diff by same name songs (ex. リメンバーリメンバー)
    • charts now sorted by style and diffClass (depend on meta property)

Add new Importer class

  • ZivSongImporter

Update ddr-sources meta

  • sortSongs: Use utils/sortSongs (order by song name) or not
  • eagate: Use EAGateSongImporter and define source URL (use on A20 or higher)
  • ziv: Use ZivSongImporter and define song DB URL
  • sanbai: Use SanbaiSongImporter or not (DDR WORLD only)
  • remy: RemyWiki page (unused nowadays)

Define Importer merge strategy

  1. EAGateSongImporter (official KONAMI e-amusement GATE site, only unlocked songs/charts)
    • name, artist (except some songs)
    • jacket
    • folder ("DanceDanceRevolution World" when new song)
    • saHash
    • charts.lvl
  2. Remy wiki
    • remyLink
    • jacket
    • bpm
    • artist
  3. SanbaiSongImporter (3bai icecream songdata)
    • name
    • flags, charts.flags
    • folder
    • jacket
    • name_translation
    • search_hint
    • saHash
    • charts.lvl
    • charts.sanbaiTier
  4. ZivSongImporter (zenius-i-vanisher, only update)
    • name_translation
    • artist_translation
    • genre

nogic1008 avatar Sep 21 '25 12:09 nogic1008

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
ddr-tools Ready Ready Preview Sep 26, 2025 0:55am

vercel[bot] avatar Sep 21 '25 12:09 vercel[bot]

#483 changes are intentionally not included.

Due to the large list of corrections[^1] required, ZivSongImporter might be discontinued in the future.

[^1]: About 300 lines (excluding levels, double if included)

nogic1008 avatar Sep 21 '25 23:09 nogic1008

@nogic1008 is attempting to deploy a commit to the noahm's projects Team on Vercel.

A member of the Team first needs to authorize it.

vercel[bot] avatar Oct 11 '25 08:10 vercel[bot]

This all looks good so far! I'm mostly just skimming through the diff rather than looking at the final form of all this, but I'm happy to see it all being updated this thoroughly.

A couple of notes:

  1. Based on your list of scraping sources and the fields they populate (thank you for doing this, it's very helpful!) I'm tempted to say we don't have any reason to use Skill Attack at all anymore. The official e-gate and sanbai sources are better, and I just didn't know about them at the time I started using SA as a source.
  2. Now that recent versions of node can easily run typescript source without any issues, I'll be looking to convert all the scripts in the project to use true typescript instead of plain js with types in comments. If you're interested in doing that yourself, you're welcome to take it on, but don't feel the need to do so otherwise.

@noahm Thanks for reviews!

Remove Skill Attack from sources

I think so too.

Skill Attack stopped updating midway through DDR A3. And, it is also possible to collect song data for DDR A, 2013-2014 from previous versions, but since there is no way to play these on arcade, it is unlikely that this will be used.

Use pure TypeScript

That's what I was going to suggest after this task. I'll be happy to get started.

nogic1008 avatar Oct 11 '25 08:10 nogic1008

Thanks for taking on such a big upgrade to the data import process!

noahm avatar Nov 14 '25 01:11 noahm