KeyError: 'tags'

[kisenera]

I get this error when trying to run this with danbooru2021

Traceback (most recent call last):
  File "danbooru_utility.py", line 497, in <module>
    main()
  File "danbooru_utility.py", line 490, in main
    resize_and_save_images_mp(data_gen, args)
  File "danbooru_utility.py", line 272, in resize_and_save_images_mp
    for example in data_gen:
  File "danbooru_utility.py", line 218, in load_data
    for y in example['tags']},
KeyError: 'tags'

[reidsanders]

Looks like Danbooru2021 changed the metadata format substantially. I'll see what needs to be done to support the new format. As a work around you should be able to use the metadata.json.tar.xz in the main folder which has the old format (data through november per gwern).

rsync --verbose rsync://176.9.41.242:873/danbooru2021/metadata.json.tar.xz ./

[kisenera]

Alright, thanks for the info

[kisenera]

Hello, I downloaded the old metadata but am now getting this error:

Traceback (most recent call last):
  File "danbooru_utility.py", line 497, in <module>
    main()
  File "danbooru_utility.py", line 490, in main
    resize_and_save_images_mp(data_gen, args)
  File "danbooru_utility.py", line 272, in resize_and_save_images_mp
    for example in data_gen:
  File "danbooru_utility.py", line 215, in load_data
    example = json.loads(line)
  File "C:\Users\firewolf\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "C:\Users\firewolf\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\firewolf\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[reidsanders]

I'll look into it. Can you post the command you used that gave that error?

[kisenera]

Sure danbooru-utility --required_tags "2girls" --directory "N:\Downloads3\danbooru2021"

and also happened with the same thing but required_tags replaced with --preview. (I made sure to append the directory of course)

[reidsanders]

You can try the latest commit with the old metadata (the metadata_dir parameter needed to be set to danbooru2021/2021-old). If that doesn't work you might want to double check that you have unpacked the metadata.json.tar.xz inside the danbooru2021 directory and the checksums match:

$ md5sum metadata.json.tar.xz adfac6ba3b254d4e77dd7b0f154815f9 metadata.json.tar.xz