There are some questions about game-detail-crawler

[GrandDuke1106]

I have now collected over 10 million game information through game-abstract-crawler and now I am trying to use game-detail-crawler to get detailed information to use in kanachan training. But there is a little problem with the Archiver of game-detail-crawler, probably because the acquired game information is too new.

Failed validating 'enum' in schema['properties']['head']['properties']['accounts']['items']['properties']['title']:
    {'description': '0: 称号なし, 600002: 原初の火, 600003: (詳細不明), 600004: 最初の雀聖, '
                    '600005: 魂の契約者1, 600006: 魂の契約者2, 600007: 魂の契約者3, '
                    '600008: 魂の啓発者1, 600009: 魂の啓発者2, 600010: 魂の啓発者3, '
                    '600011: 魂の創造者1, 600012: 魂の創造者2, 600013: 魂の創造者3, '
                    '600014: 魂の超越者, 600015: キャットフード商人, 600016: 最初の魂天, '
                    '600017: 公認プレイヤー, 600018: (詳細不明), 600019: (詳細不明), '
                    '600020: (詳細不明), 600021: にゃん国の守護者, 600022: (詳細不明), '
                    '600026: 公認プレイヤーG, 600027: (詳細不明), 600028: 1st Season '
                    '挑戦者, 600029: インターハイ王者, 600031: 成長の旅, 600032: 連勝の道, '
                    '600035: 千磨百錬, 600036: (詳細不明), 600037: (詳細不明), 600038: '
                    '(詳細不明), 600042: 麒麟位2021, 600044: 花より団子 (にじさんじ麻雀杯 '
                    '花鳥風月戦 優勝賞品), 600045: (詳細不明), 600046: (詳細不明), 600047: '
                    '(詳細不明), 600048: 伝説の名コンビ (アカギコラボ大会優勝ペア)',
     'enum': [0,
              600002,
              600003,
              600004,
              600005,
              600006,
              600007,
              600008,
              600009,
              600010,
              600011,
              600012,
              600013,
              600014,
              600015,
              600016,
              600017,
              600018,
              600019,
              600020,
              600021,
              600022,
              600026,
              600027,
              600028,
              600029,
              600031,
              600032,
              600035,
              600036,
              600037,
              600038,
              600042,
              600044,
              600045,
              600046,
              600047,
              600048],
     'title': '称号'}

For example, the error message above basically occurs on every record.

Failed validating 'enum' in schema['properties']['head']['properties']['accounts']['items']['properties']['avatar_frame']:
    {'description': '0: 無し, 305500: 若葉, 305510: 「四象戦 -夏の陣-」優勝賞品？, 305511: '
                    '(詳細不明), 305512: 「四象戦 -秋の陣-」優勝賞品？, 305514: (詳細不明), '
                    '305515: 「四象戦 -冬の陣-」優勝賞品？, 305516: (詳細不明), 305517: '
                    '(詳細不明), 305518: 「2021雀魂四象戦 ～春の陣～」優勝賞品？, 305519: '
                    '「2021雀魂四象戦 ～夏の陣～」優勝賞品？, 305520: 試練の強者 (試練の道 1st '
                    'Season ランキング11～100位報酬), 305521: 試練の賢者 (試練の道 1st '
                    'Season ランキング2～10位報酬), 305522: 試練の覇者 (試練の道 1st Season '
                    'ランキング1位報酬), 305523: 猫ちゃん軍団長, 305524: (詳細不明), 305525: '
                    '「2021雀魂双聖戦 一回戦」優勝賞品？, 305526: (詳細不明), 305527: '
                    '「2021雀魂四象戦 ～秋の陣～」優勝賞品？, 305528: (詳細不明), 305529: 秋霜の剣, '
                    '305531: (詳細不明), 305533: (詳細不明), 305534: (詳細不明)',
     'enum': [0,
              305500,
              305510,
              305511,
              305512,
              305514,
              305515,
              305516,
              305517,
              305518,
              305519,
              305520,
              305521,
              305522,
              305523,
              305524,
              305525,
              305526,
              305527,
              305528,
              305529,
              305531,
              305533,
              305534],
     'title': 'アイコンフレーム'}

And this error message comes up frequently

May I ask how this situation can be resolved?

[Cryolite]

I have added support for disabling of validation for game records into the feature/disable-game-detail-validation branch. Could you try the following command in the feature/disable-game-detail-validation branch?

DOT_AWS_DIR=${DOT_AWS_DIR} AWS_PROFILE=${AWS_PROFILE} MAHJONGSOUL_SNIFFER_GAME_DETAIL_DISABLE_VALIDATION=1 docker compose up

[GrandDuke1106]

Before starting, need to change something in the game-detail-crawler/crawler.Dockerfile file

On line 17

   pip3 install -U pip && pip3 install -U \
      boto3 \
      jsonschema \
      mitmproxy \
      pyyaml \
      redis \
      selenium && \

need to change it to the following

   pip3 install -U pip && pip3 install -U \
      boto3 \
      jsonschema \
      mitmproxy==7.0.3 \
      pyyaml \
      redis \
      selenium \
      Werkzeug==2.2.2 && \

No further problems at this time

[GrandDuke1106]

Sometimes it fails to fetch Solved by restarting the container at regular intervals currently