twitter-to-sqlite icon indicating copy to clipboard operation
twitter-to-sqlite copied to clipboard

Import archive error 'withheld_in_countries'

Open pauloxnet opened this issue 3 years ago • 0 comments

Importing the twitter archive I received this error:

$ twitter-to-sqlite import archive.db twitter-2021-12-31-<hash>.zip 
birdwatch-note-rating: not yet implemented
birdwatch-note: not yet implemented
branch-links: not yet implemented
community-tweet: not yet implemented
contact: not yet implemented
device-token: not yet implemented
direct-message-mute: not yet implemented
mute: not yet implemented
periscope-account-information: not yet implemented
periscope-ban-information: not yet implemented
periscope-broadcast-metadata: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-expired-broadcasts: not yet implemented
periscope-followers: not yet implemented
periscope-profile-description: not yet implemented
professional-data: not yet implemented
protected-history: not yet implemented
reply-prompt: not yet implemented
screen-name-change: not yet implemented
smartblock: not yet implemented
spaces-metadata: not yet implemented
sso: not yet implemented
Traceback (most recent call last):
  File "/home/paulox/.virtualenvs/dogsheep/bin/twitter-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/cli.py", line 759, in import_
    archive.import_from_file(db, filename, content)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 246, in import_from_file
    db[table_name].insert_all(rows, pk=pk, replace=True)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py", line 2625, in insert_all
    self.insert_chunk(
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py", line 2406, in insert_chunk
    result = self.db.execute(query, params)
  File "/home/paulox/.virtualenvs/dogsheep/lib/python3.9/site-packages/sqlite_utils/db.py", line 422, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: table archive_tweet has no column named withheld_in_countries

I found only a single tweet with the key withheld_in_countries in tweet.js that seems the problems:

[
{
    "tweet" : {
      "retweeted" : false,
      "source" : "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>",
      "entities" : {
        "hashtags" : [
          {
            "text" : "NowOnAndroid",
            "indices" : [
              "64",
              "77"
            ]
          }
        ],
        "symbols" : [ ],
        "user_mentions" : [
          {
            "name" : "Periscope",
            "screen_name" : "PeriscopeCo",
            "indices" : [
              "3",
              "15"
            ],
            "id_str" : "1111111111",
            "id" : "222222222"
          }
        ],
        "urls" : [
          {
            "url" : "https://t.co/xxxxxxxxx",
            "expanded_url" : "https://vine.co/v/xxxxxxxxx",
            "display_url" : "vine.co/v/xxxxxxxxxx",
            "indices" : [
              "78",
              "101"
            ]
          }
        ]
      },
      "display_text_range" : [
        "0",
        "101"
      ],
      "favorite_count" : "0",
      "id_str" : "1111111111111111111111",
      "truncated" : false,
      "retweet_count" : "0",
      "withheld_in_countries" : [
        "TR"
      ],
      "id" : "000000000000000000",
      "possibly_sensitive" : false,
      "created_at" : "Fri Aug 14 06:04:03 +0000 2015",
      "favorited" : false,
      "full_text" : "RT @periscopeco: Travel the world. LIVE. The Global Map is here #NowOnAndroid https://t.co/NZXdsPWROk",
      "lang" : "en"
    }
  }
  ]

I solved the error removing the key from the tweet.js but I'm reporting this error to improve the project.

pauloxnet avatar Jan 01 '22 16:01 pauloxnet