twitter-watch
twitter-watch copied to clipboard
🐦 Highly configurable Twitter data collection framework
Twitter Watch
A framework for Twitter data collection.
- core contains the code for data collection
- ui contains the code for the UI, it can be automatically deployed to gh-pages. you can also use the original UI client if you deployed your own core collection by changing the access URL to your own.
For a full description, please check the preliminary version of Twitter Watch's paper.
Architecture
The current version merges the core and api containers but you can use the version in the original architecture by using the docker-compose-2.yml file and commenting the second line in the launcher.sh file (nohup python api/main.py > logs_flask.txt &
).
UI
Implementation
The overall implementation is a sequential execution of groups of tasks until the scheduled (and parallel) tasks are reached.
Practical results
This tool was used to collect data on the Portuguese Twittersphere, the following figure summarizes the amount of accounts and tweets it collected.
Tips and Tricks
docker compose deploy
-
cp example.env .env
and edit -
docker-compose up
(pass-d
for detached mode)
mongo dumps
docker exec some-mongo sh -c 'exec mongodump -d twitter --archive' > PATHTOLOCALFILE/dump.archive
docker exec -it twitter-watch_core_1 bash
Pre-commit
Check pre-commit.com for more pre-commit functionality and then add it to the pre-commit config file.
To run, execute pre-commit run --all-files
.