About

This project will automatically encoded fightcade replays and upload them to archive.org: Gino Lisignoli - Archive.org or youtube: fightcade archive

A web site the view the archive.org replays is here: https://fightcadevids.com

fcreplay is primarly a python application run in a docker contaienr used to automate the generation of fightcade replays as video files.

Goal

The goal of this is to make fightcade replays accessible to anyone to watch without using an emulator.

Features

fcreplay has several features to automate the encoding process and add aditional data to the generated videos

Description generation

A desctiption is generated that contains:

1. The fightcade replay id
2. The fightcade player ids
3. The fightcade player locations
4. The game being played
5. The date the game was played
6. (Optional) A appended description

Requirements

• docker
• docker-compose
• pipenv

Database

Fcreplay uses sqlalchemy and has been tested with postgres and is used store any replay metadata

A few more notes:

To trigger recording, the file started.inf is checked. If the file exists then pyautogui is used to start recording the avi file(s)

The i3 window manager is used to ensure that the fcadefbneo window is always in the same place.

This is all done in a headless X11 session inside a docker container

Todo

• Better exception handling.
• Thumbnails are generated but not used by archive.org

Hardware

To run this, you need:

1. A VM or physical machine.
   1. With at least 4 Cores (Fast ones would be ideal)
   2. With at least 4GB Ram
   3. With at least 250GB of storage
      1. This is the amount of temporary storage required to encode a replay of up to 3 hours long. Replay recording requires ~20MB/sec
2. Running docker and docker-compose
3. Some familiarity with python, docker and linux will help

Uploading to youtube.com

To upload files to youtube.com you need to setup a youtube api endpoint. See here: https://github.com/tokland/youtube-upload

Bad words file

To prevent youtube from giving your channel strikes, you should create a 'bad_words.txt' file containing words you wish to block. These will be looked for in the description (so player names, etc). This file needs to be added as a volume to the tasker instance, and as an environment variable to the tasker instance.

Uploading to archive.org

To upload files to archive.org, set the configuration key upload_to_ia to true and configure the ia section in the configuration file. You will also need to have your .ia secrets file in your users home directory. This can be generated by running ia configure from the command line once you have setup the python virtual environment.

Installation and setup

Installation

First, clone this repository:

git clone https://github.com/glisignoli/fcreplay.git
cd fcreplay

Setup your virtual environment using pipenv:

pipenv install
pipenv install --dev

Build the docker images (this step will take a while)

docker-compose build --no-cache

Configuration

Copy the example config from fcreplay/tests/common/config_good.json to ./config.json:

:exclamation: All of the paths inside this file are locatons within the docker containers. They don't need to be changed and will be removed in a future release.

cp ./fcreplay/tests/common/config_good.json ./config.json

Copy the example docker-compose override file from docker-compose.override.yml.example to docker-compose.override.yml

cp ./docker-compose.override.yml.example docker-compose.override.yml

Create the empty log files: These files MUST exist before running any tests or running fcreplay-tasker:

• fcreplay_tasker.log
• fcreplay_check_top_weekly.log
• fcreplay_check_video_status.log
• fcreplay_retry_failed_replays.log
• fcreplay_delete_failed_replays.log

touch fcreplay_tasker.log fcreplay_check_top_weekly.log fcreplay_check_video_status.log fcreplay_retry_failed_replays.log fcreplay_delete_failed_replays.log

Inside the docker-compose.override.yml file, you will need to adjust the variable in the: services.fcreplay-tasker.environment section:

• CLIENT_SECRETS=/path/to/.client_secrets.json
  • Used to authenticate youtube-dl. This is the absolute path to your .client_secrets.json file. This file needs to exist even if you aren't going to upload to youtube.
• CONFIG=/path/to/config.json
  • This is the absolue path to your config.json file
• CPUS=4
  • This is the number of CPU cores to use for each fcreplay recording instance
• DESCRIPTION_APPEND=/path/to/description_append.txt
  • Used to append a static description to youtube uploads. This is the absolute path to your description_append.txt file This file needs to exist even if you aren't going to upload to youtube.
• IA=/path/to/.ia
  • Used to authenticate to internet archive. This is the absolute path to your .ia file. This file needs to exist even if you aren't going to upload to internet archive
• FCREPLAY_NETWORK=fcreplay_postgres,fcreplay_world
  • The names of the docker networks the recording instances will be attached to. You shouldn't need to change this.
• MAX_INSTANCES=1
  • The maximum number of fcreplay recording instances to run at a time
• MEMORY=4g
  • The maximum number of memory available to each fcreplay recording instance.
• ROMS=/path/to/ROMs
  • The absolute path to your fbneo rom files
• YOUTUBE_UPLOAD_CREDENTIALS=/path/to/.youtube-upload-credentials.json
  • Used to authenticate youtube-dl. This is the absolute path to your .youtube-upload-credentials.json file. This file needs to exist even if you aren't going to upload to youtube.
• AVI_TEMP_DIR=/path/to/large/avi_storage_temp
  • The absolute path to where you want fcreplay to use for temp storage. Ideally a location that has a lot of free disk space, as you will need at least 1 gigabyte of free space per minute of storage.
• GET_WEEKLY=false
  • Enable this if you want to automatically get and encode the top weekly replays from https://www.fightcade.com/replay
• BAD_WORDS_FILE=/path/to/bad_words.txt
  • The absolute path to a list of words that will be used to prevet a video from being uploaded to youtube. This file will need to exist even if you arent'y going to upload to youtube.

For the above file, if want to create empty ones, use the following command:

touch .client_secrets.json description_append.txt .ia .youtube-upload-credentials.json bad_words.txt
mkdir avi_storage_temp

You should now be able to run some of the python tests:

pipenv run pytest

To run a full end-to-end encoding test:

pipenv run pytest --runslow -s ./fcreplay/tests/test_functionality.py

:exclamation: For functionality tests to work, the GET_WEEKLY environment variables must be set to true. This can be disabled againe once the functionality tests are finished.

Usage

The typical useage of fcreplay is to run docker-compose up to start the task scheduler. Thist will launch the replay encoding containers when replays are subbmitted to be encoded.

Validating your config

This command will do a basic check on your config:

docker-compose run --rm -v /path/to/config.json:/root/config.json:ro fcreplay-tasker fcreplay config validate /root/config.json

Configurartion information

I might make a more detailed guide on the config.json file but in general the default settings should work fine. I would recomend having a look at config.py for some configuration infromation.

:exclamation: All of the paths inside this file are locatons within the docker containers. They don't need to be changed and will be removed in a future release.

Getting replays

This will download a replay, and place it in the database, marking it ready to be encoded:

docker-compose run --rm -v /path/to/config.json:/root/config.json:ro fcreplay-tasker fcreplay get replay <url>

docker-compose run --rm -v ./config.json:/root/config.json:ro fcreplay-tasker fcreplay get replay https://replay.fightcade.com/fbneo/sf2hf/1653982283355-4647

CLI Interface

If you want a command line interface to interact with the running server, use:

docker-compose run --rm -v ./config.json:/root/config.json:ro fcreplay-tasker fcreplay cli

Docker containers

Some information about the various containers

fcreplay-site

The frontend site used to view replays

fcreplay-tasker

Multi-use container. Typically used as a 'daemon' service that will launch encoding instances when it detects that a replay is available to be encoded.

This container can also be run manually to trigger various command line functions with:

docker-compose run --rm -v ./config.json:/root/config.json:ro fcreplay-tasker fcreplay

fcreplay-tasker-check_top_weekly

Instance of fcreplay-tasker that is used to gather the top weekly replay from https://fightcade.com/replay

Will only run when the environment variable GET_WEEKLY is set to true

fcreplay-tasker-check_video_status

Instance of fcreplay-tasker that is used to check if a video has finish post upload encoding on youtube/archive.org. Videos aren't viewable on fcreplay-site until this is successful

fcreplay-tasker-retry_failed_replays

Instance of fcreplay-tasker that is used to retry failed replays. By default, replays are retried 5 times.

fcreplay-tasker-delete_failed_replays

Instance of fcreplay-tasker that is used to delete failed replays. This usually happens after 5 failures