============== Color Code Bot

Share code snippets as beautiful syntax-highlighted images and HTML on Telegram

.. list-table:: :widths: auto :align: center

• • |telegram|
  • |quay|
  • |actions-ctnr|
  • |actions-reqs|
  • |actions-fmt|

It's a small bit of Python glue between great projects, including:

• highlight_ (lua, renders HTML)

• pyTelegramBotAPI_

• Silicon_ (rust, renders image)

• guesslang_ (uses TensorFlow; saves you the step of specifying the snippet's language)

• Iosevka_ (the most wonderful monospaced font)

• Fallback fonts:

  • Symbols Nerd Font
  • NanumGothicCoding
  • OpenMoji

The background image is from Sharon McCutcheon.

Usage

Send @colorcodebot_ the code you want highlighted, as a forwarded or original direct message.

Or, add it to your group and it will send an image of any monospace content sent in the chat.

.. image:: https://user-images.githubusercontent.com/1787385/174667742-32e414b4-e4f4-41f8-ae38-d6d64c1075f2.png :alt: Screenshot of the bot in action :align: right

As a convenience, you can get to a direct chat with it from any other chat, by typing @colorcodebot and tapping the button that pops up. A button returning you (with a shiny new image) to your original chat will be presented after you send the code.

Development & Deployment

The bot should run anywhere with Python, fontconfig, highlight_, Silicon_, and the ability to install TensorFlow. Or anywhere that can run a container image.

Depending on your hardware, you may see faster syntax guessing (from guesslang_) by installing cuda and cudnn packages. This is not done for the currently hosted container images, which is the result of ./mk/ctnr.sh -d prod --push run by a GitHub Action.

Outside of the core Python app, sops_ is used for secrets, buildah_ for container building, GitHub Actions for automated container image builds and other CI tasks, and wheezy.template_ and yamlpath_ are extremely handy for defining+rendering service definitions and other dev/ops maneuvers.

Most of the mk/ and start/ scripts are POSIX, but mk/svcs.zsh requires Zsh, and mk/ctnr.sh calls mk/svcs.zsh.

Please do send a message_ or open an issue with any questions.

Organization

An abbreviated file tree overview:

.. code:: shell

   colorcodebot/
   ├──app/                  # core app that gets deployed
   │  ├──requirements.in    # loosely versioned reqs for the bot
   │  └──sops/              # encrypted deployment-specific data
   ├──vars.<deployment>.yml # unencrypted deployment-specific data
   ├──start/                # scripts that help start the bot
   ├──mk/                   # scripts that make things
   ├──templates/            # used by mk/ scripts to generate files
   └──ops-requirements.in   # loosely versioned reqs for mk/ and start/ scripts

The following are generated by ``mk/`` scripts:

.. code:: shell

   colorcodebot/
   ├──app/
   │  ├──requirements.txt   # mk/reqs.sh     - lockfile for the bot
   │  ├──svcs/              # mk/svcs.zsh    - supervised process definitions for s6 [untracked]
   │  └──theme_previews.yml # mk/file_ids.sh - {theme_name: image_id}                [untracked]
   └──ops-requirements.txt  # mk/reqs.sh     - lockfile for mk/ and start/ scripts

When building a container image with ``mk/ctnr.sh``,
``app`` becomes ``/home/colorcodebot``.

If you want to use the container images already built from this repo,
you'll probably write or mount over:

- ``/home/colorcodebot/theme_previews.yml``
- ``/home/colorcodebot/svcs``
- ``/home/colorcodebot/sops``
- ``/home/colorcodebot/.sops.yaml``

Getting Started

To run colorcodebot.py, the environment variable TG_API_KEY must be set, with a token from @botfather_.

.. code:: console

$ python3 -m venv app/venv $ . ./app/venv/bin/activate $ python -m pip install -r app/requirements.txt $ TG_API_KEY='...' ./app/colorcodebot.py

After chatting with the bot, check the logs for your chat_id. Pass this as an additional environment variable ADMIN_CHAT_ID to get:

• an updated SQLite db file sent to that chat whenever a user sets a preferred theme

Deployments, Secrets, and Scripts

Encrypted Variables
^^^^^^^^^^^^^^^^^^^

Configure Sops
""""""""""""""

Create one or more age_ keys to use with sops_:

.. code:: console

   $ mkdir -p ~/.config/sops/age
   $ printf '%s\n' '' '# --- colorcodebot ---' >>~/.config/sops/age/keys.txt
   $ age-keygen >>~/.config/sops/age/keys.txt
   Public key: age1r50agxl277e24h4ammj0kvpqh224ut8ds67qc2d537dq0uy74shq98dh97

And use that public key in ``.sops.yaml`` to match your desired deployments.

Write colorcodebot Variables
""""""""""""""""""""""""""""

Overwrite ``app/sops/colorcodebot.<deployment>.yml`` with

.. code:: yaml

   TG_API_KEY: <put-the-real-token-here>

(and optionally ``ADMIN_CHAT_ID``) and encrypt it with

.. code:: console

   $ sops -e -i app/sops/colorcodebot.<deployment>.yaml

.. You can set ``host`` and ``port`` in ``app/sops/papertrail.<deployment>.yml``
.. the same way, if using that service.

Load colorcodebot Variables
"""""""""""""""""""""""""""

.. code:: console

   $ ./start/local.sh -h
   Start the bot locally, without process supervision or other svcs
   Args: [-d <deployment>=dev]

You can use ``start/local.sh`` to:

- ensure Python lockfile is updated
- ensure a virtual environment exists
- ensure the venv has all Python dependencies installed
- ensure the venv is activated if one is not already
- update or create ``app/theme_previews.yml`` if file IDs are present in ``vars.<deployment>.yml``
- load decrypted values from ``app/sops/colorcodebot.<deployment>.yml`` into environment variables
- launch the bot (unsupervised, no other services)

You can do just those last two (as seen in the script) with

.. code:: console

   $ sops exec-env "app/sops/colorcodebot.${deployment}.yml" app/colorcodebot.py

Unencrypted Variables
^^^^^^^^^^^^^^^^^^^^^

A deployment's unencrypted variables are defined by ``vars.<name>.yml``.

There are two top-level keys:

``theme_previews``
  mapping of theme names to Telegram file IDs; see `Generating Theme Previews`_

  used by: ``mk/file_ids.sh``, ``mk/ctnr.sh``

``svcs``
  list of mappings that each define a long-running supervised service
  (the bot and optionally a log sender for Papertrail_)

  used by: ``mk/svcs.sh``, ``mk/ctnr.sh``

The deployments ``dev`` and ``prod`` are both intended to run inside a container,
built by ``mk/ctnr.sh``.
Note the difference between the ``svc`` definitions
of ``vars.dev.yml`` and ``vars.prod.yml``:

.. code:: diff

   --- vars.dev.yml  2021-06-28 11:13:46.347838948 -0400
   +++ vars.prod.yml 2021-07-12 14:22:07.638842356 -0400
   @@ -4,7 +4,7 @@
        exec: >-
          sops exec-env
   -      sops/colorcodebot.dev.yml
   +      sops/colorcodebot.prod.yml

          "s6-setuidgid colorcodebot ./venv/bin/python
          ./colorcodebot.py"
   @@ -16,7 +16,7 @@
        exec: >-
          sops exec-file --filename log_files.yml
   -      ../log_files.dev.yml
   +      ../log_files.prod.yml

          "remote_syslog -D -c {}"
   @@ -24,7 +24,7 @@
        sops_templates:
          - src: papertrail.log_files.yml.wz
   -        dest: log_files.dev.yml
   +        dest: log_files.prod.yml

- differences:
   + which encrypted variables get set in the environment of the bot process
   + which encrypted config file is created for and read by the remote logger

Now let's compare ``vars.dev.yml`` to ``vars.local.yml``:

.. code:: diff

   --- vars.dev.yml  2021-06-28 11:13:46.347838948 -0400
   +++ vars.local.yml   2021-07-12 13:57:00.414719676 -0400
   @@ -6,14 +6,15 @@
   -      "s6-setuidgid colorcodebot ./venv/bin/python
   +      "./venv/bin/python
          ./colorcodebot.py"
        folder:
          run: ../../
          log: ../../../logs/colorcodebot
   +      cgroups: /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/svcs

      - name: papertrail
   -    enabled: true
   +    enabled: false
   @@ -22,6 +23,7 @@
        folder:
          run: log
          log: ../../../logs/papertrail
   +      cgroups: /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/svcs

- similarities:
   + which encrypted configs are used
- differences:
   + ``local``: no user changing (no ``s6-setuidgid``)
   + ``local``: overrides the default cgroup path used by services with a systemd-flavored one
   + ``local``: disables optional Papertrail remote logging service

Modify one of these to your liking, or copy to ``vars.<name>.yml`` with your own deployment name, e.g.:

.. code:: console

   $ cp vars.local.yml "vars.$(hostname).yml"

Generating Theme Previews
~~~~~~~~~~~~~~~~~~~~~~~~~

---

OUTDATED!

TODO: Fill in the missing steps in this section
now that the ``/previews`` command has been removed

---

highlight_ has *many* themes, so we picked a subset.

For the user to choose a theme, we need to generate preview images,
and save their file IDs.

Start by creating ``app/theme_previews.yml`` either manually or with ``./mk/file_ids.sh``

.. code:: console

   $ ./mk/file_ids.sh -h
   Generate theme_previews.yml, with data from vars.<deployment>.yml
   Args: [-d <deployment>=dev] [<dest>=app/theme_previews.yml]

For now the value of each entry can be garbage,
what's important is that the keys are the names of the themes you wish to offer.

Send the ``/previews`` command to the bot, and the file IDs you need
will show up in the log as preview images are generated and sent your way.

Enter those into ``vars.<deployment>.yml``,
then generate ``app/theme_previews.yml`` for local deployment with ``mk/file_ids.sh``,
which is automatically called by ``start/local.sh`` and ``mk/ctnr.sh``.


.. _@botfather: https://t.me/botfather
.. _a demo video: https://user-images.githubusercontent.com/1787385/123204250-ae9a0380-d485-11eb-981d-3302220aad58.mp4
.. _age: https://github.com/FiloSottile/age
.. _buildah: https://github.com/containers/buildah
.. _@colorcodebot: https://t.me/colorcodebot
.. _guesslang: https://github.com/yoeo/guesslang
.. _highlight: http://www.andre-simon.de/doku/highlight/highlight.html
.. _Iosevka: https://github.com/be5invis/Iosevka
.. _Papertrail: https://www.papertrail.com
.. _pyTelegramBotAPI: https://github.com/eternnoir/pyTelegramBotAPI
.. _send a message: https://t.me/andykluger
.. _Silicon: https://github.com/Aloxaf/silicon
.. _sops: https://github.com/mozilla/sops
.. _wheezy.template: https://github.com/akornatskyy/wheezy.template
.. _yamlpath: https://github.com/wwkimball/yamlpath


.. |actions-ctnr| image:: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/ci.yml/badge.svg?branch=develop
   :alt: Automated Container Build Status
   :target: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/ci.yml

.. |actions-fmt| image:: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/fmt.yml/badge.svg?branch=develop
   :alt: Format and Lint Status
   :target: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/fmt.yml

.. |actions-reqs| image:: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/reqs.yml/badge.svg?branch=develop
   :alt: Automated Python Requirements Bump Status
   :target: https://github.com/AndydeCleyre/colorcodebot/actions/workflows/reqs.yml

.. |quay| image:: https://img.shields.io/badge/Quay.io-andykluger%2Fcolorcodebot--prod--archlinux-blue?logo=redhat
   :alt: Container Image Repository
   :target: https://quay.io/repository/andykluger/colorcodebot-prod-archlinux?tab=tags

.. |telegram| image:: https://img.shields.io/badge/Telegram-%40colorcodebot-blue?logo=telegram
   :alt: Telegram user @colorcodebot
   :target: https://t.me/colorcodebot