semaphore

semaphore copied to clipboard

Hi I am a beginner in ansible semaphores. I tried using the ansible semaphore function. The "task" count limit is not show in the web ui. I would like to be able to limit the number of tasks due to issues such as disk space. For example, to keep only the history of the last 10 "tasks". I confirmed that task is removed in the web ui, when I delete the row of the "task" table from the database. I ask if it is possible to limit the "task" count in the web ui.

Hi @battlecook Currently it is not possible. But it can be implemented in future. I mark this issue as feature request.

Hi @battlecook Currently it is not possible. But it can be implemented in future. I mark this issue as feature request.

we want to use semaphores in the production environment. But mysql database increase fast. Each log is treated as a row of in task_output table

Greetings!

I've given this a little thought:

The main issue I see is that the logs spam the database as @DesireWithin metioned. A first step would be to create a config-variable like task_log_database_lifetime_days (integer) and task_log_database_cleanup (boolean) and create a backend job that removes old entries from database.

My issue with that solution is - one might occasionally need to view older logs.

• They are also important to trace user actions.
• There is currently no good way of archiving existing logs (security) except from dumping the database and parsing it later on

I personally would solve it like this:

• Add config-variables:

  "task_log": {
    "cleanup_database": true,
    "cleanup_file": true,
    "lifetime_database_days": 30,
    "lifetime_file_days": 365,
    "dump_to_file": true,
    "dump_directory": "/var/log/semaphore/tasks",
    "dump_schedule_days": 7,
  }
  

• If dump_to_file is disabled - if cleanup_database is set to true, we have the same behavior as in the 'simple solution' (just database cleanup)

• If dump_to_file is enabled:

  • Add a backend schedule that runs every dump_schedule_days and processes task-logs from database that are older than lifetime_database_days (max. logs in database would be dump_schedule_days + lifetime_database_days)

    • These logs are pulled and formatted as JSON (optimally in a format that can be sent 1-to-1 to the frontend later on)

      {  // field names are just placeholders - should use whatever makes sense most
        "task-template-id": [
          {"task-id": 50, "task-logs": [
            {"log-id": 1, ...},
            {"log-id": 2, ...}
          ]},
          {"task-id": 51, "task-logs": [
            {"log-id": 1, ...},
            {"log-id": 2, ...}
          ]}
        ]
      }
      

    • That JSON data is dumped to dump_directory - one file per task-template (and compressed using xz/gz)

      Optimally the format would also be human-readable (as shown in the example above: one log line per file-line

    • If cleanup_file is enabled we would also need a cleanup job for those

      • If cleanup_file would be set to false, users would also be able to manage the cleanup using rsyslog/logrotate or some other way

  • In the web frontend there could be two buttons at the bottom of the task-history list: show more and show all

  • The frontend could call the backend which loads the next file and send the loaded JSON data back

    • If the format of the JSON inside the dump files is already the correct one - this would not even take that much processing power
    • Note: There could be an issue if the database-schema changes and logs are still in an older format.

Loading logs from files is - of course - much slower than from database, but I don't think speed will be an issue for this use-case.

As I mentioned: just an idea (;

- AnsibleGuy

Hi,

I'm very interested in this feature too, as we want to use in in a production environment, and want to be in control of used space on disk. Is there any progress on it ? I'm not a developer, but if I can help in any way, like testing or something, feel free to ask !

Thank you !