Rake command for resync of recordings (deleted recordings, etc)

[PhilippKilian]

When recordings are deleted on the BBB site (for storage or a policy reasons), Greenlight v3 will never notice, that recordings aren't available anymore.

On CLI, the only command to sync recordings is the rake server_recordings_sync, which will not delete non-existings recordings.

The currently only known method to remove non-existing records from the Greenlight database is via GUI > Administration > All Rooms > Resnyc each room manually (via 3 dots menu), which isn't a way to go.

The problem with old non-existent recordings is the confusion on the end-user site, why some recordings are still available and other aren't.

[rautniraj]

I can suggest a more better way to delete the recordings. This will solve your issue. We can create a separate rake task for this, which will remove recordings older than N days. What you have to do is to setup a cron task which will run this rake everyday. Main idea is to control deleting of recordings from greenlight only so that the sync issue is not there.

Let me know if you need code snippet

[PhilippKilian]

I can suggest a more better way to delete the recordings. This will solve your issue. We can create a separate rake task for this, which will remove recordings older than N days. What you have to do is to setup a cron task which will run this rake everyday.

That would be a usable solution.

Main idea is to control deleting of recordings from greenlight only so that the sync issue is not there.

That's not totally possible. Recordings are stored on a dedicated storage. It's a setup with multiple BBB nodes and a load balancer. Because of GDPR and other policies, recordings cannot be hold forever on this storage. Because other system apart from Greenlight, like Moodle, Mattermost, etc. use recordings too, a server side deleting is necessary.

But it will help al lot if it's possible via Greenlight.

Let me know if you need code snippet

A code snippet would be helpful.

[rautniraj]

If u see app/models/recordings.rb There is dedicated callback defined for after delete action : destroy_bbb_recording

This is responsible for deleting the recordings from storage. Hence the recordings now get deleted from greenlight db as well as storage.

Now this recording file deletion and storage is totally something which i havent explored much personally. I mainly look greenlight side. But we have also implemented recording storage the same way as you have described.

Below is basic code snippet that you can setup :

    cutoff_date = Rails.application.config.recordings_purging_days.days.ago
    
    # Initialize a flag to track if any recordings were found
    recordings_found = false

       Recording.where("recorded_at < ?", cutoff_date)
             .select(:id,:room_id, :name, :record_id, :recorded_at)
             .find_in_batches(batch_size: Rails.application.config.batch_size) do |batch|
                
                recordings_found = true

                batch.each do |record|

                  begin
                    record_instance = Recording.find_by!(id: record.id)
                    record_instance.destroy!

                    info "Successfully deleted & logged recording: #{record.record_id}"
                  rescue => e
                    warning "Failed to delete recording: #{record.record_id} - #{e.message}"
                  end

                  puts
                  sleep(1)
                end
                sleep(1.5)
              end
    warning "No Recordings found older than #{cutoff_date} in the entire query" unless recordings_found

Note -

1. Rails.application.config.recordings_purging_days : this value you can set in production.rb, this will be number of days prior to which all recordings will be deleted. eg 365
2. Rails.application.config.batch_size : Try to set a batch size for your process so that it doesnt consume too much of memory due to eager loading. We have set this value to 500
3. After each batch i have added sleep time of 1.5s. You can tweek this as per your need.
4. Dont use destroy_all function. This doesnot trigger callback defined in your model class. Thats why i have iterated all the records and explicitly called .destroy

record_instance = Recording.find_by!(id: record.id)
record_instance.destroy!

5. In our design we have a separate table that maintains a record of what all recordings got deleted along with their day of deletion. For this i have added select(:id,:room_id, :name, :record_id, :recorded_at) . Can be avoided as per ur need.

You can tweak the above code based on your requirement. I hope it helps. Please update me if it worked or not.

Also if its within your organization policy and if its open for public use, you can please share your greenlight implemented website :)

[PhilippKilian]

If u see app/models/recordings.rb There is dedicated callback defined for after delete action : destroy_bbb_recording

@rautniraj I'm not very deep in ruby code, but I try this process and will give you feedback on how good it works.

Also if its within your organization policy and if its open for public use, you can please share your greenlight implemented website :)

We don't have a specific website, because we provide an API BBB Solution with an optional Greenlight frontend (which is the standard greenlight docker build). We just have an informational website (it's in german).

For handling the recordings, we programmed our own load balancer, which is public here on GitHub B3LB

The housekeeping of the stored recordings can be found here.

[PhilippKilian]

@rautniraj I wrote a small ruby script (my first one 😸) and tested your solution, it works very well, with some minor changes! Because it runs in the greenlight docker container...

• ... it's easier to define an environment variable, instead of custom replacing the production.rb
• ... same with batch_size
• ... have to mount this script into the container to run it via outside exec command

Other changes:

• I call the destroy from the looping record, instead of fetching the record_instance like in your snippet, which seems more efficient (pls tell me otherwise)

I share this here, for others too.

Script

=begin
Example ruby script for deleting records older than N days.

Run with `rails runner [PATH_TO_SCRIPT]`
=end

# add core_ext for easy date calculation
require 'active_support/core_ext'

# because this script runs in the greenlight docker container
# ENV is used to define the number of days for cutoff
# otherwise define in another way you like
cutoff_days = ENV["CUSTOM_RECORDING_CUTOFF_DAYS"]

if cutoff_days
    cutoff_days = cutoff_days.to_i
    warn "Check for records to delete older than #{cutoff_days} days!"
else
    warn "No recording cutoff days defined!"
    exit 0
end

recordings_found = false

# use constant batch_size instead of 'Rails.application.config.batch_size' or ENV for testing
batch_size = 500

# filter database
Recording.where("recorded_at < ?", Time.now.to_date - cutoff_days.days)
    .select(:id, :room_id,  :name, :record_id, :recorded_at)
    .find_in_batches(batch_size: batch_size) do |batch|
  begin
    # recordings where found
    recordings_found = true
    # iterate over batch
    batch.each do |record|
      begin
        # just some debug output
        warn "Found recording for deletion => id: #{record.record_id} | name: #{record.name} | recorded at: #{record.recorded_at}"
        # send delete recording request to BBB, by using the destroy mechanism
        record.destroy!
      rescue => e
          # show what went wrong
          warn "Failed to delete recording: #{record.record_id} - #{e.message}"
      # don't be to fast
      sleep(1)
      end
    # still don't be to fast
    sleep(1.5)
    end
  end
end
warn "No Recordings found older than #{cutoff_days} days!" unless recordings_found

[rautniraj]

... it's easier to define an environment variable, instead of custom replacing the production.rb

I have also done the same way. Actually I meant the same. See in Rails, reading a value directly from ENV is not preferred. First you read all your env in your production.rb file. Then From there you continue reading your variable.

... have to mount this script into the container to run it via outside exec command

I dont think script is required. If you create a rake task file then you can simply execute that. No mounting needed. You can define your cron task like 5 0 * * * /usr/bin/docker exec greenlight bundle exec rake admin:recordings_purge (Considering that the recording_purge task is inside admin.rake)

Yes you can omit record_instance Just for a safety net i kept both the variable record and record_instance separate. My thinking was i need to log the details in my db along with showing on the terminal. I thought that record.destroy will also clear the record variable and after delete i might not be able to extract the details from this variable.

I think i am wrong, variable remains intact so there isnt any problem

[PhilippKilian]

@rautniraj

I have also done the same way. Actually I meant the same. See in Rails, reading a value directly from ENV is not preferred. First you read all your env in your production.rb file. Then From there you continue reading your variable.

We have no production.rb file mounted, everything is configured via environment variables. That's why I used it this way.

I dont think script is required. If you create a rake task file then you can simply execute that. No mounting needed. You can define your cron task like 5 0 * * * /usr/bin/docker exec greenlight bundle exec rake admin:recordings_purge (Considering that the recording_purge task is inside admin.rake)

This makes no difference, I think. Because I have to add the rake task file or script file to the running docker container from an untouched docker image. Both ways looks fine to me, but I prefer the script for a minimal noninvasive way.

Thanks for your support of this issue, it can be closed I think if you haven't anything further.