ulixee
feature request: disable replay
Replay is an awesome and one of kind tool for debugging, but when it comes to production env it seems to be overkill to use it when running a lot of instances at the same time. It would be great if Replay could be disabled, thus no writing in sqlite files be done.
Thanks for the feedback @janisblaus. You can disable the UI two different ways: 1) with an environment variable SA_SHOW_REPLAY=0 and 2) during the creation of an Agent there's an option to showReplay: false.
During production, the thinking is you don't need Replay unless you have a script get blocked or break. That's intended to be one of it's main purposes - how to troubleshoot a remote script that breaks "after the fact". You can import the database from your remote machine and open it with Replay.
In our coming releases, we'll be making that a lot more seamless, and there will be ways to cleanup unnecessary databases.
I know about disabling UI, I'm talking about disabling those huge sqlite databases. If I have 1000 threads running at the same time, it does not make sense to record results on every one of them, that is why I think it should be a necessity to add an option to disable reply. :)
Good point. For what it's worth, the databases are much smaller in the upcoming release (we're still sorting out a few last things on the plugin architecture which is holding up releasing).
I was trying to say: I think we have some ideas how to keep around the 1 in 1000 databases you actually need in your scenario, but for they will come in the next release or two.
Hello,
I just wanted to hop-in this thread because we noticed that the docker container that functions as the Core had built up 22GB (!) of databases files over the course of a day. We had a cron job that daily removes these files but it seems that didn't pop of today because we saw these errors
[Nest] 1 - 01/28/2022, 8:00:34 AM ERROR [ExceptionsHandler] unable to open database file
SqliteError: unable to open database file
We have SA_SHOW_REPLAY=false on production but I don't know if after the latest update (1.6.3) the databases files seem to grow faster or we just have a lot of traffic going through the Secret Agent
Just an update, over the course of these 2 hours the files jumped from 189MB to 531MB in size. There has been a constant use of the service.
@emilio-cea Do you think you could send me one of the databases? I would like to see the breakdown of which tables are taking up the most size. SA_SHOW_REPLAY unfortunately only drives launching the UI..
--- EDITED ---
Removed a proposed solution that won't work in SecretAgent
Hi @blakebyrnes I'd like to check the database before sending it to you to see if it contains any sensitive data. I tried using replay on Windows to see the navigation but I think I must be doing something wrong as when I open the .exe and select the database file, an error pops up:
Are there any steps I missed? Any suggestions as to what contents do the databases have that could be sensitive?
Edit: maybe the versioning of the Replay program has something to do with the error?
@emilio-cea It depends on what you consider sensitive. You can send directly to me on Discord or email, so it's for my eyes only :) There are a decent number of "directory paths" in SessionLogs table and sometimes in commands table, which can sometimes reveal your personal info you don't want shared. Commands also log exact text that's typed, so that could also be revealing if you have passwords in it. Again, only sensitive if you're worried about me seeing it. I won't send to anyone else and will delete it once I'm done looking.
Replay does that on Windows when you open it directly. I would fix it, but we're switching to using Chrome for Replays in next version. You can sometimes get around this by first launching replay with a script, then going to the open script option in the menu.
Hi @blakebyrnes I sent an email with an example DB. I don't think there's much content to be extracted for that but we do that kind of navigation A LOT, so we have a lot of DB files piling up over the course of a couple hours. Hope this is of any help to you.