theophilusx
system log files - truncate, clear or delete?
This is more a question than an issue. I did search through code on repository and mscdex/ssh2 repository for a trunc method before creating this post. I found the mscdex/ssh2 unlink method which made my idea to add a trunc method myself more difficult.
I've been using this package to automate the upload application code to server and it works great. Now I am looking to automate the copy of system log files /var/log/ from server and delete the files copied. I read several articles about how to empty, clear, truncate and delete log files. Some suggest I should not outright delete the log files and some say it is okay to simply delete log files.
My question is how do I delete a log file in Linux without disturbing running applications?
Or will it not disturb running applications at all since they auto check for file first and create it if missing?
I use logrotate for Apache logs on a daily rotation basis but wanted to build something more on demand (up-to-the-minute) for other log files.
Any input about this type of file functionality is greatly appreciated. Thank you for your time!
Like many seemingly simple questions, the answer here is complicated and largely comes down to 'it depends'. This is also outside my area of expertise. It has been many years since I did sys admin work and file systems and core libraries have evolved considerably in that time. You really need to ask this question in a channel which is more specifically focused on Linux system administration. Log file management is complicated.
What I can tell you is that
-
There is no single approach. It all depends on how a specific application manages log records and log file descriptors/handlers.
-
Many applications which rely on log records for administration/maintenance provide support for log maintenance. For example, Apache and other web servers will re-initialise file descriptors after receiving a USR1 signal. Unfortunately, not all applications support this and those that do don't always use the same signal.
-
You need to consider race conditions and permissions when dealing with log management. Many systems log data using a specific user/group and specific file permissions. Any solution must maintain this. A simple copy and delete can encounter race conditions where log records can be lost. For example, you copy the log file, delete the original, but in-between those operations, new logs were written to the original file.
-
You asked this question in a repository associated with an sftp library. Note that sftp and ssh are not the same thing. There is actually a wsell defined ftp protocol and sftp is really an implementation of that protocol wrapped in an encrypted ssh tunnel. There is also a big difference between a 'real' sftp server and simply uploading/downloading data using sftp to a specific user's directory. Important to understand the differences. For example, sftp servers are often run in a chroot environment, so the file system acessible/viewable when logged into the sftp serbver is not the same as the filesystem of the sftp server host. There are also often restrictions on what commands can be executed and the permissons of processes run from within the sftp server environment.
Unless your use case is very simple, I would be avoiding implementation of some form of remote log rotation/management system. Far better off implementing a log rotation system at the host levbel where the application is running and if possible, using one of the many log rotation/management solutions available. It is very likely that the host running the application already has some form of log rotation/management support and all you have to do is configure it to be aware of your application.
WillTheFarmer WillTheFarmer created an issue (theophilusx/ssh2-sftp-client#574)
This is more a question than an issue. I did search through code on repository and mscdex/ssh2 repository for a trunc method before creating this post. I found the mscdex/ssh2 unlink method which made my idea to add a trunc method myself more difficult.
I've been using this package to automate the upload application code to server and it works great. Now I am looking to automate the copy of system log files /var/log/ from server and delete the files copied. I read several articles about how to empty, clear, truncate and delete log files. Some suggest I should not outright delete the log files and some say it is okay to simply delete log files.
My question is how do I delete a log file in Linux without disturbing running applications?
Or will it not disturb running applications at all since they auto check for file first and create it if missing?
I use logrotate for Apache logs on a daily rotation basis but wanted to build something more on demand (up-to-the-minute) for other log files.
Any input about this type of file functionality is greatly appreciated. Thank you for your time!
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.
That is an informative reply and very helpful. It certainly raises more questions and considerations to take into account before building anything serious. The permissions and race conditions were certainly a concern with the delete file option. That was the reason for seeking a 'trunc' method.
I use Logrotate for Apache logs and created a Python module to process rotated files to MySQL. I do not deal with race conditions or permissions in that situation. apache-logs-to-mysql
Now I created a similar process with a MySQL system_logs schema and LOAD DATA process for system logs for the non-profit organization I volunteer at to use in-house. This is how the Apache log project started. Right now I am manually copying the system log files from the web servers and processing them in a local folder. The reason for this post was my consideration and curiosity of using your package to implement the copy part I currently do manually.
Your post gives me plenty to think about. An update on-demand "button" to retrieve the latest logs on multiple servers has been a would-be-nice thought this week.
I greatly appreciate your input, thoughts and knowledge on this not so simple question. It might be more time than I have to invest right now. I know how these small projects can snowball. Thank you for your time!