A Comprehensive Introduction To Syslog

This article provides a detailed guide on configuring syslog. Earthly streamlines complex build processes and speeds them up. Learn more about Earthly.

What Is Syslog

Syslog is a standard protocol for sending log messages from one system to another or within the same host. It is used by servers, routers, switches, and firewalls. Syslog messages contain information about events that occur on the device, such as errors, warnings, and status changes. It ensures comprehensive visibility into the health and performance of a system by providing a standardized method for collecting, storing, and transmitting log messages from various devices and applications.

Syslog has a client-server architecture. Clients are applications, services, or dedicated devices such as routers and switches in a network. The server is the rsyslogd daemon, which handles the actual log management. In a network, the rsyslogd daemon communicates with the clients on the UDP protocol on port 514.

The information availed by the syslog entries is used to troubleshoot problems, monitor system health, and comply with security regulations. Its versatility and compatibility with a wide range of devices and platforms make it an indispensable resource for maintaining security, identifying anomalies, and facilitating seamless communication between interconnected components.

In this guide, you will learn more about the syslog daemon and its properties. Also, you will learn how to configure local logging on a server, manage syslog configuration files, and perform log rotation.

Syslog Daemon

The syslog daemon is a background service responsible for receiving, processing, and distributing log messages generated by applications, services, and devices. It operates based on a client-server model, where applications act as clients, and the syslog daemon acts as the server. When a device sends a log message to the syslog daemon, the daemon parses the message and stores it in a buffer. The daemon then checks the configuration file to determine where the message should be forwarded.

In Ubuntu and Debian-based distros, the daemon is called rsyslogd. The default file location for syslog is /var/log/syslog. It is a critical log file that contains system-related information, including events, errors, and warnings. The purpose of the file is to provide a consolidated and comprehensive record of system activities and events. The syslog daemon performs compression and archival based on a predefined file size or duration.

Syslog messages are structured and contain information such as the timestamp, hostname, facility, level, and message content.

The facility indicates the program, process, or component that generated the message, while the severity level suggests the importance of the event.

Syslog Facilities

Syslog Facilities categorize the source of a log message and provide a way to identify the system component or application that generated the message. Syslog defines 24 standard facilities by corresponding numeric codes from 0 to 23. Each facility represents a specific area of the system or application.

Here are some commonly used syslog facilities, along with their numeric codes:

0 - kernel: Messages related to the operating system kernel.

1 - user-level: Messages generated by user-level processes or applications.

2 - mail: Log entries related to mail servers and email processing.

3 - daemon: System daemons or background services generate these messages.

4 - auth: Authentication-related events, such as user login/logout.

5 - syslog: Internal messages generated by the syslog daemon itself.

6 - lpr: Printing-related messages.

7 - news: Log entries related to Usenet news.

8 - uucp: Messages related to Unix-to-Unix Copy (UUCP) communications.

9 - clock: Clock daemon messages.

10 - authpriv: Private authentication-related events.

11 - ftp: Log entries related to FTP services.

12 - NTP: Network Time Protocol-related messages.

13 - security/authorization: Security-related events and authorization failures.

14 - console: Messages displayed on the system console.

15 - solaris-cron: Log entries specific to Solaris cron.

16 - local0 to local7: These are user-defined facilities for local applications or services.

These facilities provide flexibility for customizing logging for specific needs.

What Are Syslog Levels?

Syslog defines eight categories from 0 (emergency) to 7 (Debug level). These levels are used to classify messages based on their severity. Levels 5 - 7 are mostly used by applications, while 0 - 4 are mostly used by the operating system. For example, we can enable debug logs when developing or troubleshooting an application and change the log level to errors or warnings when we deploy it. Here are the syslog levels, along with their numeric codes:

0 - emergency: System is unusable. This level indicates a catastrophic failure or a situation that requires immediate attention.

1 - alert: Immediate action is needed. This level indicates a condition that should be corrected immediately, such as a critical system component failure.

2 - critical: Critical conditions. This level indicates a severe error or problem that requires prompt attention.

3 - error: Error conditions. This level indicates non-critical errors that may require investigation or troubleshooting.

4 - warning: Warning conditions. This level indicates a potential problem or situation that should be monitored or addressed.

5 - notice: Normal but significant conditions. This level indicates noteworthy events but is not necessarily indicative of an error.

6 - informational: Informational messages. This level provides general operational information or status updates.

7 - debug: Debug-level messages. This level denotes detailed debugging and diagnostic information.

Depending on the logging configuration, messages can be filtered or handled differently based on their level.

Use Cases for Syslog

System administrators use syslog to collect, analyze, monitor, and store log messages from various devices, servers, and applications to facilitate troubleshooting, performance monitoring and mitigate security events. Below, you will learn some of the use cases in a practical setting:

• System and Network Monitoring: Collecting log messages from servers, network devices, and applications helps monitor their health, identify issues, and perform troubleshooting. Network administrators may send log events to a Graylog server for analysis and troubleshooting if devices fail.
• Security Event Logging: syslog collects security-related events such as failed SSH login attempts, firewall alerts from UFW, and Nginx access and error alerts.
• Compliance and Auditing: Compliance standards require organizations to maintain comprehensive logs for audit purposes.
• Troubleshooting and Debugging: Analyzing syslog messages provides insights into system or application issues, aiding in troubleshooting and debugging processes.
• Application Performance Monitoring: Logs collected from application components help identify performance bottlenecks. In a LAMP/LEMP stack utilizing the PHP FastCGI process manager, the FPM server will always log warnings when resource utilization increases and adjustments are needed. Developers and administrators can track application behavior under load and make necessary adjustments.
• Log Analysis: Syslog messages are loaded into analysis tools for log correlation, threat detection, and incident response.
• Forensics and Incident Response: Analyzing syslog messages can help identify the root cause of incidents and provide evidence for legal or compliance purposes.
• Log Archiving and Long-Term Storage: Syslog enables the collection and storage of logs from multiple sources in a centralized location, ensuring long-term retention for compliance, historical analysis, or future reference.

How to Interpret Syslog Entries

To interpret log entries, we must know when the event occurred, the severity level to determine its importance, and the facility to determine the message’s origin. Interpreting syslog entries requires understanding the structure and format of syslog messages. Understanding how components interact allows you to correlate events between two or more applications or components. Syslog messages contain several components, as illustrated below:

• Timestamp: The timestamp indicates when the event occurred. It includes date and time information, allowing you to understand the temporal context of the event and relate events chronologically.
• Hostname: The hostname identifies the device or system that generated the syslog message.
• Process: The process field shows the name or identifier of the process or application that generated the syslog message.
• Severity Level: Syslog messages are assigned a severity level to indicate the event’s importance or urgency.
• Message: The message field contains the actual content of the syslog entry. It includes information about the event, error details, status messages, or any other relevant information related to the logged event.
• Structured Data (Optional): Structured data provides additional context or specific information about the event. Structured data is typically presented in key-value pairs and can be used to convey precise details or metadata associated with the event.

You can use the tail command to view log entries as they are appended like so:

tail -f /var/log/syslog

Every line in the screenshot above gives us the following information about events:

• The date and time an event occurred.
• The hostname of the device on which the event occurred. In this case, it is on my laptop.
• The third column is the program that generated the event.
• The final column is the message content associated with that event.

How To Limit the Size and Number of Syslog Entries

Log files can grow to a large size when left to grow unchecked. Filesystem performance takes a hit and drags the entire server to a crawl, causing a terrible experience for the end users. To limit the number of syslog entries, you can configure logrotate to handle the rotation and retention of syslog files when they reach a specific size.

To edit syslog configuration files from the command line, you can use any file editing commands. Below, I used the nano command for editing. Configuration files are stored in the /etc directory and have write permissions limited to either the root user or the application that created them. Prefix the shell commands with sudo to edit the files as root.

In the example below, we specify the conditions that must be met before the syslog file is rotated. Once we edit the configuration, we have to test the new settings before restarting rsyslogd to apply the changes.

Open the logrotate configuration file for syslog:

sudo nano /etc/logrotate.d/rsyslog

You can configure the rotation options for syslog by adding the below contents to the configuration file. With this configuration, only the last 5 syslog files will be retained, each with a maximum size of 100 MB. Log rotation will not take place if the file is less than one week old or is less than 100 MB.

/var/log/syslog {
# Rotate the file weekly
weekly
# Rotate the syslog file when it reaches 100 MB
size 100M
# Keep  a maximum of 5 syslog files and delete older ones
rotate 5
# Compress the rotated file using Gzip
compress
# Delay the compression of the previous log file until the next rotation cycle
delaycompress
# Do not raise an error if the syslog file is missing
missingok
# Do not rotate the syslog file if it is empty
notifempty
# Create a new syslog file with the specified ownership and permissions
create 0644 syslog adm 
# Optional post-rotation script or command
postrotate 
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

• Exit nano Once you are done editing, press CTRL+O to write the contents to the file followed by CTRL+X to exit the editor.
• Test the configuration:

sudo logrotate -d /etc/logrotate.d/rsyslog

In this example, /usr/lib/rsyslog/rsyslog-rotate notifies rsyslogd about the log rotation. This script is executed by a cronjob to inform rsyslogd to write to the new syslog file.

How to Configure Logrotate for a Daily Log Rotation

In this example, you will configure daily log rotation to archive the Nginx error log when it reaches 10 MB. You can then transfer the compressed files to a separate location for analysis later.

Follow these steps to configure log rotation for the Nginx error log using logrotate:

• Create a logrotate configuration file.

sudo touch /etc/logrotate.d/nginx-logrotate.conf

• Define the log rotation configuration.

sudo nano /etc/logrotate.d/nginx-logrotate.conf

In the configuration below, we specify that the logs should be rotated every day or when they reach 10 MB. We also specify that we want to keep only the last seven files, use Gzip compression, delay compression until the next file is created, skip rotation if the file is empty, and create a new file with the specified permissions and ownership. The compressed files are stored with a .gz file extension in the same directory as the original uncompressed files.

/var/log/nginx/error.log {
# Rotate log files every day
daily
# Limit the size to 10 MB per file
size 10M
# Do not raise an error if the log file is missing
missingok
# Keep the last 7 log files and delete older ones
rotate 7
# Use Gzip compression
Compress
#Delay compressing until the new file has been created
Delaycompress
# Skip rotation if  the file is empty
notifempty
# Recreate the file with this ownership and permissions
create 0644 www-data www-data
}

• Save the configuration file and exit the text editor. Press CTRL+X and CTRL+O to write the file to disk and exit from nano.
• Test the configuration

sudo logrotate -d /etc/logrotate.d/nginx-logrotate.conf

• Restart the server daemon with the following command:

sudo systemctl restart rsyslogd.service

General Steps To Store Syslog Files on a Remote Server

It is also easier to correlate events from multiple sources by matching text patterns. Disk usage on clients is not affected by log files even when there are failures, and the data is also safe in case a device fails and the data cannot be extracted. Syslog clients are network devices, servers, or applications that send log messages to a server.

Here are the general steps to configure your syslog server and clients:

1. Choose a Syslog Server: Select a server or system to act as your centralized syslog server. Ubuntu and Debian come pre-installed with rsyslog.

2. Install and Configure Syslog Server: Set up the syslog server software on the chosen server. Some popular syslog server software options are syslog-ng, Graylog, and Splunk.

3. Configure Syslog Clients: Configure the devices, servers, or applications that send syslog messages to the remote server. Locate the syslog configuration file on each client and update it to include the remote server’s IP address.

4. Set Logging Levels and Filters: Determine the level of detail you want to capture in the syslog messages and configure appropriate filters. You can specify which log messages to forward to the remote server based on severity level or specific facilities.

5. Test Syslog Communication: Restart the syslog service on each client device and the syslog server to apply the configuration changes. Generate test log messages on the client devices to verify transmission to the remote server.

6. Manage Log Storage: Configure log rotation and retention policies on the syslog server to ensure log files are periodically rotated, compressed, and retained for a specific duration.

7. Security Considerations: Set up a VPN tunnel between the client and the server if the transmission is over an untrusted network.

Syslog Storage Full | Emergency Actions

When log files fill up rapidly, it is an indication of a component failure or a logging misconfiguration. The priority is to ensure service is uninterrupted as you work to resolve the problem. In this section, we look at how you conduct emergency actions to avoid running out of disk space or recover space.

Once you realize there are repeated log entries, the immediate goal is to narrow down on the offending client. You can then fine-tune the severity level for information messages or troubleshoot the underlying issue if the severity is higher such as warnings or errors.

How to Manually Rotate the Syslog File

You may encounter a situation where you need to rotate logs manually. I recently had to do this on one of my own servers.. In my case, there was a missing snmpd configuration directive, causing it to log all requests to syslog and resulting in a 20 GB file on a cloud-hosted virtual machine. You can read about it on the ServerFault Website. As a preventive action, this behavior was corrected by tweaking the snmpd configuration by adding dontLogTCPWrappersConnects true to the /etc/snmp/snmpd.conf file. As a corrective action, I have to manually truncate the syslog file.

Follow these steps to rotate the syslog file manually:

• Check the current size of the syslog file

ls -lh /var/log/syslog

The output shows the permissions, ownership, and size in a human-readable format and the file’s name.

• Truncate the syslog file.

: > /var/log/syslog

To understand the command above:

• The : means true and produces no output

• The > sign is the redirection operator. It redirects the output of the first command to the file specified.

• /var/log/syslog is the file we want to truncate

This empties the file while preserving the permissions, which is much better than deleting and recreating it.

• Next Step is to restart the syslog daemon:

systemctl restart rsyslog.service

• Verify that syslog is running:

sudo systemctl status rsyslog.service

You should be able to see the service active and running.

In addition to this, there are more actions we can implement to prevent a recurrence:

• Identify the cause of the problem. The first step is to identify the cause of the problem. This can be a software bug or a failing hardware
• Reduce the number of log messages. Limit the severity of messages to errors or warnings. These will be sufficient to troubleshoot issues.
• Automate the log rotation using logrotate configuration.
• Research if the issue is in the logging software itself and upgrade it

Conclusion

In this article, you have learned the syslog architecture components, syslog facilities, and severity levels. You have also learned how to fine-tune the configuration, add new configurations for specific applications, and how to troubleshoot when you have many messages being generated.

To build on this and gain more insights, a good next step would be to read the RFC 5424 document or Building Secure Servers with Linux book from O’Reilly Media. Learn how to configure remote logging for your environment with tools like Graylog or Splunk and manage your log files in a central location. You can also write custom shell scripts utilizing grep and awk to filter instances where you have failures.