Logs are extremely useful when troubleshooting system, application or network problems. Information captured in log files may also be analyzed to uncover patterns that can help you make informed decisions as a system admin.

Most Linux log files are stored in a plain ASCII text file and are in the /var/log directory and subdirectory. Logs are generated by the Linux system daemon log, syslogd or rsyslogd.

You need be the root user to view or access log files on Linux or Unix like operating systems.

Here at LinuxAPT, as part of our Server Management Services, we regularly help our Customers to troubleshoot Server problem via the log files in the system.

In this context, we shall look into how you can view standard log files on Ubuntu Linux servers.

Some commands for working with log files

On Ubuntu Linux servers, logs are normally stored in plain text format. Hence, it is important that you know the following basic Linux commands for moving around the filesystem and working with text files via the Ubuntu terminal.

1. cd – change directory

2. ls – display the contents of a directory

3. cp – copy files or folders

4. mv – rename/move files or folders

5. nano – console-based text editor

6. less – view the content of a text file one page at a time

7. head – display the first 10 lines of a text file

8. tail – view the last 10 lines of a text file

9. grep – search for specific keywords in a text file or output data

10. zcat – Displays all the contents of logfile.gz

11. zmore – See the file in pages, without decompressing the files

12. zgrep – Search inside a compressed file

13. head – View the very beginning of text files

Location of log files on Ubuntu servers

Basically, log files are stored under the /var/log directory on Ubuntu servers. 

Run the command below to change the directory to /var/log:

$ cd /var/log

Now, you may list the content of /var/log as follows:

$ ls

The /var/log directory contains several log files that can be broadly categorized into system logs and application logs.

Important Linux System Logs

Logs can tell a lot about the operations of a system. A good understanding of each type of file will help how to distinguish the respective logs.

Most directories can be grouped into one of four categories:

1. System Logs

2. Event Logs

3. Application Logs

4. Service Logs

Many of these logs can be located in the var/log subdirectory.

System Logs in Linux

Systems log files are needed for Linux to work. On its own, it contains the most significant amount of information about system functionality. Basically, System logs contain information about the operation of the Ubuntu system; including authorization logs, kernel logs, kernel ring buffer, and general system events.

The most common log files are:

1. /var/log/boot.log: System Boot log (the boot log stores all information related to booting operations)

2. /var/log/auth.log: Auth logs (the authentication log stores all authentication logs, including successful and failed attempts)

3. /var/log/httpd/: Apache access and error logs

4. /var/log/mysqld.log: MySQL database server log file

5. /var/log/debug: Debug logs (the debug log stores detailed messages related to debugging and is useful for troubleshooting specific system operations)

6. /var/log/daemon.log: Daemon logs (the daemon log contains information about events related to running the Linux operation)

7. /var/log/maillog: Mail server logs (the mail log stores information related to mail servers and archiving emails)

8. /var/log/kern.log: Kernel logs (the kernel log stores information from the Ubuntu Linux kernel)

9. /var/log/yum.log: Yum command logs

System Logging Daemon

A daemon log is a program that runs in the background and is essential for system operations. These logs have their own category of logs and are seen as the heart of the logging operations for any system.

The path for the system login daemon's configuration is /etc/syslog.conf.

Each file consists of a selector and an action entry field. 

The syslogd daemon can forward log messages as well. This can be useful for debugging purposes.

Application Logs

Several applications store log information under /var/log. For example, in figure 1 above, the clamav directory contains log files pertaining to the ClamAV anti-malware application.

Here are some examples of popular applications or services and where their log information is stored:

Apache web server logs - /var/log/apache2
NGINX web server logs - /var/log/nginx
Printing system (CUPS) logs - /var/log/cups

Authorization logs

Authorization logs are stored in /var/log/auth.log. This is where you will find information about user authorization attempts; including the use of the sudo command.

You may run the command below to inspect the content of the auth.log file:

$ sudo less /var/log/auth.log

Note: Hit the spacebar on your keyboard to scroll from page to page. Press q to exit.

You could also use the grep command to filter the information in the logs. 

Here is an example:

$ sudo less /var/log/auth.log | grep linuxapt

The information in the sample output below indicates that there was a successful remote login to my Ubuntu server via ssh by user linuxapt:

Feb 1 15:44:24 Ubuntu sshd[1594]: Accepted publickey for linuxapt from 105.0.0.100 port 35233 ssh2: RSA SHA256:B3zi4x3gdF89wm0GZw+fsAkhckLEsx8fJ0GJiU80CXH
Feb 1 15:44:24 Ubuntu sshd[1594]: pam_unix(sshd:session): session opened for user linuxapt by (uid=0)
Feb 1 15:44:24 Ubuntu systemd-logind[747]: New session 2 of user linuxapt.
Feb 1 15:44:24 Ubuntu systemd: pam_unix(systemd-user:session): session opened for user linuxapt by (uid=0)

Kernel logs

Kernel logs are held in /var/log/kern.log. This information is useful for troubleshooting kernel errors. 

The kernel controls everything in the operating system; including process management, memory management and device management.

Use the following command to display the content of the kern.log file one page at a time:

$ sudo less /var/log/kern.log

Or try this to display the first 10 lines of the kern.log file:

$ sudo head /var/log/kern.log

Find specific information in kern.log:

$ grep memory /var/log/kern.log

Kernel ring buffer

The kernel ring buffer holds kernel hardware information. The information is logged in /var/log/dmesg and can be displayed by using the dmesg command.

This information includes all detected devices at system boot time.

You can use this to troubleshoot issues with server hardware components. 

Run the command below to view the entire content of the kernel ring buffer:

$ dmesg

Try the next command to display the last 10 lines of the kernel ring buffer:

$ dmesg | tail

Or filter for specific keywords using grep:

$ dmesg | grep cpu

General system logs

Here, we are going to talk about syslog and journalctl.

Syslog

Syslog is a logging mechanism that stores general system events in /var/log/syslog. The information stored here may include events that you may not find in other log files.

Run the command below to display the content of the syslog file page by page:

$ sudo less /var/log/syslog

You could also search for specific keywords using the grep command as follows:

$ sudo grep failed /var/log/syslog

Journalctl

The journalctl command simplifies the process of examining server logs. Rather than look through individual log files, you could use journalctl to quickly find and filter the information that you need.

The command below displays all log entries from oldest to newest:

$ journalctl

The next command shows warning messages:

$ journalctl -p warning

You can display only kernel messages as follows:

$ journalctl --dmesg

You can search for specific keywords by combining the grep command and view results page by page using less:

$ journalctl | grep ssh | less

View log information since a specific date:

$ journalctl --since=2021-02-01

Or view log information since a specific time:

$ journalctl --since=12:00

You could also type journalctl and then press the tab key on your keyboard to see available options.

Supplemental GUIs for Viewing Linux Log Files

System Log Viewer is a GUI that can be used to monitor system logs.

The interface provides several functions for managing logs, including a log statistics display. It is a user-friendly log monitoring GUI.

Useful features include:

i. A live view of logs

ii. Number of lines in the log

iii. Log size

iv. Most recent log dates

v. Modifications made to logs

vi. Filters

vii. Keyboard Shortcuts

Alternatively, use Xlogmaster which can monitor a considerable number of log files.

It features three different modes:

1. Run mode: Starts a specified program and obtains stdout

2. Cat mode: Cats files within specified intervals

3. Tail mode: Checks log files within regular intervals

Xlogmaster is useful for increasing security. 

It translates all data for highlighting, hiding lines, and displays this information for taking user requested action.

Other Useful Logs

Some log files such as lastlog, wtmp may not be directly read by humans. 

The following is a brief explanation of what type of information these files contain and how you can view it.

lastlog

The information held in /var/log/lastlog pertains to users and their most recent login to the Ubuntu server. You would need to use the lastlog command to access it as follows:

$ lastlog

wtmp

The var/log/wtmp file holds comprehensive login records.

Run the last command to display a list of last logged in users. You may also see information about system boot/reboot:

$ last

Run the who command to see who is currently logged in:

$ who

The w command shows you who is currently logged in and what they are doing on the Ubuntu server:

$ w

[Need urgent assistance in fixing missing packages on Ubuntu Linux Server? We are available. ]