Log Aggregation 101: Methods, Tools, Tutorials and More

Log management is a process of handling copious volumes of logs that are made up of several processes, such as log collection, log aggregation, storage, rotation, analysis, search, and reporting.

Log aggregation, therefore, is a step in the overall management process in which you consolidate different log formats coming from different sources all into one place to make it easier for you to analyze, search, and report on your data. Let’s take a closer look at why log aggregation is important, methods, and tools that can make the process simpler.

Why Aggregate Your Logs?

Logs are very important to any system because they give you an idea of what is happening and what your system is doing. Most processes that are run on your system create logs. The problem is that these files are often found on different systems and in different formats. If you have a variety of hosts, then you might end up with different logs hosted on different locations.

If you have an error and you need to refer to your logs to see what went wrong, it would mean that you would need to search in dozens, or even hundreds, of files. Even with good tools, you will spend a lot of time doing this and it can easily frustrate even the hardiest system administrators. Log aggregation is a good way to bring together all these logs into one location.

Gamesparks shares their experience, indicating that the company faced several challenges wherein log aggregation impressively helped. For one, they had distributed servers and needed a better way to transmit multi-line log entries from different sources. They also needed their servers to be highly available or to risk losing valuable information.

Being able to aggregate logs in a centralized location with almost real-time access allowed the company to easily and promptly troubleshoot problems, while also being able to do trend analysis and pinpoint anomalies.

Methods

There are a few methods you can employ to aggregate logs.

Replicate your log files. It is very easy to copy your files to a central location using rsync and cron. This is the easy way that can serve the purpose of getting all your data in one place, but it’s technically not true aggregation. Plus, since you have to follow a cron schedule, you cannot access your files in real time.

Syslog, rsyslog, or syslog-ng. What these do is to tell processes to send log entries to them, and the configuration will direct these messages to a central location. You would need to set up a central syslog daemon somewhere on your network as well as on the different clients to enable these clients to forward messages to the daemons. Urban Airship has a great tutorial on how to set this up at this page. With syslog, chances are it already exists in your system, so it’s just a matter of configuring it right and making sure that your central syslog server is always available.

Tools

Log management and aggregation are made simple thanks to the availability of tools that can automate the process. These tools are great in that they work the same way as syslog, syslog-ng or rsyslog, but have other features that make them worth your while.

Different tools have their own strengths, but most rely on a similar architecture: using a logging client on each host and then collecting these files to a central location, making use of a storage tier that is easily scalable to accommodate all the data coming in over time. A few of the tools in this space include:

• Apache Flume: This tool can efficiently collect, aggregate and move huge amounts of information. It can also be used to move data from your different applications and stores them on the HDFS on Hadoop.
• Splunk: Splunk has been around for a long time and it gives you easy log aggregation as well as a number of other features for viewing and analyzing logs and creating reports and searches. It is very simple but capable tool – you can easily and efficiently aggregate logs in your whole IT network.
• Scribe: This C++ tool was released by Facebook on GitHub. Scribe works with just about any language and is a reliable and scalable server.
• Logstash: Logstash is a server-side data processing pipeline that can manage data from different sources. The open-source tool defines inputs, filters, and outputs while also making it easier to access, view and search your logs.
• Chukwa: This Hadoop sub-project can efficiently collect your logs and store them on HDFS.
• Graylog2: Graylog2 stores your events on elasticsearch or MongoDB. You get a UI to analyze or search your logs.

Log Aggregation as a Service

There are several cloud service providers that offer log management and aggregation as a service. This effectively takes away most of the work you need to do when it comes to storing or accessing your files, as well as guaranteeing that you would spend no time maintaining and setting up any infrastructure you may need. You just have to configure your syslog daemons or agents and these “as a service” providers will take care of the rest. A few of the cloud-based tools include:

• Retrace: For a fully-integrated solution, complete with APM, errors, metrics, monitoring, and, of course, logs, Retrace is Stackify’s comprehensive app performance superpower solution for production and pre-production servers.
• PaperTrail: PaperTrail helps you manage all your logs no matter which server they come from. It counts GitHub and InstaCart as customers.
• LogEntries: It focuses more on log aggregation and analysis. You can search, monitor or visualize your messages using this service.
• Loggly: This cloud-based service allows you to easily access and analyze all your logs even in real time.

There are a few important features to look for in a tool, including:

• Support for popular logging frameworks (NLog, log4net, etc.)
• Full-text searching (a huge time-saver)
• Structured logging (log objects, search by properties)
• Log tailing
• Monitoring and alerting (because who has time to sit around and watch their log files all day?)

For a more comprehensive list of tool options, check out our list of the top 51 log management and monitoring tools here.

[adinserter block=”33″]

Additional Resources and Tutorials

For more information on log management, monitoring, and aggregation, visit the following resources and tutorials:

• How to Take Logging to the Next Level With Improved .NET Logging Best Practices
• Hadoop Tutorial: How to Refine and Visualize Server Log Data
• Network Management & Monitoring: Log Management
• Monitoring Windows Event Logs – A Tutorial

Log aggregation is critical for modern development, as today’s developers are managing a variety of data from myriad sources – and errors and bugs could be coming from any number of them. It’s simply not efficient to manually search through dozens of files to locate the root of a problem; log aggregation eliminates this massive time-suck.

Author

angelas

More articles by angelas