Log Aggregation

In a cloud environment employing a microservices architecture, one of the major challenges is to collect the logs from different services (and multiple instances of a service) and virtual machines for purposes of troubleshooting and analysis.

One of the solutions is to use a log aggregator. A log aggregator system is a system, which collects log data from different systems/services/virtual machines at a centralized location and provides a uniform way of accessing and analyzing logs.

Log Aggregator Sub-systems:

A log aggregator generally has 3 sub-systems as listed out below.

1. Log collector
2. Log forwarder
3. Log storage system
4. Visualizer and a Dashboard

Each of these sub-systems which are illustrated in the above diagram is explained below.

1. Log collector

A log collector is an agent which will be deployed on the individual servers and is responsible for collecting logs from the application and forwarding it to a common server.

2. Log forwarder

A log forwarder is a component server which will accept logs from the collector agents; it might perform formatting of logs as needed and would communicate with the storage system to store the logs or it might also integrate with an alerting system to trigger alerts accordingly.

3. Log storage system

A log storage system is a database solution which would store the logs and provide search capabilities.

4. Visualizer and a Dashboard

A visualizer provides a visual representation of the logs in a dashboard format by retrieving it from the storage system and would also support running analytical queries on the logs.

Reference Implementation:

A variety of technologies and frameworks, open source or commercial are available which provide log aggregation.

One such technology stack is ELK stack (or another variant called as EFK stack). ELK stands for Elastic Search, Log Stash and Kibana (The F in EFK stands for Fluentd).

1. Log collection and forwarding could be done by Log stash or by FLuentd libraries
2. Log storage could be performed using Elastic Search.
3. Visualizer and Dashboard could be served using Kibana

Most of the cloud platforms (Platform as a Service), support Log aggregation by channeling all the logs from the platform to a common point and support various log aggregator integrations.