In this series of posts I am going to cover everything needed to get Elasticsearch, Logstash and Kibana (ELK) up and running on Ubuntu 16.04. In the videos I use the desktop version of Ubuntu, but the process should be the same on the server version. In addition to the ELK stack I will also be installing Nginx to be used as the proxy in front of Kibana for user authentication, since there is no built in authentication. The proxy will also provide a secure connection over HTTPS to Kibana.
If you are not familiar with the ELK stack, the general idea here is to take all of the logs you want from all of the systems you want to monitor and ship them to the ELK stack to be indexed and stored.
Once the logs are ingested into the stack you can then easily search, view and analyze the logs using Kibana.
This allows us to quickly search through large amounts of logs looking for everything from misconfigurations, OS/Application/Hardware errors, security related events and so on. Normally gathering this sort of information would require pulling the logs from each machine individually and then manually parsing through hundreds (or more) of individual log files, which as you can imagine would take a considerable amount of time and resources.
For this series I will be using a single VM to run all of the services, but once you are familiar with the setup process the ELK stack can be broken up and scaled out relatively easily. I will be giving the VM 4 vCPUs and 4GB of RAM which is a decent place to start for an ELK server, but once things are up and running with data flowing in, more memory will likely be needed as Elasticsearch can be very memory intensive.
After the ELK server has been setup, I then cover setting up Winlogbeat to gather Windows Event Logs and Filebeat to pick up the flat file logs (IIS) from a remote Windows 2012 R2 server.
Here is where you can find the individual posts:
ELK 5 on Ubuntu: Pt. 1 – Installing and Configuring Ubuntu 16.04
ELK 5 on Ubuntu: Pt. 2 – Installing and Configuring Elasticsearch, Logstash, Kibana & Nginx
ELK 5 on Ubuntu: Pt. 3 – Installing and Configuring Beats Agents on Windows Clients