How to run Filebeat with Docker and use it with ELK stack

We have seen how to install the ELK stack using  Docker Compose.
Now we add Filebeat, showing how to run it with Docker and use it with the ELK stack.
The final goal of this series of posts is in fact to show a complete example of how to read the logs of a microservice with Filebeat, and then to collect and visualize them through the ELK stack (ElasticsearchLogstashKibana).

Let’s assume we already have the ELK stack ready, after installing it as described in the previous post.
The first thing to run Filebeat on Docker is to pull it’s latest image.

docker pull

Download an example configuration file for Filebeat.

curl -L -O

We edit it to set our settings.
In addition to accessing Kibana, we indicate as input a folder of the container on which Filebeat will run and as output Logstash.

- type: log
  enabled: true
    - /logs/*.log
  host: "localhost:5601"
  username: "elastic"
  password: "changeme"

  hosts: ["localhost:5044"]

If we go to check the Logstash pipeline configuration, in the “docker-elk /logstash/pipeline/logstash.conf” file, we see that it is already set up to listen from Filebeat, port 5044.

input {
        beats {
                port => 5044

        tcp {
                port => 5000

## Add your filters / logstash plugins configuration here

output {
        elasticsearch {
                hosts => "elasticsearch:9200"
                user => "elastic"
                password => "changeme"
                ecs_compatibility => disabled

Create a directory that will cointains our log files to be read.
We suppose that this directory is inside our home directory, together with the Filebeat configuration file.

mkdir test-filebeat

Create inside the directory a test.log file with inside some dummy data.

INFO - File loaded
ERROR - File finished
INFO - Please provide a new file
INFO - File loaded

Now we can run the Filebeat container using our configuration.

docker run -d --name=filebeat --user=root --network="host" --volume="$HOME/test-filebeat:/logs" --volume="$HOME/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" filebeat -e --strict.perms=false

You can see the Filebeat container running together the ELK stack.

Filebeat with ELK stack on Docker Desktop dashboard

Check the Filebeat container logs.

docker logs -f filebeat

You will see that the test.log file has been read.

Filebeat container logs

Open Kibana, go to manage section, add a Kibana index pattern for Logstash, “logstash-*” using timestamp.

Kibana Logstash index timestamp

Go to the Kibana discover section and select last 24 hours.
You will able to see the data read from our test.log file.

Kibana Discover Filebeat log

In a future post we will create and add a Node.js microservice to our work, which will produce some more real log events.