Filebeat

Filebeat is a lightweight shipper for files, container logs, and common application modules. It is usually deployed close to the workload, reads new log lines, adds host or container metadata, and forwards those events to an output.

For Parseable, the clean path is Filebeat to Logstash, then Logstash to Parseable over HTTP. Filebeat keeps doing collection and enrichment, while Logstash handles the HTTP delivery to Parseable with the required API key and dataset headers.

Prerequisites

You need Filebeat on the host or Kubernetes node, Logstash reachable from Filebeat, a Parseable ingest endpoint, an API key, and a dataset name. The examples below use filebeat-logs.

Configure Filebeat

Create or update filebeat.yml so Filebeat reads your logs and sends them to Logstash.

filebeat.inputs:
  - type: filestream
    id: app-logs
    enabled: true
    paths:
      - /var/log/myapp/*.log
    fields:
      source_agent: filebeat
    fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

For Docker logs, mount /var/lib/docker/containers into the Filebeat container and add Docker metadata:

filebeat.inputs:
  - type: filestream
    id: docker-logs
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log
    parsers:
      - container: ~
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"

output.logstash:
  hosts: ["logstash:5044"]

For Kubernetes, Filebeat can read /var/log/containers/*.log and attach Kubernetes metadata:

filebeat.inputs:
  - type: filestream
    id: kubernetes-logs
    enabled: true
    paths:
      - /var/log/containers/*.log
    parsers:
      - container: ~
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

output.logstash:
  hosts: ["logstash:5044"]

If you use Filebeat modules, keep them enabled. The module output can still go to Logstash, and Logstash can forward the enriched events to Parseable.

Configure Logstash

Create a Logstash pipeline that receives Beats events and sends them to Parseable.

input {
  beats {
    port => 5044
  }
}

filter {
  mutate {
    add_field => { "source_pipeline" => "filebeat-logstash" }
  }
}

output {
  http {
    url => "http://<parseable-ingestor-endpoint>:8000/api/v1/ingest"
    http_method => "post"
    format => "json_batch"
    content_type => "application/json"
    headers => {
      "X-API-Key" => "<parseable-api-key>"
      "X-P-Stream" => "filebeat-logs"
    }
  }
}

Use the same Logstash pattern if Filebeat reads Nginx, MySQL, system logs, or application logs through modules. The output block is the part that decides where the records land in Parseable.

Verify Logs

Run filebeat test config to confirm the Filebeat configuration is valid. After Filebeat and Logstash start, check the Logstash logs for successful HTTP responses from Parseable, then open filebeat-logs in Parseable and look for recent events.

If no data appears, check the path Filebeat is reading, confirm Filebeat can reach Logstash on port 5044, and confirm Logstash can reach the Parseable ingest endpoint with the configured API key.