Fluent Bit

Fluent Bit is a lightweight telemetry agent that is commonly used on Kubernetes nodes, virtual machines, and edge hosts. It tails log files, enriches records with metadata, batches events, and sends them to an output.

Use Fluent Bit with Parseable when you want a small agent close to the workload. For most log pipelines, Fluent Bit reads container or host logs and sends JSON events to Parseable over HTTP.

Prerequisites

You need Fluent Bit running on the host or cluster, a Parseable ingest endpoint, an API key with write access, and a dataset name. The examples below use kubernetes-logs and docker-logs.

Kubernetes Logs

In Kubernetes, Fluent Bit usually runs as a DaemonSet. It reads /var/log/containers/*.log, adds Kubernetes metadata, and forwards the records to Parseable.

[SERVICE]
    Flush        5
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            cri
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name          http
    Match         kube.*
    Host          <parseable-ingestor-host>
    Port          8000
    URI           /api/v1/ingest
    Format        json
    Compress      gzip
    Header        Content-Type application/json
    Header        X-API-Key <parseable-api-key>
    Header        X-P-Stream kubernetes-logs

Use the hostname only in Host. If your Parseable endpoint is HTTPS, set Port 443 and add tls On in the output block.

With the official Fluent Bit Helm chart, the same output can be placed in values.yaml:

config:
  outputs: |
    [OUTPUT]
        Name          http
        Match         kube.*
        Host          <parseable-ingestor-host>
        Port          8000
        URI           /api/v1/ingest
        Format        json
        Compress      gzip
        Header        Content-Type application/json
        Header        X-API-Key <parseable-api-key>
        Header        X-P-Stream kubernetes-logs

Then install or upgrade the chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit \
  --namespace fluent-bit \
  --create-namespace \
  --values values.yaml

Docker and Host Logs

For Docker hosts, Fluent Bit can tail container log files directly. Mount /var/lib/docker/containers into the Fluent Bit container when Fluent Bit itself runs in Docker.

[INPUT]
    Name              tail
    Tag               docker.*
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    DB                /var/log/flb_docker.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On

[FILTER]
    Name    record_modifier
    Match   docker.*
    Record  source_agent fluent-bit

[OUTPUT]
    Name          http
    Match         docker.*
    Host          <parseable-ingestor-host>
    Port          8000
    URI           /api/v1/ingest
    Format        json
    Compress      gzip
    Header        Content-Type application/json
    Header        X-API-Key <parseable-api-key>
    Header        X-P-Stream docker-logs

The same pattern works for host log files. Change Path to the files you want to collect, set a useful tag, and send the output to a dataset that matches the workload or environment.

Sending OTLP Logs

If Fluent Bit is receiving OpenTelemetry logs and you want to keep the OTLP format, send them to Parseable's OTLP logs endpoint instead of /api/v1/ingest.

[OUTPUT]
    Name          opentelemetry
    Match         *
    Host          <parseable-ingestor-host>
    Port          8000
    Logs_uri      /v1/logs
    Header        Authorization Bearer <parseable-api-key>
    Header        X-P-Stream fluent-bit-otel-logs

Use /api/v1/ingest for normal JSON events. Use /v1/logs only when the payload is OTLP logs.

Verify Logs

After Fluent Bit starts, check its own logs for HTTP status codes from Parseable. A successful output usually shows the records being flushed without retry errors.

In Parseable, open the dataset named in X-P-Stream. Kubernetes records should include pod, namespace, container, and node metadata when the Kubernetes filter is enabled. Docker or host records should include the fields produced by the parser and any fields added by your filters.

Troubleshooting

• If Fluent Bit cannot connect, confirm that the Host, Port, and TLS settings match the Parseable ingest endpoint.
• If Parseable returns 401 or 403, check that the Bearer token is present and the key can write to the dataset.
• If Parseable returns 400, confirm that the output uses JSON for /api/v1/ingest, or OTLP logs for /v1/logs.
• If Kubernetes metadata is missing, check the service account permissions and the Kubernetes filter configuration.
• If Fluent Bit retries often, reduce batch size, check network latency, and make sure the Parseable ingestor has enough capacity for the incoming volume.