From bb9ccae332b5e6c3ec64790d73d6ff4fde924f91 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Sat, 5 Sep 2026 16:47:13 +0200 Subject: [PATCH] docs: pipeline: outputs: s3: document handling of buffer files from a previous run Document the startup buffer accounting and orphan cleanup added in fluent-bit f298f03df, b068b24de, and 1ad5eb6d6. - Add a Buffer files from a previous run subsection under Reliability - Note that restored buffer files are now measured and counted toward store_dir_limit_size, and that earlier versions couldn't size a file that wasn't loaded into memory so it escaped the limit - Note that a restored chunk which already exceeded retry_limit isn't retried, and record the warning the plugin logs - Note that retry_exhausted_action then applies to that chunk, and that earlier versions left it consuming space in the store_dir - Note that the same handling applies to the shutdown buffer sweep - State that the startup cleanup only targets orphaned buffer files left by a previous run - Note that the current run's timestamped directory, the multipart_upload_metadata directory, and the quarantine directory are skipped - Describe how orphaned files are handled: PutObject upload, local delete on success, locked files skipped, failure count preserved for retry, empty previous-run directory removed Signed-off-by: Eric D. Schabell --- pipeline/outputs/s3.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/pipeline/outputs/s3.md b/pipeline/outputs/s3.md index 6c0e6b2ae..601ef1992 100644 --- a/pipeline/outputs/s3.md +++ b/pipeline/outputs/s3.md @@ -450,6 +450,22 @@ Multipart uploads are ideal for most use cases because they allow the plugin to One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the `store_dir`, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the `store_dir` files will still be present on restart). +### Buffer files from a previous run + +Fluent Bit version 5.1.2 and greater measures and cleans up the orphaned buffer files it finds in the `store_dir` at startup. Only files left behind by a previous run are affected. Each run creates its own timestamped directory in the `store_dir` for the data it buffers, so the startup cleanup skips the directory belonging to the current run, along with the `multipart_upload_metadata` and `quarantine` directories. + +The plugin uploads each orphaned buffer file with the `PutObject` API and deletes the local file after a successful upload. It skips any file that's locked for an in-progress upload, and a file that fails to upload keeps its failure count so the plugin can retry it later. When every file in a directory from a previous run is cleared, the plugin removes the empty directory. + +When the plugin restores a buffer file from a previous run, it measures the file and counts its size toward `store_dir_limit_size`. In earlier versions, the plugin couldn't determine the size of a restored file that wasn't loaded into memory. Such a file didn't count toward the limit, so the `store_dir` could hold more data than the limit allows. + +A restored chunk that already exceeded `retry_limit` during the previous run isn't retried again. The plugin logs a warning naming the tag and the failure count: + +```text +Chunk for tag your-tag failed to send 6/5 times, will not retry +``` + +The plugin then applies `retry_exhausted_action` to that chunk, moving it to the quarantine area or deleting it. In earlier versions the chunk stayed in the `store_dir` and kept consuming buffer space. The same handling applies to the buffer sweep Fluent Bit performs on shutdown. + ### Using S3 without persisted disk If you run Fluent Bit in an environment without persistent disk, or without the ability to restart Fluent Bit and give it access to the data stored in the `store_dir` from previous executions, some considerations apply. This might occur if you run Fluent Bit on [AWS Fargate](https://aws.amazon.com/fargate/).