Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions pipeline/outputs/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,22 @@ Multipart uploads are ideal for most use cases because they allow the plugin to

One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the `store_dir`, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the `store_dir` files will still be present on restart).

### Buffer files from a previous run

Fluent Bit version 5.1.2 and greater measures and cleans up the orphaned buffer files it finds in the `store_dir` at startup. Only files left behind by a previous run are affected. Each run creates its own timestamped directory in the `store_dir` for the data it buffers, so the startup cleanup skips the directory belonging to the current run, along with the `multipart_upload_metadata` and `quarantine` directories.

The plugin uploads each orphaned buffer file with the `PutObject` API and deletes the local file after a successful upload. It skips any file that's locked for an in-progress upload, and a file that fails to upload keeps its failure count so the plugin can retry it later. When every file in a directory from a previous run is cleared, the plugin removes the empty directory.

When the plugin restores a buffer file from a previous run, it measures the file and counts its size toward `store_dir_limit_size`. In earlier versions, the plugin couldn't determine the size of a restored file that wasn't loaded into memory. Such a file didn't count toward the limit, so the `store_dir` could hold more data than the limit allows.

A restored chunk that already exceeded `retry_limit` during the previous run isn't retried again. The plugin logs a warning naming the tag and the failure count:

```text
Chunk for tag your-tag failed to send 6/5 times, will not retry
```

The plugin then applies `retry_exhausted_action` to that chunk, moving it to the quarantine area or deleting it. In earlier versions the chunk stayed in the `store_dir` and kept consuming buffer space. The same handling applies to the buffer sweep Fluent Bit performs on shutdown.

### Using S3 without persisted disk

If you run Fluent Bit in an environment without persistent disk, or without the ability to restart Fluent Bit and give it access to the data stored in the `store_dir` from previous executions, some considerations apply. This might occur if you run Fluent Bit on [AWS Fargate](https://aws.amazon.com/fargate/).
Expand Down
Loading