How it works

Processing pipeline link

Each file goes through the following steps:

flowchart TD
    A[File detected] --> B[Wait for file to be fully written]
    B --> C[Generate unique key]
    C --> D[Read file into memory]
    D --> E[Resize to low-resolution]
    E --> F[Upload full-res and low-res to MinIO]
    F -->|success| G[Move to processed/]
    F -->|failure| H[Move to failed/]

File detection link

The service uses FileSystemWatcher to detect new .jpg files in the watch directory. In addition, a background sweep runs every 30 seconds to catch any files that may have been missed due to OS-level event buffering or dropped events during high load.

Each file path is tracked in a concurrent dictionary to ensure it is only processed once, even if both the watcher and the sweep detect it at the same time.

Stable file check link

Before processing a file, the service waits until it is fully written to disk. It polls the file size every 500 ms and considers the file ready when two consecutive checks return the same non-zero size and the file can be opened exclusively. This prevents reading a partially written file. The check times out after 2 minutes.

Key generation link

Each photo is assigned a unique key in the format {StandId}-{UUID}.jpg. The StandId comes from configuration and identifies the physical stand. The UUID prevents collisions between photos taken on the same stand.

Upload link

The full-resolution file and the resized version are uploaded to MinIO concurrently. A semaphore limits the total number of concurrent uploads across all files to 4.

Objects are stored at:

• {FullPrefix}/{key} for the original
• {LowPrefix}/{key} for the resized version

Retry and reconnection link

All MinIO operations use exponential backoff retry logic. When a failure occurs:

1. The first failure emits a MinIO connection lost warning
2. Each subsequent attempt waits twice as long as the previous one, up to RetryMaxDelayMs
3. When the operation eventually succeeds, a MinIO connection restored log is emitted

Non-retriable errors (invalid credentials, bad bucket name, access denied) are not retried and cause the operation to fail immediately.

Graceful shutdown link

When SIGTERM is received:

1. The file watcher stops raising events
2. No new files are accepted
3. The service waits for all in-progress uploads to complete
4. The process exits cleanly

The shutdown timeout is 5 minutes. Files that are queued but not yet started are cancelled and remain in the watch directory for processing on the next startup.

Logging link

All log entries are structured and use named parameters, compatible with any log aggregator that supports structured logging (e.g. Seq, Loki).

Key log points:

Set Logging__LogLevel__Default=Debug to see all steps including resize and upload start events.