Skip to content

Revise the method of file size limiting #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 19, 2016

Conversation

nblumhardt
Copy link
Member

While investigating datalust/serilog-sinks-seq#28, I found a few unsatisfactory things lurking in the way this sink limits file size:

  • By limiting character count exactly, formats like JSON are corrupted when the limit is reached
  • Assumption of 1 char == 1 byte written
  • Overhead of counting every single byte written, skipping some optimisations that would otherwise be performed by StreamWriter
  • No way to surface to the sink that the file size has been reached
  • Unnecessary synchronisation - the counter doesn't need to be thread-safe as the lock in the outer sink covers it

(The counting machinery is necessary because FileStream.Length is an expensive call in comparison.)

This PR is an attempt to address these things. Primarily, it moves from a "character limiting text writer" to a "length counting stream wrapper". The sink uses knowledge of the count to decide whether or not to write whole events instead of letting the last event be truncated.

Some caveats:

  • Because whole events are written, the count will always be (slightly) exceeded; this seems fine to me as it still allows the sink to prevent runaway disk space, but might be a source of some confusion. XML docs updated.
  • Because the StreamWriter over the file will itself internally buffer before passing data through the stream, the partial information buffered in it (1024 bytes) may also cause the eventual file size to be exceeded.

Perf benefit: despite timings being I/O bound, the included sample program that writes 1M events to a non-buffered, size-limited log file completes with the new version in 0.93x the execution time of the current RTM sink (~13700 ms vs ~14300 ms, i7/SSD).

@serilog/reviewers-core more eyes the merrier on this one! :-)

{
if (stream == null) throw new ArgumentNullException(nameof(stream));
_stream = stream;
_countedLength = stream.Length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't this throw a NotSupportedException if the stream doesn't support seeking? It might not matter since we're in control of which types of streams that gets passed here, in this case, FileStream, but are there cases where a FileStream can throw?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it's probably less general than it appears 👍 - in the worst case though, the exception will be pretty easy to track down, so inclined to leave this one. Thanks!

@khellang
Copy link
Member

LGTM :shipit: Only thing is the potential exception when calling stream.Length, but it might not matter.

@nblumhardt nblumhardt merged commit 6805678 into serilog:dev Aug 19, 2016
@nblumhardt nblumhardt mentioned this pull request Aug 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants