README
¶
RollingWriter
RollingWriter is an auto rotate io.Writer implementation. It can work well with logger.
This repo is a fork of this rollingwriter repo. But I made it an independent repo as there were many changes for simplification and performance. Major changes are:
- The logs are always added asynchronouly to the file
- Log messages are always buffered first and hence many logs may be added to the underlying file in one shot
- Writing to the file and roll over of the file is handled by by the same go routine and hence the need for synchronization across different go routines are no longer necessary
- Locks are removed. All communications are through channels without explicit synchronization with locks
RollingWriter contains 2 separate patrs:
-
Manager: decide when to rotate the file with policy. RlingPolicy give out the rolling policy
- WithoutRolling: no rolling will happen
- TimeRolling: rolling by time
- VolumeRolling: rolling by file size
-
Writer: impement the io.Writer and do the io write
- Concurrent and safe for adding logs from multiple go routines
- Logs are added asynchronously to file without blocking the callers
- Log messages are buffered and write to files may add multiple messages in operation
Features
- Auto rotate with multi rotate policies
- Implement parallel and safe io.Writer
- Max remain rolling files with auto cleanup
- Easy for user to implement your manager
Benchmark
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/nipuntalukdar/rollingwriter
cpu: Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz
BenchmarkWrite-16 1000000 1143 ns/op 1 B/op 0 allocs/op
BenchmarkParallelWrite-16 1000000 1362 ns/op 1 B/op 0 allocs/op
PASS
ok github.com/nipuntalukdar/rollingwriter 2.596s
Quick Start
writer, err := rollingwriter.NewWriterFromConfig(&config)
if err != nil {
panic(err)
}
writer.Write([]byte("hello, world"))
or
writer, err := rollingwriter.NewWriterFromConfigFile("/path/to/config.json")
if err != nil {
panic(err)
}
writer.Write([]byte("hello, world"))
For details, check demo folder for more details.
Detailded examples with confifg file are given.
To run the examples:
cd demo
go build
# We will get the demo executable.
./demo --help
Usage of ./demo:
-configfile string
The config JSON file to use (default "config.json")
-configfromfile
Read config from file
# An example run
./demo --configfromfile --configfile config.json
An example for logger integration with hashicorp logger is available here. Any suggestion or new features needed, please create an issue .
Documentation
¶
Index ¶
Constants ¶
const ( WithoutRolling = iota TimeRolling VolumeRolling // DefaultFileMode set the default open mode rw-r--r-- by default DefaultFileMode = os.FileMode(0644) // DefaultDirMode set the default open mode rwx------ by default DefaultDirMode = os.FileMode(0700) // MinQueueSize define the minimum queue size for asynchronize write DefaultQueueSize = 8 * 1024 // MinBufferSize define the minimum buffer size for log messages, 1 MB DefaultBufferSize = 1024 * 1024 // MinQueueSize define the minimum queue size for asynchronize write MinQueueSize = 64 // MinBufferSize define the minimum buffer size for log messages MinBufferSize = 2048 )
RollingPolicies giveout 3 policy for rolling.
Variables ¶
var ( // Precision defined the precision about the reopen operation condition // check duration within second Precision = 1 // Max write to file interval in seconds MaxWriteInterval = 1 // DefaultFileFlag set the default file flag DefaultFileFlag = os.O_RDWR | os.O_CREATE | os.O_APPEND // ErrInternal defined the internal error ErrInternal = errors.New("error internal") // ErrClosed defined write while ctx close ErrClosed = errors.New("error write on close") // ErrInvalidArgument defined the invalid argument ErrInvalidArgument = errors.New("error argument invalid") // ErrQueueFull defined the queue full ErrQueueFull = errors.New("async log queue full") )
Functions ¶
Types ¶
type Config ¶
type Config struct {
// FilePath defines the full path of log file
// there comes out 2 different log file:
//
// 1. the current active file
// file path is located @:
// FilePath
//
// 2. the truncated log file
// the truncated log file is backup here:
// [fileDir]/[fileName].[fileExt].[TimeTag]
// if compressed true
// [fileDir]/[fileName].[fileExt].gz.[TimeTag]
//
// NOTICE: blank field will be ignored
// By default we using '-' as separator, you can set it yourself
TimeTagFormat string `json:"time_tag_format,omitempty"`
FilePath string `json:"file_path,omitempty"`
// Mode of log files created
FileMode os.FileMode `json:"file_mode,omitempty"`
// Directory mode, mode of directory created
DirMode os.FileMode `json:"dir_mode,omitempty"`
// MaxBackups is the maximum number of old files to retain, if set 0 will
// all old files will be retained.
MaxBackups int `json:"max_remain,omitempty"`
// RollingPolicy give out the rolling policy
// We got 3 policies(actually, 2):
//
// 1. WithoutRolling: no rolling will happen
// 2. TimeRolling: rolling by time
// 3. VolumeRolling: rolling by file size
RollingPolicy int `json:"rolling_ploicy,omitempty"`
RollingTimePattern string `json:"rolling_time_pattern,omitempty"`
RollingVolumeSize string `json:"rolling_volume_size,omitempty"`
// Compress will compress log file with gzip
Compress bool `json:"compress,omitempty"`
// FilterEmptyBackup will not backup empty file if you set it true
FilterEmptyBackup bool `json:"filter_empty_backup,omitempty"`
// Maximum buffer size
BufferSize int `json:"max_buffer_size,omitempty"`
// Max queue size for log messages
QueueSize int `json:"max_queue_size,omitempty"`
}
Config give out the config for manager
type FileMonitor ¶
type FileMonitor interface {
// RotationEvents returns a string channel
// when rotation events occur, new backup filenames will be sent on this channel
RotationEvents() chan string
// Close the Manager
Close()
}
FileMonitor writes new backup file event.
func NewManager ¶
func NewManager(c *Config) (FileMonitor, error)
NewManager generate the Manager with config
type LogFileFormatter ¶
LogFileFormatter log file format function
type Option ¶
type Option func(*Config)
Option defined config option
func WithCompress ¶
func WithCompress() Option
WithCompress will auto compress auto rotated files with gzip
func WithFilePath ¶
WithFilePath set the full path of rolling file, if dir tree does not exist, it will be created
func WithMaxBackups ¶
WithMaxBackups sets the maximum number of backup files to retain 0 will disable pruning backups files, this is the default behaviour
func WithRollingTimePattern ¶
WithRollingTimePattern set the time rolling policy time pattern obey the Corn table style visit http://crontab.org/ for details
func WithRollingVolumeSize ¶
WithRollingVolumeSize set the rolling file truncation threshold size
func WithTimeTagFormat ¶
WithTimeTagFormat set the TimeTag format string
type RollingWriter ¶
RollingWriter implement the io writer
func NewWriter ¶
func NewWriter(ops ...Option) (RollingWriter, error)
NewWriter generate the rollingWriter with given option
func NewWriterFromConfig ¶
func NewWriterFromConfig(c *Config) (RollingWriter, error)
NewWriterFromConfig generate the rollingWriter with given config
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer provide a synchronous file writer if Lock is set true, write will be guaranteed by lock
func (*Writer) RotateFile ¶
RotateFile do the rotate, open new file and swap FD then trate the old FD
Source Files