DEV Community

Dhwaneet Bhatt
Dhwaneet Bhatt

Posted on • Edited on • Originally published at dhwaneetbhatt.com

Time based log file rotation with zap

#go

zap is one of the nicely maintained, well-written and nicely performing open source logging libraries in Go. Unfortunately, not all apps in our organization are 12-factor apps and thus we still log to physical files. So, I needed a way to rotate log files while using zap. I come from Java background, and log4j has a very vast array of log rotation options. But I found it lacking in zap, clearly because in the world of containers, logging to stdout (treating logs as event streams - 12-factor logs) is very common.

To be specific, I was looking to do time based rotation (one log file per hour, helps in debugging) and not size based. On zap's FAQ, they've shown integration with lumberjack, which only supports size based rotation.

I found another library file-rotatelogs which does support time based rotation. So I used it along with zap.

The good thing about zap is that it takes any io.Writer interface as a WriteSyncer and file-rotatelogs returns *RotateLogs which implements the io.Writer interface.

Here the rotatelogs.WithRotationTime(time.Hour)) indicates hourly rotation. rotatelogs.WithMaxAge(60*24*time.Hour) indicates the files are cleaned up after 60 days.

package main

import (
    "encoding/json"
    "time"

    rotatelogs "github.com/lestrrat-go/file-rotatelogs"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func main() {
    // initialize the rotator
    logFile := "/var/log/app-%Y-%m-%d-%H.log"
    rotator, err := rotatelogs.New(
        logFile,
        rotatelogs.WithMaxAge(60*24*time.Hour),
        rotatelogs.WithRotationTime(time.Hour))
    if err != nil {
        panic(err)
    }

    // initialize the JSON encoding config
    encoderConfig := map[string]string{
        "levelEncoder": "capital",
        "timeKey":      "date",
        "timeEncoder":  "iso8601",
    }
    data, _ := json.Marshal(encoderConfig)
    var encCfg zapcore.EncoderConfig
    if err := json.Unmarshal(data, &encCfg); err != nil {
        panic(err)
    }

    // add the encoder config and rotator to create a new zap logger
    w := zapcore.AddSync(rotator)
    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encCfg),
        w,
        zap.InfoLevel)
    logger := zap.New(core)

    logger.Info("Now logging in a rotated file")
}
Enter fullscreen mode Exit fullscreen mode

This will create the file /var/log/app-2020-05-18-15.log, which is exactly how we wanted it. It will automatically start logging to the next file during the start of the next hour.

Top comments (4)

Collapse
 
simme profile image
Simme

Hi, and welcome to dev.to! 👋🏼

To keep the user experience pleasant and allowing users to browse interesting content without having to leave the site, we encourage authors to post their content here rather than to link to external sources.

I understand, though, that you might want the posts to originate from your own blog. I feel the same way, which is why I use the canonical URL property on my posts and post them in their entirety. You can actually even set up your dev.to account to sync posts from an external RSS feed.

See dev.to/settings/publishing-from-rss for more info. 👍🏼

Best regards,
Simme

Collapse
 
dhwaneetbhatt profile image
Dhwaneet Bhatt

Thanks for the detailed explanation Simon.

Collapse
 
hariprasadraja profile image
Hariprasad

It's a good one but i won't prefer that. If you move your logs to any database for analytical purpose is much appreciated than this.

Collapse
 
dhwaneetbhatt profile image
Dhwaneet Bhatt • Edited

We are preparing our log stack, but it will take time. We are still early stages of a startup and for now, we use files for debugging and finding out errors.