Repository navigation
-f does not format new logs #48
Description
Our settings:
- version v1.0.0-alpha
- json format log files
- custom config since time and key levels are different
Problem:
- when I watch file with -f parameter and new log event comes to the file then an empty record is shown followed by unformatted log event:
333
334 "asctime":"2021-03-04T08:30:32+0100".......
- if I quit jog and start watching it again (with or without -f) everything looks normal - without empty records and everything formatted.
- Well almost, I have also noticed that any time in any mode there is a trailing empty record always shown, even though there is not even a new line in the original log file, like:
289 <last log event>
290
- I think that might be the thing that breaks it with -f mode. But I dont know how to get rid of that.
our config:
colorization: true
pattern: "${timestamp} - ${level} - ${app} - ${file}:${message}${stacktrace}${others}"
line-no:
print: true
color: FgRed
print-format: "\n%s"
unknown-line:
print: true
color: FgGray
fields:
others:
name:
color: FgYellow
separator:
label: =
color: FgDefault
value:
color: FgDefault
print-format: "\n%s"
level: # 'level' field is a hard-coded field used for level filter as well, so don't rename it
# And, it must be an enum.
print-format: "%s" # follows https://golang.org/pkg/fmt/
alias: "levelname"
case-sensitive: false
enums:
case-sensitive: false
NOTSET:
alias: notset,0
color: FgGray,OpBold
DEBUG:
alias: debug,10
color: FgGreen,OpBold
INFO:
alias: info,20,
color: FgBlue,OpBold,OpFuzzy
WARNING:
alias: warn,warning,30
color: FgYellow,OpBold
ERROR:
alias: error,err,40
color: FgMagenta,OpBold
CRITICAL:
alias: critical,50
color: FgRed,OpBold
app:
print: false
alias: "name"
case-sensitive: false
color: FgLightBlue
file:
alias: "filename"
case-sensitive: false
color: FgDefault
message:
alias: "message"
case-sensitive: false
color: FgCyan,OpItalic
print-format: "\n%s"
stacktrace:
alias: "exc_info"
case-sensitive: false
color: FgDefault
print-format: "\nStack trace: \n%s"
timestamp: # 'timestamp' field is a hard-coded field used for time range filter as well, so don't rename it
alias: "asctime"
case-sensitive: false
color: FgYellow
Activity
qiangyt commentedon Mar 4, 2021
@LukasJerabek , is it possible for you to attach the log files with sensitive information get rid of?
LukasJerabek commentedon Mar 4, 2021
file named xxx-yyy.zzz.log has records like:
{"asctime":"2021-03-04T07:59:25+0100","levelname":"INFO","name":"some_component","filename":"some_file","message":"some_message"}
{"asctime":"2021-03-04T08:02:46+0100","levelname":"CRITICAL","name":"other_component2","filename":"other_file2","message":"other_message2"}
{"asctime":"2021-03-04T09:31:20+0100","levelname":"WARNING","name":"other_component","filename":"other_file","message":"other_message","exc_info":"optional_traceback"}
is this enought?
qiangyt commentedon Mar 4, 2021
Yes, I will check.
LukasJerabek commentedon Mar 4, 2021
One more thing I have noticed in -f mode, If a second log comes fast (/immediately) after the first one, the empty record line is not shown and second log is formatted nicely. If there is some delay after the second log, the empty record line is shown and the third log would be again badly formatted.
qiangyt commentedon Mar 5, 2021
@LukasJerabek,
I tried but cannot reproduce the issue.
Actually the new empty line always appears before all records but I think they're from your configuration? (search '\n' in your configuration yaml file).
Below is what I observed, the 'issue48.jog.yml' the configuration file you pasted above, the issue48.log is the log rows you paster above too:
6 is the original one, line 78 is the appended lines after several seconds.LukasJerabek commentedon Mar 5, 2021
I think you missunderstood. Empty line is alright. Empty record is the problem - meaning number without any text as in examples above. And when I see the original log file, there is not anything why it should be there. How to get rid of that empty record when using jog. Obviously you dont have it in you outputs, otherwise the -f mode would look this way in my case:
LukasJerabek commentedon Mar 5, 2021
this is what it looks like. First three records where already in file when I issued jog -f ... and the other to came after few seconds. There is only 6 records in original file, but I see 8 numbers in the output.
qiangyt commentedon Mar 5, 2021
I think it is related to how your log source outputs. It seems your log source outputs each record as 2 separated steps: first, a '{' first, then remaining part of the new record. And then jog takes it as 2 records: the first '{' letter is ignored, and the second remaining part is printed by jog as 'unknow-line'. You can find the 'unknown-line' settings in your configuration yaml as:
It is printed with gray which matches your settings.
Let me find how to fix.
LukasJerabek commentedon Mar 5, 2021
could be, but even then I would not understand the last empty record (number 8 on the screenshot)
LukasJerabek commentedon Mar 5, 2021
I forgot to mention, in this example I made it manually. I simply opened the file in vim added logs which are shown on lines 5, 6,7 manually closed and saved the file. So under this circumstance it seems unrealted to log source output to me...
fix issue #48
qiangyt commentedon Mar 11, 2021
Release 1.0.1 to fix this issue.