Skip to content

Log from pipes #49

Closed
Closed

Description

@j0hnsmith

It would be great to pipe to remote_syslog.

Activity

  1. troy commented on Dec 11, 2014

    @troy
    Contributor

    We considered that too, but check out logger. It's available with most OSes that have GNU or BSD tools installed, and does exactly this.

    If logger doesn't do it, can you explain more about your environment, what you'd pipe, and why? Thanks, too!

  2. troy commented on Dec 11, 2014

    @troy
    Contributor

    Oh, here's the manpage for logger. Basically it can read from a pipe or file and send to the local syslog, which sends to Papertrail.

  3. j0hnsmith commented on Dec 11, 2014

    @j0hnsmith
    Author

    I'm using svlogd which copies when rotating [1], there doesn't seem to be any hooks to restart remote_syslog so I'd like to be able to replace svlogd with remote_syslog

    http://smarden.org/runit/svlogd.8.html#sect4

  4. troy commented on Dec 11, 2014

    @troy
    Contributor

    Thanks for the extra info, that helps a lot. I can see a couple possibilities here:

    • restart remote_syslog2, by using the PID file it writes (ie, what the restart handlers in these init files do -- more on this method).
    • use svlogd's u config option and have it send to to Papertrail
    • I'd say add HUP handling to remote_syslog, except that I don't see any evidence that svlogd has an easier way to send a HUP than it does to simply restart

    If remote_syslog2 supported piped input, what would handle the other tasks that svlogd does in your environment, like rotation? If you don't need or want rotation, it seems like the u flag or piping to logger would both work equally well, but I may be missing constraints.

  5. j0hnsmith commented on Dec 12, 2014

    @j0hnsmith
    Author

    Is UDP the standard way to transmit logs to papertrail? I'd much prefer to use TCP rather than UDP, also I'm not sure that TLS is supported.

    If remote_syslog were to take input from stdin (eg a pipe) I think it should output everything it receives (configurable perhaps) untouched to stdout (even if it couldn't send the log messages for any reason, maybe errors output to a separate remote_syslog log file).

    Here's an example using the command I currently run

    exec svlogd -tt /var/log/remote_syslog_runit
    

    It would be cool if I could do

    exec remote_syslog -n logfile-name-in-papertrail.log -d some-host -p 12345 -l path/to/remote_syslog.log | svlogd -tt /var/log/remote_syslog_runit
    

    By having remote_syslog handle the messages before they get to the logfile all the rotation issues/other complications go away, additionally there's no remote_syslog process to be managed, it becomes a part of the process that's being logged.

  6. j0hnsmith commented on Dec 19, 2014

    @j0hnsmith
    Author

    I wrote a python script to do this however it's not a viable option as each new log line makes a new TLS connection which takes around 0.7s, so it's impractical.

    To use an existing connection for multiple messages you have to have some kind of daemon which keeps the connection, I have ideas for how it could work but it's a far more complex solution than I initially thought.

  7. troy commented on Dec 19, 2014

    @troy
    Contributor

    @j0hnsmith agreed all around. UDP solves that (and is supported ~specifically to solve that problem of non-blocking stateless "fire and forget" logging), but it sounds like you don't want to use UDP for other reasons.

    I think we'd consider a pull request to remote_syslog which accepts data from a pipe and/or outputs it, but that's a much bigger undertaking and I think most implementors prefer to keep a local copy of logs anyway.

    It's sort of tangential to this, but the sv* tool set has an unusual enough set of features and preferred implementation that it leads to wanting this. That came up once before and it convinced me I'd be better served by a different init structure, even aside from this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址