Python Client support

Using the UDPLog client directly

The Python UDPLog client is in udplog.udplog. Use like so:

from udplog import udplog
logger = udplog.UDPLogger()
logger.log('some_category', {'a_key': 'a_value', 'another_key': 17})

The dictionary passed as the second arg must be safe for JSON encoding.

Using the Python logging facility

This is the preferred method when the log events should also appear in local log files. Every Python source file will add this at the top:

import logging

logger = logging.getLogger(__name__)

This will set up a local logger that carries the module path as the logger name.

Then, logger is a logging.Logger on which you can call methods like debug(), info() and error():

logger.info("Function %(func)s took %(time_elapsed)s",
            {'func': func,
             'time_elapsed': time.time() - start_time})

The logged message will be passed through the Python logging facility, to be delivered to log handlers. Using a format string and a dictionary with values, instead of a pre-formatted log message, makes those values individually available to the log handlers. For UDPLogHandler, explained below, those values become available as keys in the JSON sent out. Then, when the log events are sent on to Logstash, they appear as fields that can be used for filtering and queries.

When an exception occurs that needs to be logged, you can use exception() from an exception handler. This will allow log handlers to add tracebacks to the logged entries. In this case, UDPLogHandler will add three fields to the log event: excText, excType and excValue. Respectively, they hold the rendered traceback, the exception type and the exception value (usually the arguments passed to the exception when raised):

a = {}
try:
    b = a['foo']
except:
    log.exception("Oops, no %(bar)s", {'bar': 'foo'})

UDPLogHandler further adds fields for the filename and the line number where the log event was created (filename and lineno), as well as the function name (funcName) and the log level and logger name (logLevel and logName).

The log handler usually only has to be setup once per application, for example in the application’s main function:

logging.basicConfig()
root = logging.getLogger()
root.setLevel(logger.INFO)
root.addHandler(udplog.UDPLogHandler(UDPLogger(), 'python_logging'))

This will set up default logging to stderr, set the minimum log level to INFO and then add a handler for logging to UDPLog. The second argument passed to the handler is the UDPLog category. You can override this on individual log events by adding a category field in the dictionary passed as the second argument to the log methods.

The handler also supports the extra keyword argument to the logger methods, adding the values to the emitted dictionary. The logging module has the very useful LoggerAdapter to wrap a regular logger to add extra data to every log event.

A complete Python logging example:

"""
Example of using the standard logging facility to send events to udplog.
"""

import logging
import socket
import warnings

from udplog.udplog import UDPLogger, UDPLogHandler

# Get a logger in the idiomatic way.
logger = logging.getLogger(__name__)

# Set up logging to stdout
logging.basicConfig(level=logging.DEBUG)

# Capture warnings, too.
logging.captureWarnings(True)

# Add the UDPLog handler to the root logger.
udplogger = UDPLogger(defaultFields={
                          'appname': 'example',
                          'hostname': socket.gethostname(),
                          })
root = logging.getLogger()
root.setLevel(logging.DEBUG)
root.addHandler(UDPLogHandler(udplogger, category="python_logging"))

def main():
    logger.debug("Starting!")
    logger.info("This is a simple message")
    logger.info("This is a message with %(what)s", {'what': 'variables'})

    extra_logger = logging.LoggerAdapter(logger, {'bonus': 'extra data'})
    extra_logger.info("Bonus ahead!")

    a = {}
    try:
        print a['something']
    except:
        logger.exception("Oops!")

    warnings.warn("Don't do foo, do bar instead!", stacklevel=2)

main()

The call to exception() results in this event dictionary:

{
  "appname": "example",
  "category": "python_logging",
  "excText": "Traceback (most recent call last):\n
                File \"doc/examples/python_logging.py\", line 39, in main\n
                  print a['something']\n
              KeyError: 'something'",
  "excType": "exceptions.KeyError",
  "excValue": "'something'",
  "filename": "doc/examples/python_logging.py",
  "funcName": "main",
  "hostname": "localhost",
  "lineno": 41,
  "logLevel": "ERROR",
  "logName": "__main__",
  "message": "Oops!",
  "timestamp": 1379508311.437895
}

Using the Twisted logging system

Twisted has its own logging system in twisted.python.log and udplog.twisted.UDPLogObserver can be set up to send all logged events onto a UDPLog server. It has special support for rendering exceptions and warnings. See UDPLogObserver.emit for details.

The following Twisted logging example sets up the log observer and uses the Twisted logging system to emit a few log events:

"""
Example of using the Twisted logging facility to send events to udplog.
"""

import sys
import warnings

from twisted.python import log

from udplog.udplog import UDPLogger
from udplog.twisted import UDPLogObserver

# Set up logging to stdout
log.startLogging(sys.stdout)

# Set up the udplog observer
udplogger = UDPLogger()
observer = UDPLogObserver(udplogger, defaultCategory='twisted_logging')
log.addObserver(observer.emit)

def main():
    log.msg("Starting!")
    log.msg("This is a simple message")
    log.msg(format="This is a message with %(what)s", what='variables')

    a = {}
    try:
        print a['something']
    except:
        log.err(None, "Oops!")

    warnings.warn("Don't do foo, do bar instead!", stacklevel=2)

main()

The call to log.err has None as the first argument, so that the most recent exception is retrieved from the execution context. Alternatively, you can pass an exception instance or failure.Failure. The second argument is the why of the log event, and ends up in the message field of the event dictionary:

{
  "category": "twisted_logging",
  "excText": "Traceback (most recent call last):\n
                File \"doc/examples/twisted_logging.py\", line 34, in <module>\n
                  main()\n
              --- <exception caught here> ---\n
                File \"doc/examples/twisted_logging.py\", line 28, in main\n
                print a['something']\nexceptions.KeyError: 'something'\n",
  "excType": "exceptions.KeyError",
  "excValue": "'something'",
  "isError": true,
  "logLevel": "ERROR",
  "message": "Oops!",
  "system": "-",
  "timestamp": 1379507871.564469
}

The warning is rendered as follows:

{
  "category": "twisted_logging",
  "filename": "doc/examples/twisted_logging.py",
  "isError": false,
  "lineno": 34,
  "logLevel": "WARNING",
  "message": "Don't do foo, do bar instead!",
  "system": "-",
  "timestamp": 1379507871.564662,
  "warningCategory": "exceptions.UserWarning"
}