Start your Python project with optparse and logging

Python continues to be my favorite language to hack in. It’s useful for tasks big and small and has the advantage of being more readable and maintainable than a lot of other scripting languages. I remember when, having learned the basics of Python, I decided to rewrite my home-built CD-to-MP3 script suite from its perl incarnation (perl was my previous favorite language for system programming projects). I knocked that project out in only about 2 hours, including a major redesign made easy by Python’s object syntax and built in “pickle” for serialization.

Since then, I’ve usually chosen Python for a variety of system programming tasks. This includes things like making backups, deploying software, configuration file templating, web site scraping, health monitoring, and some bigger data crunching/graphing. Time after time, these quick projects (often at work) turned into something bigger. Eventually they get rolled into a product or adopted by the production operations group as standard kit.

At some point I realized that there was something that I was doing that made this transition from “quick hack” to “standard tool” easy: I always start my projects with logging and optparse in place from day zero. During development, this means that I don’t have to scatter print-style debugging statements throughout the code, I can just use logging.debug() and turn them on and off at will (via command line flags). Once deployed or passed on, it means other people using it can immediately start interacting with my script just like other familiar Unix utilities.

So, here is a variation on what my typical python script starts out with.

#!/usr/bin/python

import logging

def foo():
    """These will only get output if you turn up verbosity."""
    logging.debug("This is debug.")
    logging.info("This is info.")

def bar():
    """These will all be output a default logging levels."""
    logging.warn("Warning!  Things are getting scary.")
    logging.error("Uh-oh, something is wrong.")
    try:
        raise Exception("ZOMG tacos.")
    except:
        logging.exception("Just like error, but with a traceback.")

if '__main__' == __name__:
    # Late import, in case this project becomes a library, never to be run as main again.
    import optparse

    # Populate our options, -h/--help is already there for you.
    optp = optparse.OptionParser()
    optp.add_option('-v', '--verbose', dest='verbose', action='count',
                    help="Increase verbosity (specify multiple times for more)")
    # Parse the arguments (defaults to parsing sys.argv).
    opts, args = optp.parse_args()

    # Here would be a good place to check what came in on the command line and
    # call optp.error("Useful message") to exit if all it not well.

    log_level = logging.WARNING # default
    if opts.verbose == 1:
        log_level = logging.INFO
    elif opts.verbose >= 2:
        log_level = logging.DEBUG

    # Set up basic configuration, out to stderr with a reasonable default format.
    logging.basicConfig(level=log_level)

    # Do some actual work.
    foo()
    bar()

This is obviously a pretty minimal setup, but it achieves what most people need. You get option parsing and checking along with usage information. You get warnings and errors spat out to sys.stderr (leaving sys.stdout for actual program output). You can specify -v to crank up verbosity.

More than once, I’ve had a pager-frazzled sysadmin ask me how the heck to use the new tool. I get to reply, “Just run it with --help, it has full usage instructions built in.” This makes sysadmins happy; really happy. It means they don’t have to remember odd semantics or dig around for the how-to page on the internal wiki at 3:00am when they need to use it. Culturally, there is almost nothing more valuable than a sysadmin who has had success with your code, and nothing worse than one that has had trouble with it. To me, the first step in granting a sysadmin that sort of success is to make your code act like everything else in /usr/bin and that’s why I’ll continue starting my projects in this manner.