Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

How do you send the output of an upstart script to a terminal so to find tracebacks in python code? It's taking me for ever to do things without trace-backs that used to take just a second. I'm having to place several file write calls to track down errors. What took second to find before with a traceback is turning in to several minutes minutes. This is miserable. This has been going on for a few weeks now and I'm sick of it. would some speak up on this please. I feel like I'm using assembly without a debugger again.

share|improve this question
add comment

3 Answers

When I write a python daemon I catch all the exceptions and throw then to the log file. I not only use for debug, but in production too. I have a a small script that I run every morning that looks for something upsetting in the logs.

It also helps in keeping the daemon running, of course.

Some sample code (I remove the not interesting parts):

import logging

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s',
                    filename=LOG_FILE,
                    filemode='w')
    logging.info("Sincrod inicializado")
    if not DEBUG:
        daemonize()
    while True:
        try:
            actua()
        except:
            logging.error(sys.exc_info())
        if (datetime.datetime.now().hour > NOITE_EMPEZA\
         and datetime.datetime.now().hour < NOITE_REMATA):
            time.sleep(INTERVALO_NOITE)
        else:
            time.sleep(INTERVALO_DIA)

Where actua() is the real daemon (it writes to log too). Note that I also have a DEBUG variable in a settings file, when it's True, I don't fork the daemon so it's executes on the console.

Daemons

Daemons are the unix equivalent to windows services. They are processes that run in the background independent from other processes. That means that their father is usually init, and that they are detached from any tty. As they are independent, there is no predefined place to put their output.

There are lots of python libraries and snippets to make a daemon, in the above example I use my own function, that combines some ideas from Steinar Knutsens and Jeff Kunces versions. It's as simple as possible, note that I fork twice.

def daemonize():
    """Forks this process creating a daemon and killing the original one"""
    if (not os.fork()):
        # get our own session and fixup std[in,out,err]
        os.setsid()
        sys.stdin.close()
        sys.stdout = NullDevice()
        sys.stderr = NullDevice()
        if (not os.fork()):
            # hang around till adopted by init
            ppid = os.getppid()
            while (ppid != 1):
                time.sleep(0.5)
                ppid = os.getppid()
        else:
            # time for child to die
            os._exit(0)
    else:
        # wait for child to die and then bail
        os.wait()
        sys.exit()
share|improve this answer
    
Well, OK. since you're already logging to syslog, then just filter out your daemon messages and dump them to the console. I don't see why this is specific to upstart? SysV init would have the same problem. –  ppetraki Mar 22 '12 at 15:44
    
You are right, it's not specific to upstart, to tell the truth most of my servers run 8.04, no upstart. But it's valid for upstart too. OP was asking how to debug python scripts with upstart, not for a method that only works with upstart. I'm not logging to syslog but to a specific file, and the 'trick' here is catching all exceptions and dumping the stack trace to that file. –  Javier Rivera Mar 22 '12 at 19:01
    
well, that's just managing stdout based on the context right? I know plenty of unix daemons that have equivalent logging verbosity regardless of whether it's attached to a tty or functioning as a daemon. If this were Ruby, I would over-ride or decorate the base class method that exceptions use to output. I'm sure something similar can be done in Python. You might be better served asking this question on stack exchange proper. This is more of a basic unix daemon coding/design problem, and as you stated, it has nothing specific to do with init scripts. –  ppetraki Mar 22 '12 at 19:51
    
I'm still getting familiar with the lingo. I assume by daemon, you mean a specific script that runs in the background. In your code, I just put my script in place of actua() to get the callbacks for that script call? Is there anyway to channel it to a console instead of a file? –  bambuntu Mar 22 '12 at 22:01
1  
daemons in the stand alone sense are usually detached from the tty they were started on, have closed their original file handles to stdin, stdout, and stdin, and are a child of init. So if you want to print exceptions to someplace specific, find out how they are outputed, and direct them from there. linfo.org/daemon.html. Again, this doesn't have anything to do with upstart, or even init for that matter. Get your program functioning correctly in true daemon mode and then move it upstart. –  ppetraki Mar 23 '12 at 21:00
show 9 more comments

If you use Upstart 1.4 or newer, put console log into your Upstart job and all the output to stdout/stderr will end up to /var/log/upstart/<job>.log. Then you can do tail -f /var/log/upstart/<job>.log & to have the output appear in terminal.

share|improve this answer
    
Kind of late to the party, but this answer saved me :) Also looks like this works for me without any special setting in the upstart conf file. For my part this should be the accepted answer. –  rslite Aug 14 '13 at 18:10
add comment

There's a whole section on debugging techniques in the Upstart Cookbook. The easiest thing you could do is add --debug to your kernel arguments, which will increase upstart's verbosity and dump everything to syslog. Yes, debugging is complex, it's a reflection of the net complexity required to create a parallelized init system. I'm sure there's room for improvement.

share|improve this answer
2  
the cook book doesn't properly explain a debugging environment to a new comer. I've seen similar explanations before. There are either lacking or make guru assumptions. It's very frustrating to people who want to add to the community and just starting out. I've never ran into a programming environment that didn't provide the line of code where the error occurs except in assembly, where you are reinventing the wheel, so that can be forgiven. –  bambuntu Mar 23 '12 at 19:09
    
Well, what would you suggest then? It's an open document. If you've got a debugging technique that's leaps and bounds above what's presented there then please add it. The OP's problems are more a result of not understanding how to manage basic unix paradigms inside his additional runtime of choice vs the context it's being deployed in. Just because you're using python or [insert fancy runtime language here] doesn't mean you get to ignore the fundamental runtime, UNIX. –  ppetraki Mar 23 '12 at 20:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.