16

I'm writing a Python + GObject app that needs to read a non-trivial amount of data from disk upon start. The data is read synchronously and it takes about 10 seconds to finish the read operation, during which time the loading of the UI is delayed.

I'd like to run the task asynchronously, and get a notification when it's ready, without blocking the UI, more or less like:

def take_ages():
    read_a_huge_file_from_disk()

def on_finished_long_task():
    print "Finished!"

run_long_task(task=take_ages, callback=on_finished_long_task)
load_the_UI_without_blocking_on_long_task()

I've used GTask in the past for this sort of thing, but I'm concerned that its code hasn't been touched in 3 years, let alone been ported to GObject Introspection. Most importantly, it's no longer available in Ubuntu 12.04. So I'm looking for an easy way to run tasks asynchronously, either in a standard Python way or in a GObject/GTK+ standard way.

Edit: here's some code with an example of what I'm trying to do. I've tried python-defer as suggested in the comments, but I could not manage to run the long task asynchronously and let the UI load without having to wait for it to finish. Browse the test code.

Is there an easy and widely used way of running asynchronous tasks and get notified when they're finished?

3
  • It's not a pretty example, but I'm pretty sure this is what you're looking for: raw.github.com/gist/1132418/… May 27, 2012 at 12:52
  • Cool, I think your async_call function might be what I need. Would you mind expanding on it a bit and adding an answer, so that I can accept it and credit you after I test it? Thanks! May 27, 2012 at 14:14
  • 1
    Great question, very useful! ;-) May 27, 2012 at 16:46

5 Answers 5

15

Your problem is a very common one, therefore there are tons of solutions (sheds, queues with multiprocessing or threading, worker pools, ...)

Since it is so common, there is also a python build-in solution (in 3.2, but backported here: http://pypi.python.org/pypi/futures) called concurrent.futures. 'Futures' are available in many languages, therefore python calls them the same. Here are the typical calls (and here is your full example, however, the db part is replaced by sleep, see below why).

from concurrent import futures
executor = futures.ProcessPoolExecutor(max_workers=1)
#executor = futures.ThreadPoolExecutor(max_workers=1)
future = executor.submit(slow_load)
future.add_done_callback(self.on_complete)

Now to your problem, which is much more complicated than your simple example suggests. In general you have threads or processes to solve this, but here is why your example is so complicated:

  1. Most Python implementations have a GIL, which makes threads not fully utilize multicores. So: do not use threads with python!
  2. The objects you want to return in slow_load from the DB are not pickelable, which means that they can not simply be passed between processes. So: no multiprocessing with softwarecenter results!
  3. The library you call (softwarecenter.db) is not threadsafe (seems to include gtk or similar), therefore calling these methods in a thread results in strange behaviour (in my test, everything from 'it works' over 'core dump' to simple quitting without results). So: no threads with softwarecenter.
  4. Every asynchronous callback in gtk should not do anything except sheduling a callback which will be called in the glib mainloop. So: no print, no gtk state changes, except adding a callback!
  5. Gtk and alike does not work with threads out of the box. You need to do threads_init, and if you call a gtk or alike method, you have to protect that method (in earlier versions this was gtk.gdk.threads_enter(), gtk.gdk.threads_leave(). see for example gstreamer: http://pygstdocs.berlios.de/pygst-tutorial/playbin.html).

I can give you the following suggestion:

  1. Rewrite your slow_load to return pickelable results and use futures with processes.
  2. Switch from softwarecenter to python-apt or similar (you probably don't like that). But since your employed by Canonical, you could ask the softwarecenter developers directly to add documention to their software (e.g. stating that it is not thread safe) and even better, making softwarecenter threadsafe.

As a note: the solutions given by the others (Gio.io_scheduler_push_job, async_call) do work with time.sleep but not with softwarecenter.db. This is, because it all boils down to threads or processes and threads to not work with gtk and softwarecenter.

2
  • Thanks! I'm going to accept your answer as it points me in very much detail to why it is not doable. Unfortunately, I cannot use software that is not packaged for Ubuntu 12.04 in my app (it is for Quantal, though launchpad.net/ubuntu/+source/python-concurrent.futures), so I guess I'm stuck with not being able to run my task asynchronously. Regarding the note to talk to the Software Center developers, I'm in the same position as any volunteer to contribute changes to the code and documentation or to talk to them :-) May 28, 2012 at 12:31
  • GIL is released during IO so it is perfectly fine to use threads. Though it is not necessary if async IO is used.
    – jfs
    Sep 10, 2012 at 6:58
10

Here's another option using GIO's I/O Scheduler (I've never used it before from Python, but the example below seems to run fine).

from gi.repository import GLib, Gio, GObject
import time

def slow_stuff(job, cancellable, user_data):
    print "Slow!"
    for i in xrange(5):
        print "doing slow stuff..."
        time.sleep(0.5)
    print "finished doing slow stuff!"
    return False # job completed

def main():
    GObject.threads_init()
    print "Starting..."
    Gio.io_scheduler_push_job(slow_stuff, None, GLib.PRIORITY_DEFAULT, None)
    print "It's running async..."
    GLib.idle_add(ui_stuff)
    GLib.MainLoop().run()

def ui_stuff():
    print "This is the UI doing stuff..."
    time.sleep(1)
    return True

if __name__ == '__main__':
    main()
3
  • See also GIO.io_scheduler_job_send_to_mainloop(), if you want to run something in the main thread once slow_stuff finishes. May 27, 2012 at 16:27
  • Thanks Sigfried for the answer and the example. Unfortunately, it seems that with my current task I have no chance to use the Gio API to make it run asynchronously. May 28, 2012 at 12:33
  • This was really useful, but as far as I can tell Gio.io_scheduler_job_send_to_mainloop doesn't exist in Python :(
    – sil
    Jul 18, 2012 at 22:20
2

You can also use GLib.idle_add(callback) to call the long running task once the GLib Mainloop finishes all it's higher priority events (which I believe includes building the UI).

4
  • Thanks Mike. Yes, that would definitely help with starting the task when the UI is ready. But on the other hand, I understand that when callback is called, that would be done synchronously, thus blocking the UI, right? May 27, 2012 at 14:08
  • The idle_add doesn't quite work like that. Making blocking calls in an idle_add is still a bad thing to do, and it will prevent updates to the UI from happening. And even asynchronous API can still be blocking, where the only way to avoid blocking the UI and other tasks, is to do it in a background thread.
    – dobey
    May 27, 2012 at 14:49
  • Ideally you'd split your slow task into chunks, so you can run a bit of it in an idle callback, return (and let other stuff like UI callbacks run), continue doing some more work once your callback is called again, and so on. May 27, 2012 at 16:22
  • A gotcha with idle_add is that the return value of the callback matters. If it's true, it will be called again.
    – Flimm
    Apr 18, 2013 at 19:20
2

Use the introspected Gio API to read a file, with its asynchronous methods, and when making the initial call, do it as a timeout with GLib.timeout_add_seconds(3, call_the_gio_stuff) where call_the_gio_stuff is a function which returns False.

The timeout here is necessary to add (a different number of seconds may be required, though), because while the Gio async calls are asynchronous, they are not non-blocking, meaning that the heavy disk activity of reading a large file, or large number of files, can result in blocked UI, as the UI and I/O are still in the same (main) thread.

If you want to write your own functions to be async, and integrate with the main loop, using Python's file I/O APIs, you'll have to write the code as a GObject, or to pass callbacks around, or use python-defer to help you do it. But it's best to use Gio here, as it can bring you a lot of nice features, especially if you're doing file open/save stuff in the UX.

7
  • Thanks @dobey. I'm not actually reading a file from disk directly, I should have probably made that clearer in the original post. The long-running task that I'm running is reading the Software Center database as per the answer to askubuntu.com/questions/139032/…, so I'm not sure I can use the Gio API. What I was wondering is whether there is a way to run any generic long-running task asynchronously in the same way GTask used to do. May 27, 2012 at 13:57
  • I don't know what GTask is exactly, but if you mean gtask.sourceforge.net then I don't think you should be using that. If it's something else, then I don't know what it is. But it sounds like you will have to take the second route I mentioned, and implement some asynchronous API to wrap that code, or just do it all in a thread.
    – dobey
    May 27, 2012 at 14:46
  • There's a link to it in the question. GTask is (was): chergert.github.com/gtask May 27, 2012 at 15:31
  • 1
    Ah, that looks very similar to the API provided by python-defer (and twisted's deferred API). Perhaps you should look at using python-defer?
    – dobey
    May 27, 2012 at 20:31
  • 1
    You still need to delay that being called, until after the main priority events have happened, by using GLib.idle_add() for example. Like this: pastebin.ubuntu.com/1011660
    – dobey
    May 28, 2012 at 17:44
1

I think it bears noting that this is a convoluted way to do what @mhall suggested.

Essentially, you've got a run this then run that function of async_call.

If you want to see how it works, you can play with the sleep timer and keep clicking the button. It's essentially the same as @mhall's answer except that there's example code.

Based on this which is not my work.

import threading
import time
from gi.repository import Gtk, GObject



# calls f on another thread
def async_call(f, on_done):
    if not on_done:
        on_done = lambda r, e: None

    def do_call():
        result = None
        error = None

        try:
            result = f()
        except Exception, err:
            error = err

        GObject.idle_add(lambda: on_done(result, error))
    thread = threading.Thread(target = do_call)
    thread.start()

class SlowLoad(Gtk.Window):

    def __init__(self):
        Gtk.Window.__init__(self, title="Hello World")
        GObject.threads_init()        

        self.connect("delete-event", Gtk.main_quit)

        self.button = Gtk.Button(label="Click Here")
        self.button.connect("clicked", self.on_button_clicked)
        self.add(self.button)

        self.file_contents = 'Slow load pending'

        async_call(self.slow_load, self.slow_complete)

    def on_button_clicked(self, widget):
        print self.file_contents

    def slow_complete(self, results, errors):
        '''
        '''
        self.file_contents = results
        self.button.set_label(self.file_contents)
        self.button.show_all()

    def slow_load(self):
        '''
        '''
        time.sleep(5)
        self.file_contents = "Slow load in progress..."
        time.sleep(5)
        return 'Slow load complete'


    
if __name__ == '__main__':
    win = SlowLoad()
    win.show_all()
    #time.sleep(10)
    Gtk.main()
    

Additional note, you have to let the other thread finish before it will terminate properly or check for a file.lock in your child thread.

Edit to address comment:
Initially I forgot GObject.threads_init(). Evidently when the button fired, it initialized threading for me. This masked the mistake for me.

Generally the flow is create the window in memory, immediately launch the other thread, when the thread completes update the button. I added an additional sleep before I even called Gtk.main to verify that the full update COULD run before the window was even drawn. I also commented it out to verify that the thread launch doesn't impede window drawing at all.

5
  • 1
    Thanks. I'm not sure I can follow it. For one, I would have expected slow_load to be executed shortly after the UI starts, but it never seems to be called, unless the button is clicked, which confuses me a bit, as I thought the purpose of the button was just to provide visual indication of the state of the task. May 27, 2012 at 15:57
  • Sorry, I missed one line. That did it. I forgot to tell GObject to get ready for threads. May 27, 2012 at 16:14
  • But you're calling into the main loop from a thread, which can cause problems, though they may not be easily exposed in your trivial example which doesn't do any real work.
    – dobey
    May 27, 2012 at 16:53
  • Valid point, but I didn't think a trivial example merited sending the notification via DBus(which I think a non-trivial app should be doing) May 27, 2012 at 16:59
  • Hm, running async_call in this example works for me, but it brings mayhem when I port it to my app and I add the real slow_load function I've got. May 27, 2012 at 23:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .