Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

Scenario: Mongo 2.2 running on Ubuntu 10.10 on Amazon EC2. The Mongo DB files are on an Amazon EBS volume mounted as follows:

/dev/sdg        /data/db        ext4    noatime,noexec,nodiratime       0       0

Every week I dump about 30GB into the Mongo database and most of the time everything works correctly. Sometimes, however, it will hang partway through and never recover. Rebooting allows me to restart the data dump, but does not prevent the hang from happening again later (IOW it fixes a hang but not all hangs).

When Mongo is hung, iostat will show something like this (note the %util on xvdg)

10/04/12 13:54:05
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdb              0.00     0.50    0.00    1.50     0.00    16.00    10.67     0.00    0.00   0.00   0.00
xvdg              0.00     0.00    0.00    0.00     0.00     0.00     0.00   138.00    0.00   0.00 100.00

There is an interesting and probably complicated relationship between Amazon service, Mongo's file access, the Ubuntu platform, and of course my imperfect knowledge. I'm exploring the Amazon and Mongo angles in different forums and I'd like to ask about Ubuntu (and me) here.

I see the 100% util, and I know that I've stopped adding to the Mongo work to be done (and this machine does nothing else), so I assume that this is just a fixed amount of data that needs to be written to the drive but is hung up somehow.

Can I find out exactly what is going on with the filesystem? Maybe I'm doing something wrong here?

Questions & advice welcomed, thanks!

EDIT: so now I've heard about dmesg and here are two entries from it (out of 4):

INFO: task jbd2/sdg-8:508 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdg-8    D ffff880003dec980     0   508      2 0x00000000
 ffff8801d5f7fd10 0000000000000246 0000000000000000 0000000000015980
 ffff8801d5f7ffd8 0000000000015980 ffff8801d5f7ffd8 ffff8801d5fc5b80
 0000000000015980 0000000000015980 ffff8801d5f7ffd8 0000000000015980
Call Trace:
 [<ffffffff81227f86>] jbd2_journal_commit_transaction+0x1b6/0x1350
 [<ffffffff81008696>] ? __switch_to+0x166/0x320
 [<ffffffff81006b3d>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff810072bf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8107f080>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff815a404e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
 [<ffffffff8122d87d>] kjournald2+0xbd/0x220
 [<ffffffff8107f080>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8122d7c0>] ? kjournald2+0x0/0x220
 [<ffffffff8107eb26>] kthread+0x96/0xa0
 [<ffffffff8100aee4>] kernel_thread_helper+0x4/0x10
 [<ffffffff8100a313>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff815a45dd>] ? retint_restore_args+0x5/0x6
 [<f.fffffff8100aee0>] ? kernel_thread_helper+0x0/0x10

.

INFO.: task mongod:673 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mongod        D ffff880003dec980     0   673      1 0x00000000
 ffff8801d5e73be8 0000000000000286 0000000000000000 0000000000015980
 ffff8801d5e73fd8 0000000000015980 ffff8801d5e73fd8 ffff8801d594db80
 0000000000015980 0000000000015980 ffff8801d5e73fd8 0000000000015980
Call Trace:
 [<ffffffff812263b1>] start_this_handle+0x251/0x4b0
 [<ffffffff810072d2>] ? check_events+0x12/0x20
 [<ffffffff8107f080>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81006b3d>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff812267e5>] jbd2_journal_start+0xb5/0x100
 [<ffffffff81202468>] ext4_journal_start_sb+0xf8/0x130
 [<ffffffff811ecba5>] ? ext4_meta_trans_blocks+0x75/0xf0
 [<ffffffff811f00ff>] ext4_da_writepages+0x25f/0x640
 [<ffffffff810838df>] ? hrtimer_try_to_cancel+0x3f/0xd0
 [<ffffffff8110a791>] do_writepages+0x21/0x40
 [<ffffffff8110129b>] __filemap_fdatawrite_range+0x5b/0x60
 [<ffffffff811012fa>] filemap_write_and_wait_range+0x5a/0x80
 [<ffffffff8117a54a>] vfs_fsync_range+0x5a/0xa0
 [<ffffffff8117a5fc>] vfs_fsync+0x1c/0x20
 [<ffffffff81127cb4>] sys_msync+0x144/0x1d0
 [<ffffffff8100a0f2>] system_call_fastpath+0x16/0x1b

Is this a rabbit hole?

EDIT2: Well it turns out that when a disk is locked in the manner described above, the machine gets less and less usable until you're forced to reboot. And in the process of doing that, I realized that it may make more sense to try the latest kernel and a newer OS.

So I'm giving up this configuration and moving from Ubuntu 10.10 (kernel 2.6.35) to Amazon Linux (kernel 3.2.something).

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.