0

OK, this is my setup:

FC Switches IBM/Brocade, Switch1 and Switch2, independent fabrics.

Server IBM x3650 M2, 2x QLogic QLE2460, 1 connected to each FC Switch.

Storage IBM DS3524, 2x controllers with 4x FC ports each, but only 2x connected on each.

+-----------------------------------------------------------------------+
|               HBA1             Server              HBA2               |
+-----------------------------------------------------------------------+
                 |                                     |
                 |                                     |
                 |                                     |
+-----------------------------+          +------------------------------+
|            Switch1          |          |            Switch2           |
+-----------------------------+          +------------------------------+
         |                |                  |                 |
         |                |                  |                 |
         |                |                  |                 |
         |                |                  |                 |
         |                |                  |                 |
+-----------------------------------+-----------------------------------+        
| Contr A, port 3 | Contr A, port 4 | Contr B, port 3 | Contr B, port 4 |
+-----------------------------------+-----------------------------------+
|                                 Storage                               |
+-----------------------------------------------------------------------+

My /etc/multipath.conf is from the IBM redbook for the DS3500, except I use a different setting for prio_callout, IBM uses /sbin/mpath_prio_tpc, but according to http://changelogs.ubuntu.com/changelogs/pool/main/m/multipath-tools/multipath-tools_0.4.8-7ubuntu2/changelog, this was renamed to /sbin/mpath_prio_rdac, which I'm using.

devices {
  device {
    #ds3500
    vendor "IBM"
    product "1746   FAStT"
    hardware_handler "1 rdac"
    path_checker rdac
    failback 0
    path_grouping_policy multibus
    prio_callout "/sbin/mpath_prio_rdac /dev/%n"
  }
}

multipaths {
  multipath {
    wwid xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    alias array07
    path_grouping_policy multibus
    path_checker readsector0
    path_selector "round-robin 0"
    failback "5"
    rr_weight priorities
    no_path_retry "5"
  }
}

The output of multipath -ll with controller A as the preferred path:

root@db06:~# multipath -ll
sdg: checker msg is "directio checker reports path is down"
sdh: checker msg is "directio checker reports path is down"
array07 (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) dm-2 IBM     ,1746      FASt
[size=4.9T][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ 5:0:1:0 sdd 8:48  [active][ready]
 \_ 5:0:2:0 sde 8:64  [active][ready]
 \_ 6:0:1:0 sdg 8:96  [failed][faulty]
 \_ 6:0:2:0 sdh 8:112 [failed][faulty]

If I change the preferred path using IBM DS Storage Manager to Controller B, the output swaps accordingly:

root@db06:~# multipath -ll
sdd: checker msg is "directio checker reports path is down"
sde: checker msg is "directio checker reports path is down"
array07 (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) dm-2 IBM     ,1746      FASt
[size=4.9T][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ 5:0:1:0 sdd 8:48  [failed][faulty]
 \_ 5:0:2:0 sde 8:64  [failed][faulty]
 \_ 6:0:1:0 sdg 8:96  [active][ready]
 \_ 6:0:2:0 sdh 8:112 [active][ready]

According to IBM, the inactive path should be "[active][ghost]", not "[failed][faulty]".

Despite this, I don't seem to have any I/O issues, but my syslog is being spammed with this every 5 seconds:

Jun  1 15:30:09 db06 multipathd: sdg: directio checker reports path is down
Jun  1 15:30:09 db06 kernel: [ 2350.282065] sd 6:0:2:0: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  1 15:30:09 db06 kernel: [ 2350.282071] sd 6:0:2:0: [sdh] Sense Key : Illegal Request [current] 
Jun  1 15:30:09 db06 kernel: [ 2350.282076] sd 6:0:2:0: [sdh] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Jun  1 15:30:09 db06 kernel: [ 2350.282083] sd 6:0:2:0: [sdh] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jun  1 15:30:09 db06 kernel: [ 2350.282092] end_request: I/O error, dev sdh, sector 0
Jun  1 15:30:10 db06 multipathd: sdh: directio checker reports path is down
Jun  1 15:30:14 db06 kernel: [ 2355.312270] sd 6:0:1:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  1 15:30:14 db06 kernel: [ 2355.312277] sd 6:0:1:0: [sdg] Sense Key : Illegal Request [current] 
Jun  1 15:30:14 db06 kernel: [ 2355.312282] sd 6:0:1:0: [sdg] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Jun  1 15:30:14 db06 kernel: [ 2355.312290] sd 6:0:1:0: [sdg] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jun  1 15:30:14 db06 kernel: [ 2355.312299] end_request: I/O error, dev sdg, sector 0

Does anyone know how I can get the inactive path to show "[active][ghost]" instead of "[failed][faulty]"? I assume that once I can get that right then the spam in my syslog will end as well.

One final thing worth mentioning is that the IBM redbook doc targets SLES 11 so I'm assuming there's something a little different under Ubuntu that I just haven't figured out yet.

Update: As suggested by Mitch, I've tried removing /etc/multipath.conf, and now the output of multipath -ll looks like this:

root@db06:~# multipath -ll
sdg: checker msg is "directio checker reports path is down"
sdh: checker msg is "directio checker reports path is down"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdm-1 IBM     ,1746      FASt
[size=4.9T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
 \_ 5:0:2:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 5:0:1:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 6:0:1:0 sdg 8:96  [failed][faulty]
\_ round-robin 0 [prio=0][enabled]
 \_ 6:0:2:0 sdh 8:112 [failed][faulty]

So its more or less the same, with the same message in the syslog every 5 minutes as before, but the grouping has changed.

2 Answers 2

1

I really do not know much about the IBM DS3524. But if it is anything like an EMC Clariion CX-4, then:

Try moving the multipath.conf away from the etc folder and restart multipath-tools.

1
  • Upvoted because this is a good suggestion, even if it didn't work in this specific instance. The default config may work depending on specific distro/release/storage hardware. Jun 4, 2012 at 10:02
1

I finally found a config that works here: http://pig.made-it.com/multipath.html. Summarising here in case anyone else has this in future.

device {
    vendor "IBM"
    product "1745|1746"
    path_grouping_policy group_by_prio
#   getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
    path_selector "round-robin 0"
    path_checker rdac
    features "2 pg_init_retries 50"
    hardware_handler "1 rdac"
    prio_callout "/sbin/mpath_prio_rdac /dev/%n"
    failback immediate
    rr_weight uniform
    no_path_retry 15
    rr_min_io 1000
}

I commented getuid_callout because Ubuntu places the scsi_id program in /lib/udev, not /sbin, and Ubuntu's default value for getuid_callout is /lib/udev/scsi_id -g -u -s, so I haven't put it in my config explicitly.

No specific settings are needed in a multipath { ... } block, but you can obviously create friendly alias names if needed using:

multipaths {
  multipath {
    wwid xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    alias array07
  }
}

multipath -ll output now looks like this:

root@db06:~# multipath -ll                                                                                                                                                                                                                                             
array07 (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) dm-1 IBM     ,1746      FASt
[size=4.9T][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][enabled]
 \_ 5:0:1:0 sdd 8:48  [active][ghost]
 \_ 5:0:2:0 sde 8:64  [active][ghost]
\_ round-robin 0 [prio=6][active]
 \_ 6:0:1:0 sdg 8:96  [active][ready]
 \_ 6:0:2:0 sdh 8:112 [active][ready]

I'm not getting spam in /var/log/syslog anymore and failover/failback are working perfectly.

1
  • In case someone gets here -- after 9 years. Just got a used DS3512, and it seems that DS series doesn't support multibus configuration, or at least I haven't managed to get it work. So path_grouping_policy should be set group_by_prio or the others, otherwise the syslog will be spammed. Jun 15, 2021 at 10:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .