Last month I wrote about a problem I saw with scsi_id and UDEV in OL5.8. As it screwed up all my UDEV rules is was a pretty important issue for me. It turned out this was due to a mainline security fix (CVE-2011-4127) affecting the latest kernels of both RHEL/OL5 and RHEL/OL6. The comments on the previous post show a couple of workarounds.
Over the weekend I started to update a couple of articles that mentioned UDEV rules (here and here) and noticed the problem had dissapeared. I updated two VMs (OL5.8 and OL6.2) with the latest changes, including the UEK updates and ran the tests again and here’s what I got.
# cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.8 (Tikanga) # uname -r 2.6.39-100.6.1.el5uek # scsi_id -g -u -s /block/sda/sda1 SATA_VBOX_HARDDISK_VB535d493d-7a44eb0f_ # # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago) # uname -r 2.6.39-100.6.1.el6uek.x86_64 # /sbin/scsi_id -g -u /dev/sda1 1ATA_VBOX_HARDDISK_VB2b5dc561-4ae6e154 #
So it looked like normal service had been resumed. 🙂 Unfortunately, the MOS Note 1438604.1 associated with this issue is still not public, so I couldn’t tell if this was a unilateral change in UEK, or part of a mainline fix for the previous change.
To check I fired up a CentOS 6.2 VM with the latest kernel updates and switched an Oracle Linux VM to the latest RHEL compatible kernel and did the test on both. As you can see, they both still don’t report the scsi_id for partitions.
# cat /etc/redhat-release CentOS release 6.2 (Final) # uname -r 2.6.32-220.13.1.el6.x86_64 # /sbin/scsi_id -g -u /dev/sda1 # # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago) # uname -r 2.6.32-220.13.1.el6.x86_64 # /sbin/scsi_id -g -u /dev/sda1 #
It could be the associated fix has not worked through the mainline to RHEL and CentOS yet. I’ll do a bit of digging around to see what is going on here.
Cheers
Tim…
http://t.co/IGPWshXK (scsi_id and UDEV issues (update)… | The ORACLE-BASE Blog)
I wrote this rule to handle LUNs which have multiple partitions, not all of which are to be used for Oracle ASM.
This example, first 2 partitions will be used for Linux LVM, 3 for DATA, and 4 for FRA.
The rule handles LUNS with single or multiple partions and handles setting different permissions on different partitions on the same LUN.
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL==”sd?[0-2]”, BUS==”scsi”, PROGRAM==”/usr/bin/udevinfo -q name -p %p”, RESULT==”%k”, PROGRAM==”scsi_id -g -u -d /dev/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB25d5dd60-604b68ef_”, NAME=”lvm-disk1p%n”
KERNEL==”sd?[3-4]”, BUS==”scsi”, PROGRAM==”/usr/bin/udevinfo -q name -p %p”, RESULT==”%k”, PROGRAM==”scsi_id -g -u -d /dev/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB25d5dd60-604b68ef_”, NAME=”asm-disk1p%n”, OWNER=”gstanden”, GROUP=”gstanden”, MODE=”0660″
KERNEL==”sd?[0-9]”, BUS==”scsi”, PROGRAM==”/usr/bin/udevinfo -q name -p %p”, RESULT==”%k”, PROGRAM==”scsi_id -g -u -d /dev/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB12909768-447465d2_”, NAME=”asm-disk2p%n”, OWNER=”gstanden”, GROUP=”gstanden”, MODE=”0660″
[root@building-db5a ~]# ls -lrt /dev/*-disk*
brw-rw—- 1 gstanden gstanden 8, 33 Oct 20 00:16 /dev/asm-disk2p1
brw-r—– 1 root disk 8, 17 Oct 20 00:16 /dev/lvm-disk1p1
brw-rw—- 1 gstanden gstanden 8, 19 Oct 20 00:16 /dev/asm-disk1p3
brw-r—– 1 root disk 8, 18 Oct 20 00:16 /dev/lvm-disk1p2
brw-rw—- 1 gstanden gstanden 8, 20 Oct 20 00:16 /dev/asm-disk1p4
Regards
Gil
I would like to share another device persistence scenario that recently had to be solved here. RHEL 6.3 using Linux multipath for the Oracle ASM storage. In RHEL6 you can no longer set the UID and GID in the /etc/multipath.conf file. Here, the recommended method was used: UDEV rules in the 12-dm-permissions.rules (an uncustomized copy should be in your distro at /usr/share/doc/device-mapper-1.02.74/ (of course the 1.02.74 may not be your version).
Here are the rules used to set permissions and persistence on partitions. The method involves using both /etc/multipath.conf and /etc/udev/rules.d/12-dm-permissions.rules file to accomplish both the persistent naming and the proper permissions and ownership.
/etc/multipath.conf:
devices {
device {
vendor “HP”
product “*”
prio “alua”
path_grouping_policy group_by_prio
features “1 queue_if_no_path”
failback immediate
hardware_handler “1 alua”
}
}
multipaths {
multipath {
wwid 3600c0ff00011cbc145cbb75001000000
alias ASM_RAID_5
}
multipath {
wwid 3600c0ff0001384a681ccb75001000000
alias ASM_RAID10
}
}
/etc/udev/rules.d/12-dm-permissions.rules
(only the multipath section shown/used):
# MULTIPATH DEVICES
#
# Set permissions for specific multipath devices
# ENV{DM_UUID}==”mpath-3600c0ff0001384a681ccb75001000000″, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”660″
ENV{DM_UUID}==”mpath-3600c0ff00011cbc145cbb75001000000″, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”660″
# Set permissions for all multipath devices
# ENV{DM_UUID}==”mpath-?*”, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”660″
# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
ENV{DM_UUID}==”part[1-2]-mpath-3600c0ff0001384a681ccb75001000000″, OWNER:=”grid”, GROUP:=”asmadmin”, MODE:=”660″
In RHEL6 /dev/mapper names are now soft links which point to /dev/dm* devices. So the /dev/mapper names are given to ASM via asm_diskstring=/dev/mapper/ASM*
[root@lw-6027test rules.d]# ls -lrt /dev/mapper
total 0
crw-rw—- 1 root root 10, 58 Dec 14 14:59 control
lrwxrwxrwx 1 root root 7 Dec 14 14:59 vg1-swap -> ../dm-0
lrwxrwxrwx 1 root root 7 Dec 14 14:59 vg1-root -> ../dm-1
lrwxrwxrwx 1 root root 7 Dec 14 14:59 ASM_RAID10 -> ../dm-2
lrwxrwxrwx 1 root root 7 Dec 14 14:59 ASM_RAID_5 -> ../dm-3
lrwxrwxrwx 1 root root 7 Dec 14 14:59 ASM_RAID10p1 -> ../dm-4
lrwxrwxrwx 1 root root 7 Dec 14 14:59 ASM_RAID10p2 -> ../dm-5
and the permissions and ownership are set on the actual dm* devices:
[root@lw-6027test rules.d]# ls -lrt /dev/dm*
brw-rw—- 1 root disk 253, 1 Dec 14 14:59 /dev/dm-1
brw-rw—- 1 root disk 253, 0 Dec 14 14:59 /dev/dm-0
brw-rw—- 1 grid asmadmin 253, 5 Dec 14 14:59 /dev/dm-5
brw-rw—- 1 grid asmadmin 253, 3 Dec 21 14:44 /dev/dm-3
brw-rw—- 1 root disk 253, 2 Dec 21 14:44 /dev/dm-2
brw-rw—- 1 grid asmadmin 253, 4 Dec 21 16:11 /dev/dm-4
HTH, Gil
Here is another handy rule for providing persistent naming for ASM disk partitions on RHEL6 when it’s a standalone server with internal disk (non-multipath, non-SAN). This rule is nice because it gives you exactly the same setup as you get when you use /etc/multipath.conf + /etc/udev/rules.d/12-dm-permissions.rules for the RAC case (i.e. multipath disks presented to RAC from a shared SAN). This method produces the exact same symlinks in /dev/mapper for the standalone internal case as you get when you use device-mapper-multipath for the RAC/SAN case with 12-dm-permissions.rules.
Credit for this rule (at least where I found it) goes to em31amit at this link: http://www.linuxquestions.org/questions/linux-server-73/rhel6-udev-naming-override-941229/
All I did was modify it a bit for RHEL6 . It works especially nice for RHEL6 because it puts the symlinks in exactly the same location and form as the ones that device-mapper-multipath would put in /dev/mapper for multipathed SAN disks, which is nice because now your standalone and RAC boxes will have storage setup exactly the same. I modified the PROGRAM section to use “-d” switch on scsi_id instead of “-s” switch and use “/dev/$parent” instead of “%p” and I added the OWNER, GROUP and MODE settings. It works great for RHEL6. Here’s at example of the rule:
KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”3600508b1001c1194320f841a5c1474ef”, SYMLINK+=”mapper/ASM_DATAp%n”, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”3600508b1001c948206b524f68ea2049f”, SYMLINK+=”mapper/ASM_DATA1p%n”,OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
And it gives you this in /dev/mapper :
lrwxrwxrwx. 1 root root 7 Aug 20 17:02 ASM_DATAp1 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Aug 20 17:02 ASM_DATA1p1 -> ../sdd1
lrwxrwxrwx. 1 root root 7 Aug 20 17:02 ASM_DATAp2 -> ../sdc2
lrwxrwxrwx. 1 root root 7 Aug 20 17:02 ASM_DATA1p2 -> ../sdd2
and here’s what you see in /dev :
brw-rw—-. 1 grid asmadmin 8, 33 Aug 20 17:02 /dev/sdc1
brw-rw—-. 1 grid asmadmin 8, 49 Aug 20 17:02 /dev/sdd1
brw-rw—-. 1 grid asmadmin 8, 34 Aug 20 17:02 /dev/sdc2
brw-rw—-. 1 grid asmadmin 8, 50 Aug 20 17:02 /dev/sdd2
Nice! Thanks em31amit!