Cloud Control 12.1.0.5 : Some minor issues (fixed)

em-12cSince the upgrade to Cloud Control 12.1.0.5, we’ve been having a couple of issues, mostly around EMCLI.

Some of our databases use Service Guard, so you don’t know which node they are running on. Rather than having an agent per package, we have one on each node. One of my colleagues wrote a little script to check which node the instance is running on, and relocate it if it has moved. This is done using EMCLI and was working fine before the move to 12.1.0.5. Since the upgrade it’s been rather erratic. It would work for a while, then fail. After watching for a while I noticed a couple of things.

EMCLI calls to the OMS that sent SQL were intermittently killing in the Agent on the Cloud Control server itself. It seemed to cause Out Of Memory errors. The heap size for the agent on the CC server was set to something like 1860M. Changing it to 2048M seemed to fix that issue. The setting is in the “agent_inst/sysman/config/emd.properties” file. We now have this.

agentJavaDefines=-Xmx2048M -XX:MaxPermSize=128M

Note. The agents on the monitored servers have a tiny heap size. Nothing like this bad-boy. 🙂

After running OK for a few days, we started to get the following type of errors from EMCLI.

$ emcli sync
Error: Session expired. Run emcli login to establish a session.

We fixed this by running the “setup” command again.

emcli setup -username=sysman -password="MyPassword" -url="https://myserver.example.com:7799/em"

Since then, EMCLI has seemed to behave itself.

Note. As part of the upgrade, we downloaded the latest EMCLI jar file from the server and did the setup, so this part was working fine for a while. Not sure why it started to screw up…

Anyway, all seems fine now. I’m guessing if we weren’t using EMCLI, we would never have had a problem in the first place.

Cheers

Tim…

Oracle VM [not] running inside VirtualBox…

I know what you are thinking, sounds like a dumb idea right? Well yes, it is pretty stupid, but it’s nice to do it for testing Oracle VM when you don’t want to dedicate a whole server to it. I’ve done this before with no worries. The screen grabs for this article were taken from an installation of Oracle VM inside VirtualBox.

The reason for this post is it doesn’t seem to work for me anymore. Since I last did this successfully I’ve upgraded my host OS (to Fedora 14) and VirtualBox several times (now at 4.0.6). So before I start meddling with downgrading my OS, I would like to know if anyone else has any issues of this type of installation?

I’ve already tried installing old versions of VirtualBox (back to 3.2.8) and that doesn’t fix the issue, so it looks to me like it is the OS that is the issue (sigh). I get the same issue on two servers running Fedora14 as the host OS.

The problem: The installation of Oracle VM inside a VirtualBox VM works fine, but during the post install reboot the VM hangs and I get the following in my “/var/log/messages” file on the host.

May 10 10:42:26 maggie kernel: [  308.782339] BUG: unable to handle kernel paging request at 0000000000002dc4
May 10 10:42:26 maggie kernel: [  308.782345] IP: [] g_abExecMemory+0x1cee8/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782359] PGD 21600b067 PUD 216319067 PMD 0
May 10 10:42:26 maggie kernel: [  308.782363] Oops: 0000 [#1] SMP
May 10 10:42:26 maggie kernel: [  308.782366] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
May 10 10:42:26 maggie kernel: [  308.782370] CPU 3
May 10 10:42:26 maggie kernel: [  308.782371] Modules linked in: vboxnetadp vboxnetflt vboxdrv fuse cifs nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc ipv6 nls_utf8 udf uinput snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep iTCO_wdt ppdev parport_pc parport r8169 iTCO_vendor_support sky2 mii asus_atk0110 i2c_i801 snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc microcode ata_generic pata_acpi firewire_ohci firewire_core crc_itu_t pata_jmicron nouveau usb_storage ttm drm_kms_helper drm i2c_algo_bit video output i2c_core [last unloaded: scsi_wait_scan]
May 10 10:42:26 maggie kernel: [  308.782377]
May 10 10:42:26 maggie kernel: [  308.782377] Pid: 5584, comm: VirtualBox Not tainted 2.6.35.13-91.fc14.x86_64 #1 P5K-VM/P5K-VM
May 10 10:42:26 maggie kernel: [  308.782377] RIP: 0010:[]  [] g_abExecMemory+0x1cee8/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377] RSP: 0018:ffff8801d63ffa68  EFLAGS: 00010286
May 10 10:42:26 maggie kernel: [  308.782377] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000003
May 10 10:42:26 maggie kernel: [  308.782377] RDX: 0000000000000001 RSI: ffff8801d63ffa38 RDI: 00007f579403be30
May 10 10:42:26 maggie kernel: [  308.782377] RBP: ffff8801d63ffae8 R08: 0000000000000008 R09: 0000000000000000
May 10 10:42:26 maggie kernel: [  308.782377] R10: 0000000000000fa0 R11: ffffffffa041f430 R12: 00007f57950ef260
May 10 10:42:26 maggie kernel: [  308.782377] R13: ffffc900118d3000 R14: ffffc900118eeb00 R15: 0000000000000008
May 10 10:42:26 maggie kernel: [  308.782377] FS:  00007f5795bae700(0000) GS:ffff880002180000(0000) knlGS:0000000000000000
May 10 10:42:26 maggie kernel: [  308.782377] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 10 10:42:26 maggie kernel: [  308.782377] CR2: 0000000000002dc4 CR3: 00000001e6c77000 CR4: 00000000000026e0
May 10 10:42:26 maggie kernel: [  308.782377] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 10 10:42:26 maggie kernel: [  308.782377] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 10 10:42:26 maggie kernel: [  308.782377] Process VirtualBox (pid: 5584, threadinfo ffff8801d63fe000, task ffff8801fdce1740)
May 10 10:42:26 maggie kernel: [  308.782377] Stack:
May 10 10:42:26 maggie kernel: [  308.782377]  ffffffffa03c9830 00000000000001f4 ffff8801d63ffae8 ffffffffa03d8c17
May 10 10:42:26 maggie kernel: [  308.782377] <0> 00000000000b8000 ffff880100000002 000000000000064e ffff8801d63ffc27
May 10 10:42:26 maggie kernel: [  308.782377] <0> ffffc900118eeb00 0000000300000000 ffffc900118ee000 0000000000000002
May 10 10:42:26 maggie kernel: [  308.782377] Call Trace:
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? g_abExecMemory+0x1e070/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? g_abExecMemory+0x2d457/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0x3508b/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? g_abExecMemory+0x5907e/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0x2cece/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0x11e1e/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? g_abExecMemory+0xf840/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? g_abExecMemory+0xf25e/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0xa18d/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0x49f97/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] g_abExecMemory+0x1599b/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? unlock_page+0x27/0x2c
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? __do_fault+0x342/0x379
May 10 10:42:26 maggie kernel: [  308.782377]  [] supdrvIOCtlFast+0x50/0x54 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] VBoxDrvLinuxIOCtl+0x44/0x1b0 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  [] ? pmd_offset+0x19/0x40
May 10 10:42:26 maggie kernel: [  308.782377]  [] vfs_ioctl+0x36/0xa7
May 10 10:42:26 maggie kernel: [  308.782377]  [] do_vfs_ioctl+0x468/0x49b
May 10 10:42:26 maggie kernel: [  308.782377]  [] sys_ioctl+0x56/0x79
May 10 10:42:26 maggie kernel: [  308.782377]  [] system_call_fastpath+0x16/0x1b
May 10 10:42:26 maggie kernel: [  308.782377] Code: 24 60 45 89 f8 48 8b 55 a0 41 ff d3 85 c0 89 c1 44 8b 55 88 0f 85 91 fe ff ff 45 89 ff 42 8b 0c bd c0 70 40 a0 41 d3 e2 4d 01 16  83 c4 2d 00 00 10 0f 84 6a fe ff ff 41 c7 46 30 00 00 00 00
May 10 10:42:26 maggie kernel: [  308.782377] RIP  [] g_abExecMemory+0x1cee8/0x180000 [vboxdrv]
May 10 10:42:26 maggie kernel: [  308.782377]  RSP
May 10 10:42:26 maggie kernel: [  308.782377] CR2: 0000000000002dc4
May 10 10:42:26 maggie kernel: [  308.782663] ---[ end trace b439b59bc93da8ee ]---

I’ve done the standard Googling, but nothing jumped out at me as a possible solution.

Cheers

Tim…