8i | 9i | 10g | 11g | 12c | 13c | 18c | 19c | 21c | 23c | Misc | PL/SQL | SQL | RAC | WebLogic | Linux
SYSRESV Utility and "ORA-01041: internal error. hostdef extension doesn't exist"
This article provides an example of using the SYSRESV utility to identify the shared memory segments associated with an Oracle instance.
This specific issue is not the only cause of an ORA-01041.
The Problem
A couple of times we've had problems with instances not responding. When we try to connect to the database from the server we get something like this.
$ export ORACLE_SID=orcl $ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-01041: internal error. hostdef extension doesn't exist $
When you check on the OS you can see the processes running, but there is no way to connect. Even a preliminary connection fails. There is nothing you can do with the instance, so the obvious thing to try and do is kill it, but you can’t do this with a "shutdown" or "shutdown abort" because you can't connect. Instead you try to kill it at the OS level.
$ kill -9 `ps -ef | grep orcl | awk '{print $2}'`
After this you see all the processes gone, but you still can't connect to the instance.
$ export ORACLE_SID=orcl $ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 13 08:41:23 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-01041: internal error. hostdef extension doesn't exist $
The Solution (SYSRESV Utility)
The last time this happened we guessed it might be about shared memory segments hanging around after the instance has died. You can check the shared memory directly, but it can be difficult to relate this back to the instances. One way is by comparing the creation time of the shared memory segment with the V$INTANCE.STARTUP_TIME column of other running instances on the server and use a process of elimiation. The type of output you can expect from "ipcs -mpt" is shown below.
$ ipcs -mpt IPC status from /dev/kmem as of Wed Jul 13 10:04:09 2016 T ID KEY MODE OWNER GROUP CPID LPID ATIME DTIME CTIME Shared Memory: m 0 0x411c0123 --rw-rw-rw- root root 957 29670 7:14:00 7:14:00 9:47:13 m 1 0x4e0c0002 --rw-rw-rw- root root 957 4731 9:06:51 9:15:11 9:47:13 m 2 0x41200006 --rw-rw-rw- root root 957 29670 7:14:00 7:14:00 9:47:13 m 3 0x00a5c581 --rw------- sfmdb users 2065 2102 9:48:14 9:48:14 9:48:14 m 4 0x06347849 --rw-rw-rw- root root 2623 2629 9:48:50 9:48:47 9:48:47 m 5 0x0c6629c9 --rw-r----- root root 2644 4295 10:33:52 10:34:17 9:48:47 m 32774 0xa90c08cc ----------- root root 2954 2955 9:49:03 no-entry 9:49:03 m 7 0x012009d0 --rw-rw-r-- root root 2660 2660 9:48:48 9:48:48 9:48:48 m 8 0x012009cc --rw-rw-r-- root root 2662 2662 9:48:48 9:48:48 9:48:48 m 9 0x012009c4 --rw-rw-r-- root root 2666 2666 9:48:48 no-entry 9:48:48 m 10 0x49143b35 --rw-r--r-- root root 2629 2629 10:04:00 10:04:00 9:48:49 m 11 0x012009ce --rw-rw-r-- root root 2709 2709 9:48:49 no-entry 9:48:49 m 32780 0x00000000 --rw-r----- oracle oinstall 6230 11639 10:03:44 10:03:44 12:00:36 m 13 0x00000000 --rw-r----- oracle oinstall 6230 11639 10:03:44 10:03:44 12:00:36 m 14 0x00000000 --rw-r----- oracle oinstall 6230 11639 10:03:44 10:03:44 12:00:36 m 15 0x588c2d20 --rw-r----- oracle oinstall 6230 11639 10:03:44 10:03:44 12:00:36 m 32784 0x00000000 --rw-r----- oracle oinstall 6560 11683 10:04:03 10:04:03 12:01:02 m 17 0x00000000 --rw-r----- oracle oinstall 6560 11683 10:04:03 10:04:03 12:01:02 m 18 0x375d46a0 --rw-r----- oracle oinstall 6560 11683 10:04:03 10:04:03 12:01:02 m 32787 0x00000000 --rw-r----- oracle oinstall 7117 11663 10:04:00 10:03:46 12:04:33 m 20 0x00000000 --rw-r----- oracle oinstall 7117 11663 10:04:00 10:03:46 12:04:33 m 21 0x00000000 --rw-r----- oracle oinstall 7117 11663 10:04:00 10:03:46 12:04:33 m 22 0x00000000 --rw-r----- oracle oinstall 7117 11663 10:04:00 10:03:46 12:04:33 m 23 0x10badea0 --rw-r----- oracle oinstall 7117 11663 10:04:00 10:04:00 12:04:33 m 98328 0x00000000 --rw-r----- oracle oinstall 7969 11681 10:04:02 10:04:02 12:06:39 m 25 0x00000000 --rw-r----- oracle oinstall 7969 11681 10:04:02 10:04:02 12:06:39 m 26 0x00000000 --rw-r----- oracle oinstall 7969 11681 10:04:02 10:04:02 12:06:39 m 27 0x00000000 --rw-r----- oracle oinstall 7969 11681 10:04:02 10:04:02 12:06:39 m 28 0x09c1865c --rw-r----- oracle oinstall 7969 11681 10:04:02 10:04:02 12:06:39 m 196637 0x00000000 --rw-r----- oracle oinstall 8741 11665 10:04:00 10:03:49 12:08:13 m 30 0x00000000 --rw-r----- oracle oinstall 8741 11665 10:04:00 10:03:49 12:08:13 m 31 0x00000000 --rw-r----- oracle oinstall 8741 11665 10:04:00 10:03:49 12:08:13 m 32 0x00000000 --rw-r----- oracle oinstall 8741 11665 10:04:00 10:03:49 12:08:13 m 33 0xb747ec2c --rw-r----- oracle oinstall 8741 11665 10:04:00 10:04:00 12:08:13 m 426018 0x00000000 --rw-r----- oracle oinstall 2869 9042 10:03:02 10:04:00 8:55:00 m 32803 0x00000000 --rw-r----- oracle oinstall 2869 9042 10:03:02 10:04:00 8:55:00 m 32804 0x00000000 --rw-r----- oracle oinstall 2869 9042 10:03:02 10:04:00 8:55:00 m 32805 0x00000000 --rw-r----- oracle oinstall 2869 9042 10:03:02 10:04:00 8:55:00 m 32806 0x472f4a20 --rw-r----- oracle oinstall 2869 9042 10:03:02 10:04:00 8:55:00 $
Instead, you can use the "$ORACLE_HOME/bin/sysresv" command to give you the shared memory segments associated with a named instance.
$ sysresv -l orcl IPC Resources for ORACLE_SID "orcl" : Shared Memory: ID KEY 98338 0x00000000 35 0x00000000 36 0x00000000 37 0x00000000 38 0x472f4a20 Semaphores: ID KEY 12312 0xdd03b17c Oracle Instance not alive for sid "orcl" $
So we can see there are shared memory segments for an instance that is now dead.
It worked as expected, but if for some reason it is not able to link the dead instance to the shared memory segments, you can always work backwards. Check the segments for all the running instances. If there are any shared memory segments for "oracle oinstall" from the "ipcs -mpt" command that are not seen in the following output, by a process of elimination, they must belong to the dead instance.
$ sysresv -l instance1 instance2 instance3
Once we know the dodgy shared memory segments (directly or indirectly) we can we can remove them using "ipcrm".
$ ipcrm -m 98338 $ ipcrm -m 35 $ ipcrm -m 36 $ ipcrm -m 37 $ ipcrm -m 38
Once that is done, you should be able to connect to, and start the instance normally.
It's possible once you remove the first segment (98338) the others will no longer be seen as linked to the instance, but you will still need to remove them or you will be able to connect, but the instance will not start.
For more information see:
Hope this helps. Regards Tim...