out of process memory

All posts relating to Oracle database administration.

Moderator: Tim...

out of process memory

Postby royalyogi » Wed Jun 12, 2013 7:39 am

we are using windows server 2008 R2 and oracle database version 11.2.0.2
Have two points here : I/O performance and Out of memory issue

1.) ADDM Report recommended solution is to stripe all data files for to reduce "Wait class "User I/O" was consuming"
But I am not sure It will improve more I/O performance(it will resolved the issue) and same issue we are getting on many servers.
Need you help to resolve this issue. Please help us

ADDM Report Snipped:

Findings and Recommendations
----------------------------

Finding 1: I/O Throughput
Impact is .28 active sessions, 44.97% of total activity.
--------------------------------------------------------
The throughput of the I/O subsystem was significantly lower than expected.

Recommendation 1: Host Configuration
Estimated benefit is .28 active sessions, 44.97% of total activity.
-------------------------------------------------------------------
Action
Consider increasing the throughput of the I/O subsystem. Oracle's
recommended solution is to stripe all data files using the SAME
methodology. You might also need to increase the number of disks for
better performance.
Rationale
During the analysis period, the average data files' I/O throughput was
221 K per second for reads and 44 K per second for writes. The average
response time for single block reads was 26 milliseconds.

Recommendation 2: Host Configuration
Estimated benefit is .22 active sessions, 35.45% of total activity.
-------------------------------------------------------------------
Action
The performance of some data and temp files was significantly worse than
others. If striping all files using the SAME methodology is not
possible, consider striping these file over multiple disks.
Rationale
For file E:\APP\ORACLE\ORADATA\OEMGC\MGMT.DBF, the average response time
for single block reads was 34 milliseconds, and the total excess I/O
wait was 772 seconds.
Related Object
Database file
"E:\APP\ORACLE\ORADATA\OEMGC\MGMT.DBF"
Rationale
For file E:\APP\ORACLE\ORADATA\OEMGC\SYSTEM01.DBF, the average response
time for single block reads was 22 milliseconds, and the total excess
I/O wait was 432 seconds.
Related Object
Database file
"E:\APP\ORACLE\ORADATA\OEMGC\SYSTEM01.DBF"
Rationale
For file E:\APP\ORACLE\ORADATA\OEMGC\SYSAUX01.DBF, the average response
time for single block reads was 64 milliseconds, and the total excess
I/O wait was 144 seconds.
Related Object
Database file
"E:\APP\ORACLE\ORADATA\OEMGC\SYSAUX01.DBF"

Symptoms That Led to the Finding:
---------------------------------
Wait class "User I/O" was consuming significant database time.
Impact is .37 active sessions, 59.04% of total activity.


2.) Same issue we got in another server and we increased memory thrice but again we got same below error. Now we are getting same below
error another server. As it look like OS issue but It can be Oracle issue as well. Please through some light here also.

Alert log Snipped:=
Wed Jun 12 07:12:15 2013
Process startup failed, error stack:
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_psp0_3316.trc:
ORA-27300: OS system dependent operation:CreateThread failed with status: 8
ORA-27301: OS failure message: Not enough storage is available to process this command.
ORA-27302: failure occurred at: ssthrddcr
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73309):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jun 12 07:12:16 2013
Process m000 died, see its trace file
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73311):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73312):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73313):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73383):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73384):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73402):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73429):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73430):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73431):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jun 12 07:12:39 2013
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_ora_4664.trc (incident=73432):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jun 12 07:13:12 2013
Errors in file E:\APP\ORACLE\diag\rdbms\oemgc\oemgc\trace\oemgc_j000_10096.trc (incident=73292):
ORA-04030: out of process memory when trying to allocate 1048592 bytes (kxs-heap-w,kcbpioqInitClientBuffer)
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jun 12 07:13:17 2013
Sweep [inc][73432]: completed
Sweep [inc][73431]: completed
Sweep [inc][73430]: completed
Sweep [inc][73429]: completed
Sweep [inc][73402]: completed
Sweep [inc][73384]: completed
royalyogi
Senior Member
 
Posts: 166
Joined: Tue Apr 23, 2013 3:59 pm

Re: out of process memory

Postby Tim... » Wed Jun 12, 2013 8:00 am

Hi.

Storage performance: What do databases really do? They store data on disk. What impact do you think having a faster disk subsystem will have? It will improve database performance, provided disk I/O is your biggest limiting factor. Does disk I/O seem to be a problem for you? According to ADDM it is a big problem for you. Therefore, improving storage performance is the single biggest factor you need to consider to improve the performance of your system.

How do you make your storage faster? It depends. You could have faster disks or use more disks. If you are using more disks, the recommendation is to use SAME (Stripe And Mirror Everything).

You don;t say what these disks are on your system. Are they local disks in the server, or LUNs on the a SAN or NAS? If they are the latter, then you need to speak to your storage guys and ask why your LUNs are so slow. If they are the former, then there probably isn't much you can do to improve things. If you only have 2-3 disks, then trying to be too clever about file placement is pointless. You needs to get some proper storage, or just accept your current levels of performance.

Errors: Regarding the error messages, although one looks like a disk error, it is actually likely they are all down to memory. Your system has run out of memory, hence the "ORA-04030: out of process memory..." errors. My guess is you've been allocating more memory to Oracle and used up all your physical memory and started to use Swap. At some point you have used up all your swap and things have gone gaga.

On small systems, you should not allocate more than about 60-70% or the total physical memory on the server to Oracle. You have to leave some behind for the OS to do its thing. When ADDM is telling you to allocate more memory to a database, it doesn't mean keep adding it past the point where you run out of physical memory and start swapping. In that case, you would have to buy more memory...

Also, if you have multiple instances running on a box, they are sharing the physical memory, so you have to reduce their allocations.

So reduce your Oracle memory allocations until you reach a point where your server is no longer swapping...

Cheers

Tim...
Tim...
Oracle ACE Director
Oracle ACE of the Year 2006 - Oracle Magazine Editors Choice Awards
OakTable Member
OCP DBA 7.3, 8, 8i, 9i, 10g, 11g
OCP Advanced PL/SQL Developer
Oracle Database: SQL Certified Expert
My website: http://www.oracle-base.com
My blog: http://www.oracle-base.com/blog
Tim...
Site Admin
 
Posts: 17935
Joined: Mon Nov 01, 2004 5:56 pm
Location: England, UK

Re: out of process memory

Postby royalyogi » Tue Jun 18, 2013 7:19 am

Sorry for late replied.

We are running only one database in this server box and total RAM is 20GB. we are allocated 10GB memory to database(50% of RAM).
But yesterday we got same error again. So I am suspecting cuase of ORAVSS.exe becuase we are using VSS feature.Kindly through some light on this.
royalyogi
Senior Member
 
Posts: 166
Joined: Tue Apr 23, 2013 3:59 pm

Re: out of process memory

Postby Tim... » Tue Jun 18, 2013 7:24 am

Hi.

There are certainly some references on the net suggesting VSS is taking lots of memory. :)

Well, the first thing you need to do is identity what is taking the memory, rather than guessing. You should able to see this on the OS using Task Manager on Windows, or TOP in Unix/Linux. Once you know what is using it, then you can think about finding a solution.

If it is VSS, then you need to contact Oracle Support.

Cheers

TIm...
Tim...
Oracle ACE Director
Oracle ACE of the Year 2006 - Oracle Magazine Editors Choice Awards
OakTable Member
OCP DBA 7.3, 8, 8i, 9i, 10g, 11g
OCP Advanced PL/SQL Developer
Oracle Database: SQL Certified Expert
My website: http://www.oracle-base.com
My blog: http://www.oracle-base.com/blog
Tim...
Site Admin
 
Posts: 17935
Joined: Mon Nov 01, 2004 5:56 pm
Location: England, UK


Return to Oracle Database Administration

Who is online

Users browsing this forum: No registered users and 11 guests