Large SGA On Linux
On 32-bit Linux the total SGA size is limited to around 2Gig. There are several ways to allocate more memory than this, but the most robust and scalable method requires the use of a shared memory file, like shmfs in RHEL 2.1 or ramfs in RHEL 3-5. This article presents the methods necessary to create very large buffer caches on Red Hat Enterprise Linux and its clones.
- Red Hat Enterprise Linux (RHEL) 2.1
- Red Hat Enterprise Linux (RHEL) 3, 4 and 5
- Setting the SHMMAX value
- Instance Parameters
Red Hat Enterprise Linux (RHEL) 2.1
Create a shared memory file system (shmfs)
The shmfs is a memory file system so it can be as large as the maximum allowable virtual memory supported by Red Hat Linux AS2.1, currently 16 GB, although the enterprise kernel theoretically supports up to 64 GB of RAM.
The shmfs is created using the following command as the root user.
mount -t shm shmfs -o size=3g /dev/shm
The shared memory file system can be mounted automatically by adding the following line into "/etc/fstab" file.
shmfs /dev/shm shm size=3g 0 0
In the above example I've created the shmfs with a size of 3G as that is the size of the buffer cache I am planning to use. The other elements of the SGA are placed in regular memory, not this shared memory file system, so they should not be included when deciding on the size of the shmfs. It is advisable to size this slightly bigger than the actual size needed, but in this example I've used a 3G shmfs for a 3G buffer cache.
Enabling big pages
Big pages are enabled by adding the
bigpages=xMB to the relevant kernel entry in the boot loader file /boot/grub/grub.conf file, where "x" is calculated as follows:
(Total SGA size in Gig) x 1024
Then round this value to the nearest hundredth. So for a 4G SGA we would do the following.
4 x 1024 = 4096 = 4100
So the "/boot/grub/grub.conf" file entry might look like this.
kernel /vmlinuz-2.4.9-e.40enterprise ro root=/dev/cciss/c0d0p2 bigpages=4100MB
With this entry saved the system should be rebooted. Once the system is available you must perform the following command as the root user.
echo 2 > /proc/sys/kernel/shm-use-bigpages
Alternatively you can add the following entry into the "/etc/sysctl.conf" file so this value persists between reboots.
kernel.shm-use-bigpages = 2
Red Hat Enterprise Linux (RHEL) 3, 4 and 5
Mount the ramfs filesystem and make sure it is usable by Oracle.
umount /dev/shm mount -t ramfs ramfs /dev/shm chown oracle:oinstall /dev/shm
Add these commands into the "/etc/rc.local" file. As with the RHEL 2.1 method, only the buffer cache is stored in this location.
For Oracle to lock shared memory for the shared pool, we must adjust the "memlock" parameter in the "/etc/security/limits.conf". The "memlock" figure is specified in Kb, so for a 2G shared pool we would need to set at least (2 * 1024 * 1024 = 2097152), as shown below.
oracle soft memlock 2097152 oracle hard memlock 2097152
In reality, it is better to oversize it a little.
Setting the SHMMAX value
For 32 bit architecture, the shmmax value should be set at half the physical memory up to a maximum of 4294967295. For a server with 6G of memory we can set this value to 3G (half physical memory) using the following command as the root user.
echo 3221225472 > /proc/sys/kernel/shmmax
Alternatively it can be set in the /etc/sysctl.conf file with the following entry.
kernel.shmmax = 3221225472
The contents of your "/etc/sysctl.conf" file may look something like this.
kernel.shmmax = 3221225472 kernel.shmmni = 4096 kernel.shmall = 2097152 kernel.sem = 1000 32000 100 150 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 kernel.shm-use-bigpages = 2
Alterations to the "/etc/sysctl.conf" file can be applied without a reboot by issuing the following command as root.
Some instance parameter changes are necessary to allow the Oracle instance to use the shared memory file system. The spfile parameters can be manipulated using the ALTER SYSTEM SET command in a running instance, or by modifying the spfile contents offline.
-- Change the parameter value in the spfile directly. ALTER SYSTEM SET parameter = value SCOPE=spfile -- Create a pfile with the contents of the current spfile. CREATE PFILE='/tmp/pfile' FROM SPFILE; -- Manually manipulate the contents of the pfile. -- Recreate the spfile from the amended pfile. CREATE SPFILE FROM PFILE='/tmp/pfile';
The following parameter should be added to the spfile or pfile.
Also, any references to db_cache_size and db_xK_cache_size parameters should be removed and replaced with the old style db_block_buffers parameter entry.
# 3Gig for an 8K db_block_size. ((3 * 1024 * 1024) / 8) = 393216 db_block_buffers = 393216
This means that the multiple block size feature is not available when using this method. Remember that the buffer cache is only one part of the SGA.
For more information see:
Hope this helps. Regards Tim...