Grid Agent fails to start

All posts relating to Oracle database administration.

Moderator: Tim...

Grid Agent fails to start

Postby jnrpeardba » Mon Dec 23, 2013 10:08 am

Morning Tim,

I've tested and trouble-shooted until I can't go any further

Basically I have 3 agents running on this server, 2 of them start and are working ok but 1 fails to start with http listener error. The log file does not give much information as to what could be the cause.
I just wanted to run this by you to see if you could spot something that I am missing:

As you can see agent is not running
Code: Select all
emctl: launching /u02/oradata/LKRESD/agent11g/agent11g/perl/bin/perl /u02/oradata/LKRESD/agent11g/agent11g/bin/emctl.pl status agent
Env changes in emctl recorded in /tmp/emctl.env.28716.diff
Hostname: 'gbl04957.systems.uk.hsbc'
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
OC4J home for agent is : .
EM home for agent is : /u02/oradata/LKRESD/agent11g/agent11g.
URL for agent is :
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
NOHUP File is: /u02/oradata/LKRESD/agent11g/agent11g/sysman/log/emagent.nohup
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
---------------------------------------------------------------
Agent is Not Running
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin #


Within emd.properties - I have the 2 main parameters set
Code: Select all
AgentListenOnAllNICs=FALSE

EMD_URL=https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/


via nslookup all is OK
Code: Select all
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin # nslookup htse-lkres-live.systems.uk.hsbc
Server:         128.163.133.51
Address:        128.163.133.51#53

Name:   htse-lkres-live.systems.uk.hsbc
Address: 128.164.154.232



Code: Select all
/sbin/ifconfig -a|grep -i 128.164.154.232
          inet addr:128.164.154.232  Bcast:128.164.154.255  Mask:255.255.255.0
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/sysman/log #


I can even telnet to the server and port successfully

Code: Select all
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin # telnet
telnet> open htse-lkres-live.systems.uk.hsbc 1830
Trying 128.164.154.232...
Connected to htse-lkres-live.systems.uk.hsbc.
Escape character is '^]'.
]

Connection closed by foreign host.
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin #


the tail end of the emctl.log file shows this
Code: Select all
11010 :: Mon Dec 23 09:40:45 2013::AgentLifeCycle.pm: Processing start agent
11010 :: Mon Dec 23 09:40:45 2013::AgentLifeCycle.pm: EMHOME is /u02/oradata/LKRESD/agent11g/agent11g
11010 :: Mon Dec 23 09:40:45 2013::AgentLifeCycle.pm: service name is
11010 :: Mon Dec 23 09:40:45 2013::EM_SECURE_HOSTNME: htse-lkres-live.systems.uk.hsbc
11010 :: Mon Dec 23 09:40:45 2013::EM_SECURE_PORT: 1830
11010 :: Mon Dec 23 09:40:45 2013::EM_LISTEN_ON_ALL_NICS: FALSE
11010 :: Mon Dec 23 09:40:45 2013::AgentLifeCycle.pm:status agent returned with retCode=1
11010 :: Mon Dec 23 09:40:50 2013::AgentLifeCycle.pm:Watch dog processs id: 11039 exited with an exit code of 55
11010 :: Mon Dec 23 09:40:50 2013::AgentLifeCycle.pm: Exited loop with retCode=1
28734 :: Mon Dec 23 09:51:05 2013::AgentLifeCycle.pm: Processing status agent
28734 :: Mon Dec 23 09:51:05 2013::AgentStatus.pm:Processing status agent
28734 :: Mon Dec 23 09:51:05 2013::AgentStatus.pm:emdctl status returned 1
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/sysman/log # tail -200 emctl.log


My agent home is also set
Code: Select all
 echo $AGENT_HOME
/u01/sq/agent11g/
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/sysman/log #


but when I go to start the agent it fails with this error

Code: Select all
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin # ./emctl start agent
emctl: launching /u02/oradata/LKRESD/agent11g/agent11g/perl/bin/perl /u02/oradata/LKRESD/agent11g/agent11g/bin/emctl.pl start agent
Env changes in emctl recorded in /tmp/emctl.env.12343.diff
Hostname: 'gbl04957.systems.uk.hsbc'
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
OC4J home for agent is : .
EM home for agent is : /u02/oradata/LKRESD/agent11g/agent11g.
URL for agent is :
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
NOHUP File is: /u02/oradata/LKRESD/agent11g/agent11g/sysman/log/emagent.nohup
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LKRESD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LKRESD/agent11g/agent11g
Starting agent ....... failed.
Failed to start HTTP listener.
Consult the log files in: /u02/oradata/LKRESD/agent11g/agent11g/sysman/log
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/bin #


The error in the trace file shows this:
Code: Select all
2013-12-23 09:57:12,272 Thread-4124921216 ERROR ssl: 8: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:12,273 Thread-4124921216 ERROR http: 8: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:12,273 Thread-4124921216 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
2013-12-23 09:57:13,431 Thread-292658560 ERROR ssl: 8: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:13,432 Thread-292658560 ERROR http: 8: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:13,432 Thread-292658560 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
2013-12-23 09:57:14,590 Thread-1377066368 ERROR ssl: 8: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:14,590 Thread-1377066368 ERROR http: 8: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:14,590 Thread-1377066368 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
2013-12-23 09:57:15,737 Thread-1226624384 ERROR ssl: 7: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:15,738 Thread-1226624384 ERROR http: 7: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:15,738 Thread-1226624384 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
2013-12-23 09:57:15,740 Thread-4217113984 ERROR ssl: 8: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:15,741 Thread-4217113984 ERROR http: 8: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:15,741 Thread-4217113984 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
2013-12-23 09:57:16,894 Thread-2426445184 ERROR ssl: 8: Common Name = "gbl04957.systems.uk.hsbc" Does not Match Hostname = "htse-lkres-live.systems.uk.hsbc"
2013-12-23 09:57:16,895 Thread-2426445184 ERROR http: 8: Unable to initialize ssl connection with server, aborting connection attempt: ret -1
2013-12-23 09:57:16,895 Thread-2426445184 ERROR main: nmectla_agentctl: Error connecting to https://htse-lkres-live.systems.uk.hsbc:1830/emd/main/. Returning status code 1
gbl04957_dummy /u02/oradata/LKRESD/agent11g/agent11g/sysman/log #


I can still telnet to the host and port

Code: Select all
gbl04957_dummy /u02/oradata/LEMHD/agent11g/agent11g/bin # telnet
telnet> open htse-lkres-live.systems.uk.hsbc 1830
Trying 128.164.154.232...
Connected to htse-lkres-live.systems.uk.hsbc.
Escape character is '^]'.
]

Connection closed by foreign host.


As you can see below the agents for the other databases are working fine and I can stop and start these at will

Code: Select all
gbl04957_dummy /u02/oradata/LNCBCD/agent11g/agent11g/bin # ./emctl status agent
emctl: launching /u02/oradata/LNCBCD/agent11g/agent11g/perl/bin/perl /u02/oradata/LNCBCD/agent11g/agent11g/bin/emctl.pl status agent
Env changes in emctl recorded in /tmp/emctl.env.28530.diff
Hostname: 'gbl04957.systems.uk.hsbc'
EM HOME ROOT:  /u02/oradata/LNCBCD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LNCBCD/agent11g/agent11g
OC4J home for agent is : .
EM home for agent is : /u02/oradata/LNCBCD/agent11g/agent11g.
URL for agent is :
EM HOME ROOT:  /u02/oradata/LNCBCD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LNCBCD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LNCBCD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LNCBCD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LNCBCD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LNCBCD/agent11g/agent11g
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
NOHUP File is: /u02/oradata/LNCBCD/agent11g/agent11g/sysman/log/emagent.nohup
EM HOME ROOT:  /u02/oradata/LNCBCD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LNCBCD/agent11g/agent11g
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u02/oradata/LNCBCD/agent11g/agent11g
Agent binaries    : /u02/oradata/LNCBCD/agent11g/agent11g
Agent Process ID  : 30612
Parent Process ID : 30395
Agent URL         : https://htse-lkncbc-live.systems.uk.hsbc:3872/emd/main/
Repository URL    : https://ogcgui.systems.uk.hsbc:1159/em/upload
Started at        : 2013-12-20 12:25:37
Started by user   : oracle
Last Reload       : 2013-12-20 13:35:47
Last successful upload                       : 2013-12-23 09:56:31
Total Megabytes of XML files uploaded so far :    51.28
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    87.29%
Last successful heartbeat to OMS             : 2013-12-23 10:02:16
---------------------------------------------------------------
Agent is Running and Ready
gbl04957_dummy /u02/oradata/LNCBCD/agent11g/agent11g/bin #


Code: Select all
gbl04957_dummy /u02/oradata/LEMHD/agent11g/agent11g/bin # ./emctl status agent
emctl: launching /u02/oradata/LEMHD/agent11g/agent11g/perl/bin/perl /u02/oradata/LEMHD/agent11g/agent11g/bin/emctl.pl status agent
Env changes in emctl recorded in /tmp/emctl.env.8943.diff
Hostname: 'gbl04957.systems.uk.hsbc'
EM HOME ROOT:  /u02/oradata/LEMHD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LEMHD/agent11g/agent11g
OC4J home for agent is : .
EM home for agent is : /u02/oradata/LEMHD/agent11g/agent11g.
URL for agent is :
EM HOME ROOT:  /u02/oradata/LEMHD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LEMHD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LEMHD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LEMHD/agent11g/agent11g
EM HOME ROOT:  /u02/oradata/LEMHD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LEMHD/agent11g/agent11g
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
NOHUP File is: /u02/oradata/LEMHD/agent11g/agent11g/sysman/log/emagent.nohup
EM HOME ROOT:  /u02/oradata/LEMHD/agent11g/agent11g/gbl04957.systems.uk.hsbc
EMHOME ==================  /u02/oradata/LEMHD/agent11g/agent11g
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u02/oradata/LEMHD/agent11g/agent11g
Agent binaries    : /u02/oradata/LEMHD/agent11g/agent11g
Agent Process ID  : 6731
Parent Process ID : 6697
Agent URL         : https://htse-lkemh-live.systems.uk.hsbc:1831/emd/main/
Repository URL    : https://ogcgui.systems.uk.hsbc:1159/em/upload
Started at        : 2013-12-20 12:21:09
Started by user   : oracle
Last Reload       : 2013-12-20 13:36:17
Last successful upload                       : 2013-12-23 09:57:00
Total Megabytes of XML files uploaded so far :    52.41
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    87.28%
Last successful heartbeat to OMS             : 2013-12-23 10:04:27
---------------------------------------------------------------
Agent is Running and Ready
gbl04957_dummy /u02/oradata/LEMHD/agent11g/agent11g/bin #


If you have any ideas or can point me in the right direction that would be most appreciated and helpful Tim,

Cheers

Jnrpeardba
jnrpeardba
Advisor
 
Posts: 392
Joined: Wed May 04, 2011 3:14 pm

Re: Grid Agent fails to start

Postby Tim... » Mon Dec 23, 2013 10:24 am

Hi.

First, let me confirm something. Are you really talking about a grid agent, or are you talking about the agent that gets installed as part of enterprise manager DBControl that comes with the databases. They are different things.

- Grid Agent. This is a totally separate software installation and nothing to do with the database or application server software. You should only have one installation on the server, because each server only needs one grid agent to monitor and administer all the services on that server.

- DBControl Agent. This is nothing to do with grid control, so calling it a grid agent is wrong. This is just used for the local DBControl installation and there will be one per database.

So what are we actually talking about here?

Cheers

Tim...
Tim...
Oracle ACE Director
Oracle ACE of the Year 2006 - Oracle Magazine Editors Choice Awards
OakTable Member
OCP DBA 7.3, 8, 8i, 9i, 10g, 11g
OCP Advanced PL/SQL Developer
Oracle Database: SQL Certified Expert
My website: http://www.oracle-base.com
My blog: http://www.oracle-base.com/blog
Tim...
Site Admin
 
Posts: 17937
Joined: Mon Nov 01, 2004 5:56 pm
Location: England, UK

Re: Grid Agent fails to start

Postby jnrpeardba » Mon Dec 23, 2013 10:57 am

Apologies Tim,

this is DBControl Agent

And yes one each per database
jnrpeardba
Advisor
 
Posts: 392
Joined: Wed May 04, 2011 3:14 pm

Re: Grid Agent fails to start

Postby Tim... » Mon Dec 23, 2013 12:00 pm

Hi.

Well, it's many years since I've used this on a real system, since I use Grid Control/Cloud Control for everything these days. It's much simpler, as it only has one agent per server, and it wastes less resources.

All I can point you at is this, which you've probably already seen.

http://www.oracle-base.com/articles/mis ... ooting.php

Cheers

Tim...
Tim...
Oracle ACE Director
Oracle ACE of the Year 2006 - Oracle Magazine Editors Choice Awards
OakTable Member
OCP DBA 7.3, 8, 8i, 9i, 10g, 11g
OCP Advanced PL/SQL Developer
Oracle Database: SQL Certified Expert
My website: http://www.oracle-base.com
My blog: http://www.oracle-base.com/blog
Tim...
Site Admin
 
Posts: 17937
Joined: Mon Nov 01, 2004 5:56 pm
Location: England, UK

Re: Grid Agent fails to start

Postby jnrpeardba » Mon Dec 23, 2013 4:02 pm

Hi Tim,

thanks for getting back to me and yes I looked at that document as a first phase in trouble-shooting, but to no avail

Thanks all the same, I'll continue to try and find a solution and if so, I will update this post

Cheers again Tim,

Jnrpeardba
jnrpeardba
Advisor
 
Posts: 392
Joined: Wed May 04, 2011 3:14 pm

Re: Grid Agent fails to start

Postby Tim... » Mon Dec 23, 2013 4:06 pm

:)
Tim...
Oracle ACE Director
Oracle ACE of the Year 2006 - Oracle Magazine Editors Choice Awards
OakTable Member
OCP DBA 7.3, 8, 8i, 9i, 10g, 11g
OCP Advanced PL/SQL Developer
Oracle Database: SQL Certified Expert
My website: http://www.oracle-base.com
My blog: http://www.oracle-base.com/blog
Tim...
Site Admin
 
Posts: 17937
Joined: Mon Nov 01, 2004 5:56 pm
Location: England, UK


Return to Oracle Database Administration

Who is online

Users browsing this forum: No registered users and 8 guests

cron