Question False alert that AppServers & WebSpeed brokers failled to start.

[Cecil]

Hi all,

This is bugged me for about year and it started off with OE 11.1. When starting some of my AppServers & WebSpeed Brokers OpenEdge Explorer returns and Error: 'Unable to start broker blar blar blar... check the broker log file'.

However the Brokers & Agents start perfectly with absolutely with no errors in the broker or server log files.

It doesn't mater if I am starting the brokers from the Command line or vie OE Management I get the false error.

I have got Tech Support call open and I sending them every possible log file you can imagine.

Now the workaround solution is to delete and recreate the broker, but it's hit or miss if it will fix the phenomenon.

OE: 11.1~4
OS:CentOS 6.5 64bit

Here is the results of stopping, starting and querying one of my AppServers. Note the error after I start the AppServer Broker:

Unable to start ASOnlineReportingDEV (8297)

proenv>asbman -name ASOnlineReportingDEV -stop
OpenEdge Release 11.4 as of Fri Jul 25 18:21:09 EDT 2014


Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)
Searching for ASOnlineReportingDEV (8288)
Connecting to ASOnlineReportingDEV  (8276)
Shut down ASOnlineReportingDEV (8277)
proenv>asbman -name ASOnlineReportingDEV -start
OpenEdge Release 11.4 as of Fri Jul 25 18:21:09 EDT 2014


Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)
Searching for ASOnlineReportingDEV (8288)
Connecting to ASOnlineReportingDEV  (8276)
Unable to start ASOnlineReportingDEV (8297)
proenv>asbman -name ASOnlineReportingDEV -query
OpenEdge Release 11.4 as of Fri Jul 25 18:21:09 EDT 2014


Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)
Searching for ASOnlineReportingDEV (8288)
Connecting to ASOnlineReportingDEV  (8276)

Broker Name                    : ASOnlineReportingDEV
Operating Mode                 : Stateless
Broker Status                  :  ACTIVE
Broker Port                    : 5565
Broker PID                     : 531
Active Servers                 : 5
Busy Servers                   : 0
Locked Servers                 : 0
Available Servers              : 5
Active Clients (now, peak)     : (0, 0)
Client Queue Depth (cur, max)  : (0, 0)
Total Requests                 : 0
Rq Wait (max, avg)             : (0 ms, 0 ms)
Rq Duration (max, avg)         : (0 ms, 0 ms)

PID   State     Port  nRq    nRcvd  nSent  Started          Last Change
00567 AVAILABLE 02002 000000 000000 000000 Sep 25, 2014 08:54 Sep 25, 2014 08:54
00569 AVAILABLE 02003 000000 000000 000000 Sep 25, 2014 08:54 Sep 25, 2014 08:54
00571 AVAILABLE 02004 000000 000000 000000 Sep 25, 2014 08:54 Sep 25, 2014 08:54
00573 AVAILABLE 02005 000000 000000 000000 Sep 25, 2014 08:54 Sep 25, 2014 08:54
00575 AVAILABLE 02006 000000 000000 000000 Sep 25, 2014 08:54 Sep 25, 2014 08:54

I even produced a youTube video back in June this year.

 

[TheMadDBA]

Did they open this KB/defect for you? Defect PSC00310164

http://knowledgebase.progress.com/articles/Article/000051987

The 8297 error in this specific situation appears to be a timing problem. When we execute asbman -i AppServerBrokerName -start, the asbman makes a request of the AdminServer (via RMI) to start the AppServer. The asbman waist for confirmation that the AppServer started. The confirmation comes from the AdminServer which started the AppServer and is waiting for it to become ACTIVE. The AdminServer knows that the AppServer is active when the AppServer responds to the AdminServer that it is active. If the AppServer takes time that is greater than the wait time to start, then the asbman -start will return the 8297 message. An asbman -q after that will show that the server is started or starting. This is a timing issue and may be associated with the amount of work or number of databases being connected to on startup of the database.

 

[Cecil]

Hey that is very interesting, I did not see that KB article. It does seem to be known issue. Also Tech Support didn't even mentioned this. Plus there is no workaround, apart from ignoring the error.

One the thing that does make my 'spider sense tingle' is that I'm running my development in a VM environment. I've have moved my VM across different hosted machines in the last 12 months, all of different processors & memory and it's made no difference when stating the AppSevers/WebSpeed Brokers. Hummm.

 

[TheMadDBA]

Talking to first level PSC support is kind of hit or miss. A lot of times they are just searching the KB. If you are able to get past the first line of defense the support people start to get much more helpful

Depending on how the VM is set up it might need to claim memory/resources from the main box or other VMs when you start using it. For prod boxes I prefer hard min/max values for CPU,memory,etc if I have to run on a VM.

Anything special going on at appserver startup? Like connecting to multiple databases, or dbs using -H/-S or startup procedures?

 

[Cecil]

Anything special going on at appserver startup? Like connecting to multiple databases, or dbs using -H/-S or startup procedures?

The AppServer and WebSpeed brokers are connecting to single database via shared memory.

The word from tech. guy is that it is a known bug but has low priority and hence "Under Consideration for OE11.5". Normally from my experience if bugs have around a few releases it becomes "Expected Behaviour".