Question Problems with Appservers

Greetings

Working at a clients whose system (Solaris / 10.2A02 - yes, I know) looks like this:

dot net front end -- external appserver (state-free) -- internal appserver (stateless) -- database

Everything was working (quite) well until a week ago when we noticed that asbman -q -i internal was reporting Active Clients as 511/now and 512/peak and eventually nothing would connect to the database.

Restarted the internal appserver and all became calm again. We have noticed though that on a daily basis (we are currently restarting the internal nightly) Active climbs to between 20+ to 40+. Of the 35 available servers only a few are ever busy sending. As a perfect example, when we saw the 511/512 figures, all 35 servers were running "Available".

We started to debug the situation by running asbman -listclients, seeing that in this case it was conn handle 748 and feeding that in turn to "asbman -q -i internal -clientdetail 748". This gave us the agent pid which we used to send a SIGUSR2 to the process. Generated a protrace which had an empty ABL stack

We then grabbed the connection id and started to search through the broker and server logs.

A relevant few lines are these:

[15/05/25@10:01:31.234+1000] P-013643 T-000001 2 AS AS Application Server connected with connection id: 10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-7e7. (8358)
[15/05/25@10:01:31.234+1000] P-013643 T-000001 3 AS AS ASK Protocol is disabled.
[15/05/25@10:01:31.234+1000] P-013643 T-000001 3 AS AS CSS_CONN before ACK, connection=10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-7e7 bound=0 changed=0
[15/05/25@10:01:31.236+1000] P-013810 T-000001 3 AS AS requestID= 0a356d900000023199143204602200003521580000021885
[15/05/25@10:01:31.236+1000] P-013810 T-000001 2 AS AS -- TRACE: Non-PERSISTENT Procedure 'eris/internal/ReceiptPayment.p' START. (5498)
[15/05/25@10:01:31.245+1000] P-013810 T-000001 2 AS 4GLTRACE Run CheckDiary in me/meperprc.p [Main Block - eris/internal/ReceiptPayment.p @ 775]
[15/05/25@10:01:31.245+1000] P-013810 T-000001 3 AS 4GLTRACE Return from CheckDiary [me/meperprc.p]

What I can't understand is that given the connection id I'm following is the one that ends with -7e7", it seems to disappear almost immediately.

I'm just about at my limit of appserver debugging - can anyone make any suggestions as to where I should go from here or even better point me at some half-decent documentation that might help?

TIA

N/

Added to this, can anyone advise what the following lines mean?

[15/05/25@10:03:26.015+1000] P-013810 T-000001 2 AS AS Application Server connected with connection id: 10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-647. (8358)
[15/05/25@10:03:26.015+1000] P-013810 T-000001 3 AS AS ASK Protocol is disabled.
[15/05/25@10:03:26.015+1000] P-013810 T-000001 3 AS AS CSS_CONN before ACK, connection=10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-647 bound=0 changed=0

its SIGUSR1 that generates the ABL stack - I don't know what USR2 would do.

The active clients (now, peak) isn't very helpful for us. I concentrate on how many agents are available.

The lines in the logs you have highlighted seem to be normal lines - I don't enable trace logging on the app server in prod, maybe your level of activity isn't very high so it won't impact your system.


Basically the application hangs, if you restart the app server then processing will pick back up ?

I'd look to see what the agents are doing in the database when they are processing a request.