Question Remote servers exhausted with pending connections

RealHeavyDude

Well-Known Member
OpenEdge 11.7.3 on RHEL 7.5 (migrated from 11.3.3 Solaris last weekend).

We are live with the new system and are facing a strange issue in that the users are getting the following error message:

The server or the system has no more resources. Please contact Progress Technical Support. (748)

Relevant database startup parameters:

Code:
-S 47311                                         # The service or port number used by a broker process on the host machine
-minport 8400                                    # The lowest port number in a specified range of port numbers accessible to a client
-maxport 8499                                    # The highest port number in a specified range of port numbers accessible to a client
-Mpb 35                                          # The maximum number of servers that multiple brokers can start to serve remote users for any one protocol
-n 550                                # The maximum number of OpenEdge processes (users) on the database
-Mn 24                                # The maximum number of remote servers
-Ma 15                                # The maximum number of clients per remote server
-Mi 5                                 # The minimum number clients per reomte server

In promon I can see that there are only about 60 users connected.

But in the servers ( R&D - 1 - 3 ) I see that most remote servers are exhausted with 15 pending users.

Therefore I'll take it that issue is not that I've configured the database startup wrong.

The KB Progress KB - Network issues cause client-server connections to fail with error 1154 and 748 suggest to use the -PendConnTime startup parameter which I do not use yet.

Does anybody have any experience with the scenario and the -PendConnTime startup parameter?

Thanks in Advance
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
First, congrats on getting off Solaris. :)

I haven't used -PendConnTime but the first thing that jumps out is that -Mn is only 24. This determines the number of records in the _Servers VST and the number of user numbers reserved in _Connect for servers. So this means you will never reach 35 servers (the value of -Mpb).

With -Mi 5, the broker will want to spawn another server every 5 users (assuming no logouts). Your limit should be 360 remote users (-Mn * -Ma). Did all 24 servers get spawned? Check promon R&D 1 17.

As for pending connections, that means a connecting client contacted the broker and was directed to connect to a server but did not. The server is awaiting that connection and increments its pending count. This can happen if, for example, the firewall rules permit connection on the broker port but not on the full range of server ports. If there is a firewall between client machine(s) and DB server, does it allow connections on all ports between 8400 and 8499?
 

RealHeavyDude

Well-Known Member
Hi Rob,

Thanks for you swift response.

Apart from this issue I am so happy with the new RHEL System. Almost everything server-side is up to 6 times faster.

I did check promon and I saw that all 24 servers were spawned - and most of them were exhausted with 15 pending connections.

In the meantime I got reports from users that sometimes they get an SSL handshake error when trying to start the application. If they try again they can start the application successfully most of the times. The bad thing is, I can't reproduce the SSL handshake error in my test environment to see whether I can see something in the database log. On the production system I did not find anything in the database log that would stand out.

Since there are no firewalls involved I am inclined to think that the pending users are the ones where the database connect failed with the SSL handshake error that I am yet to reproduce.


The value of -Mpb is there just case we need to start a secondary login broker for SQL users which would be some auditor that would need to scan our database for client identifying data. Up to now I was able to prevent that but I don't want to be forced to an SOS deployment into production just to start the SQL engine. The script to start the secondary login broker is there but not used yet.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
The value of -Mpb is there just case we need to start a secondary login broker for SQL users which would be some auditor that would need to scan our database for client identifying data. Up to now I was able to prevent that but I don't want to be forced to an SOS deployment into production just to start the SQL engine. The script to start the secondary login broker is there but not used yet.
For that use case, it sounds like you could get away with having a single SQL server. As it is multi-threaded it can still handle multiple SQL client connections efficiently, if needed. So your SQL broker (if started) could use settings like -m3 -ServerType SQL -S <nnnnn> -Mpb 1 -Ma 5 -Mi 1 -minport <xxxxx> -maxport <xxxxx>. The min/max range could be a single port.

But the value of -Mn should be the sum of all brokers' -Mpb values plus the number of secondary connection brokers. So if you do want to be able to spawn 35 4GL servers (as per -Mpb 35 above) then you would want -Mn to be at least 37. Also, with up to 525 4GL users (35 * 15), plus some number of SQL clients, plus maybe some utilities, helpers, etc., I think -n 550 is a bit lower than I would want. Especially if there are any self-service clients in the mix as well. But this is a lower priority as -n is currently high enough that it won't cause (748) errors.

I've never used -ssl before, though some of my clients are starting to inquire about it now so I may have to bite that bullet eventually. Not sure how to diagnose those SSL errors but it sounds like that is the cause of the pending connections.
 
Top