BROKER: Could not spawn a server. (1890)

#1
HI All,

Today we received below application connectivity error.
Broker : Could not spawn a server. (1890).

For 5 mins users are able to connect to application after few mins they facing the same.

Can i know reason during which situation we will face this error.Please provide your inputs.

I checked in Knowledge base we need to increase -Mn parameter values.
But we have good enough -Mn parameter value and is set to 35.

And also i cross checked in database log file i not found any error.

I checked -in Promon > R&D > servers.

09/11/18 Status: Servers

11:38:40 Sv Pend. Cur. Max. Port

No Pid Type Protocol Logins Users Users Users Num

31 572330 Auto tcp 10 0 4 15 13425

32 431622 Auto tcp 10 0 5 15 13430

33 242904 Auto tcp 7 0 5 15 13431

34 362192 Auto tcp 7 0 5 15 13436

35 204204 Auto tcp 8 0 5 15 13440


Progress version : 9.1e
OS : AIX 5.3
 

Attachments

Rob Fitzpatrick

ProgressTalk.com Sponsor
#2
Can i know reason during which situation we will face this error.Please provide your inputs.
Client receives error message (1890) on attempted connection to a running database

A connection broker is attempting to spawn a server in response to a connection request from a client. It can't do that because it has run out of eligible unused ports with the range specified for -minport and -maxport.

Please post the complete startup parameters of your primary database broker and any secondary connection brokers.

I checked in Knowledge base we need to increase -Mn parameter values.
This is not necessarily true.
 
#3
-B 70000 # Number of (8K) Blocks in database buffer
-bibufs 200 # Number of before-image buffers
-aibufs 300 # Number of after-image buffers
#-Mi 2 # Min processes on a client server
-Ma 15 # Max number of REMOTE clients per db server
-Mn 35 # Max number of REMOTE client servers
-Mxs 32768 # Shared memory overflow size (override)
-spin 15000 # Number of spin lock retries
-L 2048000 # Lock Table entries
-minport 13100 # Min Port Number
-maxport 14999 # Max Port Number
-spin 1000 # Spin lock tries
-bibufs 100 # Number of before-image buffers
-aibufs 50 # Number of after-image buffers
-semsets 256 # Dan Foreman's recommendation

database log file


11:50:22 AIW 42: Started. (2518)
11:50:23 BROKER 0: Progress OpenEdge Release 9.1E on AIX. (4234)
11:50:23 BROKER 0: Server started by pgresdba on batch. (4281)
11:50:23 BROKER 0: Started using pid: -28712. (6574)
11:50:23 BROKER 0: Physical Database Name (-db): /progdata/prd/slcdb1/prdslc. (4235)
11:50:23 BROKER 0: Database Type (-dt): PROGRESS. (4236)
11:50:23 BROKER 0: Force Access (-F): Not Enabled. (4237)
11:50:23 BROKER 0: Direct I/O (-directio): Not Enabled. (4238)
11:50:23 BROKER 0: Number of Database Buffers (-B): 70000. (4239)
11:50:23 BROKER 0: Maximum private buffers per user (-Bpmax): 64. (9422)
11:50:23 BROKER 0: Excess Shared Memory Size (-Mxs): 33554432. (4240)
11:50:23 BROKER 0: The shared memory segment is not locked in memory. (10014)
11:50:23 BROKER 0: Current Size of Lock Table (-L): 2048000. (4241)
11:50:23 BROKER 0: Hash Table Entries (-hash): 18289. (4242)
11:50:23 BROKER 0: Current Spin Lock Tries (-spin): 15000. (4243)
11:50:23 BROKER 0: Number of Semaphore Sets (-semsets): 256. (6526)
11:50:23 BROKER 0: Crash Recovery (-i): Enabled. (4244)
11:50:23 BROKER 0: Database Blocksize (-blocksize): 8192. (6573)
11:50:23 BROKER 0: Delay of Before-Image Flush (-Mf): 3. (4245)
11:50:23 BROKER 0: Before-Image File I/O (-r -R): Reliable. (4247)
11:50:23 BROKER 0: Before-Image Truncate Interval (-G): 60. (4249)
11:50:23 BROKER 0: Before-Image Cluster Size: 4194304. (4250)
11:50:23 BROKER 0: Before-Image Block Size: 8192. (4251)
11:50:23 BROKER 0: Number of Before-Image Buffers (-bibufs): 200. (4252)
11:50:23 BROKER 0: BI File Threshold size (-bithold): 0.0 Bytes. (9238)
11:50:23 BROKER 0: BI File Threshold Stall (-bistall): Disabled. (6552)
11:50:23 BROKER 0: After-Image Stall (-aistall): Not Enabled. (4254)
11:50:23 BROKER 0: After-Image Block Size: 8192. (4255)
11:50:23 BROKER 0: Number of After-Image Buffers (-aibufs): 300. (4256)
11:50:23 BROKER 0: Storage object cache size (-omsize): 1024 (8527)
11:50:23 BROKER 0: Maximum Number of Clients Per Server (-Ma): 15. (4257)
11:50:23 BROKER 0: Maximum Number of Servers (-Mn): 36. (4258)
11:50:23 BROKER 0: Minimum Clients Per Server (-Mi): 5. (4259)
11:50:23 BROKER 0: Maximum Number of Users (-n): 1051. (4260)
11:50:23 BROKER 0: Host Name (-H): p5lp5. (4261)
11:50:23 BROKER 0: Service Name (-S): prdslc. (4262)
11:50:23 BROKER 0: Network Type (-N): tcp. (4263)
11:50:23 BROKER 0: Character Set (-cpinternal): iso8859-1. (4264)
11:50:23 BROKER 0: Parameter File: /prd/dba/etc/pf/db_prdslc.pf. (4282)
11:50:23 BROKER 0: Minimum Port for Auto Servers (-minport): 13100. (5648)
11:50:23 BROKER 0: Maximum Port for Auto Servers (-maxport): 14999. (5649)
11:50:23 BROKER 0: This broker supports both 4GL and SQL server groups. (8865)
11:50:23 BROKER 0: Large database file access has been enabled. (9426)
 
#4
proserve /progdata/prd/slcdb1/prdslc -m3 -Mi 3 -Ma 5 -Mn 4 -Mpb 4 -S prdslcoi -ServerType SQL
proserve /progdata/prd/slcbard1/prdslcb -m3 -Mi 3 -Ma 5 -Mn 4 -Mpb 4 -S prdslcboi -ServerType SQL
 
#5
Progress 9.1e (and AIX 5.3) is ancient, obsolete and unsupported. You should upgrade immediately. It is irresponsible to continue to run these releases. You data is at risk.
 
#6
You show -m3 broker startup for 2 different databases. Which one matches your .lg extract? (Presumably prdslc but it is good to check...)

Are there any other brokers starting? Like maybe a 4gl broker? You did apparently start a "both" broker with the db itself.

-Mn should not be on the command line with -m3. Your overall -Mn is 36 (the .lg file says that). If the "both" broker is using more than 32 of those then you could run out before the SQL broker can use 4.
 
#7
HI Tom,

we have only SQL broker for connecting SQL connections.

prdslc.pf file


-B 70000 # Number of (8K) Blocks in database buffer
-bibufs 200 # Number of before-image buffers
-aibufs 300 # Number of after-image buffers
#-Mi 2 # Min processes on a client server
-Ma 15 # Max number of REMOTE clients per db server
-Mn 35 # Max number of REMOTE client servers
-Mxs 32768 # Shared memory overflow size (override)
-spin 15000 # Number of spin lock retries
-L 2048000 # Lock Table entries
-minport 13100 # Min Port Number
-maxport 14999 # Max Port Number
-spin 1000 # Spin lock tries
-bibufs 100 # Number of before-image buffers
-aibufs 50 # Number of after-image buffers
-semsets 256 # Dan Foreman's recommendation

Promon info..

Code:
09/11/18        Status: Startup Parameters
14:57:28

Maximum clients:                   1051
Maximum servers:                   35
Maximum clients per server:        0
Lock table size:                   2048000 entries
Database buffers:                  70000 (560000 kb)
APW queue check time:              100 milliseconds
APW scan time:                     1 seconds
APW buffers to scan:               116
APW max writes / scan:             25
Spinlock tries before timeout:     15000
Before-image buffers:              200 (1600 kb)
Before-image file name:            -
After-image file name:             -
After-image buffers:               300 (2400 kb)


And also we not found any pending users in R&D> servers

09/11/18        Status: Servers
14:57:56

Sv                                     Pend.   Cur.   Max.   Port
No    Pid  Type       Protocol Logins  Users  Users  Users    Num

  0 102360 Login      tcp         346      0      0     15  13017
  1 497924 Auto       tcp          14      0      5     15  13446
  2 105162 Auto       tcp           6      0      5     15  13447
  3 464340 Auto       tcp           5      0      5     15  13448
  4 418184 Login      TCP           0      0      0      5 -10535
  5 707766 Auto       tcp          10      0      5     15  13449
  6  14228 Auto       tcp          10      0      5     15  13450
  7  61066 Auto       tcp           7      0      5     15  13451
  8 486788 Auto       tcp           6      0      5     15  13452
  9 456470 Auto       TCP           8      0      0      5   1025
10 567540 Auto       tcp           6      0      5     15  13453
11 565514 Auto       tcp          12      0      5     15  13454
12 505310 Auto       tcp          11      0      5     15  13455
13 450196 Auto       tcp           7      0      5     15  13456
14 687526 Auto       tcp           7      0      5     15  13457
15 233926 Auto       tcp          13      0      5     15  13458
16 499892 Auto       tcp          10      0      5     15  13460
17 509182 Auto       tcp           9      0      5     15  13461
18 358322 Auto       tcp           9      0      5     15  13464
19 131562 Auto       tcp          19      0      5     15  13465
20  58274 Auto       tcp           7      0      5     15  13466
21 567038 Auto       tcp          12      0      5     15  13467
22 673642 Auto       tcp          11      0      5     15  13468
23 409578 Auto       tcp           7      0      5     15  13470
24 404054 Auto       tcp          11      0      5     15  13471
25 650668 Auto       tcp           8      0      5     15  13472
26   5642 Auto       tcp           7      0      5     15  13474
27 383292 Auto       tcp           9      0      5     15  13477
28 266354 Auto       tcp          19      0      5     15  13478
29 298332 Auto       tcp          40      0      5     15  13479
30 267870 Auto       tcp          18      0      5     15  13481
31 441898 Auto       tcp          14      0      5     15  13482
32 548208 Auto       tcp          18      0      5     15  13483
33 496718 Auto       tcp           7      0      1     15  13485
34      0 Inactive                 0      0      0      0      0
35      0 Inactive                 0      0      0      0      0
 
Last edited by a moderator:

Rob Fitzpatrick

ProgressTalk.com Sponsor
#10
11:50:23 BROKER 0: Minimum Clients Per Server (-Mi): 5. (4259)

34 0 Inactive 0 0 0 0 0
35 0 Inactive 0 0 0 0 0

11:50:23 BROKER 0: Minimum Port for Auto Servers (-minport): 13100. (5648)
11:50:23 BROKER 0: Maximum Port for Auto Servers (-maxport): 14999. (5649)

When all servers of a broker have reached their minimum user count (-Mi) and a client requests a connection, and the broker has not yet spawned its maximum number of servers, it will attempt to spawn another one.

These are your server port numbers in use, according to promon:
Code:
1025

13446
13447
13448
13449
13450
13451
13452
13453
13454
13455
13456
13457
13458

13460
13461

13464
13465
13466
13467
13468

13470
13471
13472

13474

13477
13478
13479

13481
13482
13483

13485
The gaps indicate non-sequential port numbers.

Clearly:
  • You haven't specified -minport/-maxport on the SQL broker, as there is a port 1025 in use, I assume by a SQL server.
  • Either there are multiple database brokers running on this machine with overlapping -minport/-maxport ranges (likely) or there is a lot of cruft in /etc/services within the established -minport/-maxport ranges.
I cannot stress this enough: change your port configuration. For all brokers on all databases. With planning and proper configuration based on user requirements and system resources, this problem is completely avoidable.

Re-using a very large -minport/-maxport range across multiple brokers and databases on a single server is really easy and doesn't require any planning and "just works". Until it doesn't. At that point, unraveling exactly why you got this error is a huge pain. So don't do it.

Do some planning. If a 4GL broker will spawn 40 servers then it needs no more than 40 ports in its -minport/-maxport range. Specifically, select ports that are not defined, and never will be, in /etc/services. Make a habit of adding that range as a comment in /etc/services, keeping it in numerical order. That way, someone coming along after the fact to look at the file for an available port or port range for a broker or servers won't re-use that range. Add a comment that tells people the DB name and broker type. That way you know what to remove from the file when it comes time to decommission the database.

Use a separate range for the servers of every broker in every database, every time. If you maintain that discipline, this problem will become a thing of the past. It may seem like more work but it isn't difficult and in the long run it will save you time.
 
#11
I had the exact issue Rob is explaining, but with replication agents when i was testing replication sets.

(Copy paste fail between prop files, but yeah, min/max port ranges crossed and i lost a good few hours figuring it out)
 
#12
Thanks Rob..

Application team contacted QAD vendor and they suggested us to enlarge ports and to add max/min port for SQl Broker also.
we are going to implement the same after Change freeze window in next week.

Once again we faced application connectivity and found below findings on our database.

10/02/18 Status: Servers
00:15:03

Sv Pend. Cur. Max. Port
No Pid Type Protocol Logins Users Users Users Num

0 152384 Login tcp 878 0 0 15 13017
1 262326 Login TCP 0 0 0 5 -10535
2 733564 Auto tcp 14 0 0 15 13473
3 177712 Auto tcp 78 0 45 15 13130
4 320244 Auto TCP 15 0 3 5 1026
5 432450 Auto tcp 25 0 3 15 13467
6 0 Inactive 0 0 0 15 0
7 126338 Auto tcp 131 0 45 15 13136
8 185360 Auto tcp 58 0 45 15 13138
9 97490 Auto tcp 53 0 45 15 13140
10 142068 Auto tcp 36 0 45 15 13150
11 133908 Auto tcp 43 0 45 15 13156
12 138814 Auto tcp 22 0 46 15 13163
13 258636 Auto tcp 21 0 46 15 13210
14 117834 Auto tcp 24 0 46 15 13285
15 359014 Auto tcp 10 0 45 15 13345
16 358352 Auto tcp 21 0 45 15 13360
17 379144 Auto tcp 31 0 45 15 13370
18 386148 Auto tcp 19 0 45 15 13378
19 380880 Auto tcp 22 0 46 15 13380
20 79900 Auto tcp 7 0 46 15 13391
21 474050 Auto tcp 14 0 45 15 13393
22 414992 Auto tcp 15 0 46 15 13398
23 501598 Auto tcp 15 0 45 15 13400
24 522000 Auto tcp 16 0 46 15 13402
25 56006 Auto tcp 8 0 45 15 13406
26 524638 Auto tcp 19 0 45 15 13412
27 501944 Auto tcp 27 0 45 15 13413
28 465974 Auto tcp 26 0 45 15 13415
29 559502 Auto tcp 27 0 45 15 13417
30 580636 Auto tcp 18 0 46 15 13420
31 557754 Auto tcp 14 0 45 15 13425
32 554236 Auto tcp 26 0 46 15 13429
33 601192 Auto tcp 22 0 45 15 13431
34 521014 Auto tcp 7 0 46 15 13446
35 644282 Auto tcp 9 0 45 15 13449


894 MB
FR chain 0 blocks RM chain 6 blocks
Shared Memory 680020 K Segments 6

34 Servers, 9 Users (6 Local, 3 Remote, 6 Batch),4 Apws


Actually max limit for each server 15 users but curr users reached to 45/46.How it is possible.

And also in .pf we mentioned port range from 13000 to 14999 but i never seen any port using from 14000 on server or in promon report.

How to check port 14000 to 149999 is working or not?
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
#13
Please use [ CODE ] tags for fixed-width output, e.g. from promon, to make it legible. For everyone's benefit, this is the table from the previous post:
Code:
10/02/18 Status: Servers
00:15:03

Sv                                 Pend.  Cur. Max.  Port
No Pid    Type     Protocol Logins Users Users Users  Num
                  
0  152384 Login    tcp      878    0     0     15   13017
1  262326 Login    TCP      0      0     0     5   -10535
2  733564 Auto     tcp      14     0     0     15   13473
3  177712 Auto     tcp      78     0     45    15   13130
4  320244 Auto     TCP      15     0     3     5    1026
5  432450 Auto     tcp      25     0     3     15   13467
6  0      Inactive 0        0      0     15    0
7  126338 Auto     tcp      131    0     45    15   13136
8  185360 Auto     tcp      58     0     45    15   13138
9  97490  Auto     tcp      53     0     45    15   13140
10 142068 Auto     tcp      36     0     45    15   13150
11 133908 Auto     tcp      43     0     45    15   13156
12 138814 Auto     tcp      22     0     46    15   13163
13 258636 Auto     tcp      21     0     46    15   13210
14 117834 Auto     tcp      24     0     46    15   13285
15 359014 Auto     tcp      10     0     45    15   13345
16 358352 Auto     tcp      21     0     45    15   13360
17 379144 Auto     tcp      31     0     45    15   13370
18 386148 Auto     tcp      19     0     45    15   13378
19 380880 Auto     tcp      22     0     46    15   13380
20 79900  Auto     tcp      7      0     46    15   13391
21 474050 Auto     tcp      14     0     45    15   13393
22 414992 Auto     tcp      15     0     46    15   13398
23 501598 Auto     tcp      15     0     45    15   13400
24 522000 Auto     tcp      16     0     46    15   13402
25 56006  Auto     tcp      8      0     45    15   13406
26 524638 Auto     tcp      19     0     45    15   13412
27 501944 Auto     tcp      27     0     45    15   13413
28 465974 Auto     tcp      26     0     45    15   13415
29 559502 Auto     tcp      27     0     45    15   13417
30 580636 Auto     tcp      18     0     46    15   13420
31 557754 Auto     tcp      14     0     45    15   13425
32 554236 Auto     tcp      26     0     46    15   13429
33 601192 Auto     tcp      22     0     45    15   13431
34 521014 Auto     tcp      7      0     46    15   13446
35 644282 Auto     tcp      9      0     45    15   13449
Once again we faced application connectivity and found below findings on our database.
What does this mean exactly? If there were error messages, please share them.

34 Servers, 9 Users (6 Local, 3 Remote, 6 Batch),4 Apws
Clearly this promon output was taken from a different time than the table above (3 remote users), so it doesn't seem relevant.

Actually max limit for each server 15 users but curr users reached to 45/46.How it is possible.
It shouldn't be possible. My guess would be a promon display bug in the Servers screen. In a situation like this I would compare this screen to the list of connected users (promon 1 1) or to the contents of the _connect VST, to try to determine how many users are actually connected per server. Unfortunately, you're on an older release and it has bugs. Another example is that it incorrectly displays port numbers above 32767 as negative numbers, e.g. your SQL broker port. It is shown as a signed integer (-10535) rather than an unsigned integer (55001).

By the way, it is best to avoid using port numbers above 32767 for brokers and servers, though not for this reason. Those high-numbered ports are in the range used by the operating system for dynamically-assigned local ports for outbound connections. In this case, if such a port had already been assigned port 55001 prior to your database start, you would have been unable to spawn your SQL broker.

And also in .pf we mentioned port range from 13000 to 14999 but i never seen any port using from 14000 on server or in promon report.
That isn't surprising. I count 32 4GL servers in use, in a port range of 2000 ports. The broker attempts to use port numbers sequentially, from minport to maxport. In this case, the highest port number it used was 13473. There was no need to try to use a higher number than that.
 
#14
Thanks Rob.

Please find attached screenshot for the error we received during application connection and also i have attached promon report for issue first occurrence and last one.

Error in Database : prdslc

===================================

23:18:59 SRV 18: Connection timed out on socket=110 for usernum 1, attempt disconnect. (1280)

23:19:12 SRV 17: Server's received count 1 does not equal client(2)'s send count 1414744096. (1055)

23:20:52 SRV 18: Server's received count 1 does not equal client(1)'s send count 1414744096. (1055)

23:28:15 SQLSRV2 2: SYSTEM ERROR: Memory violation. (49)

++++++++++++++++++


Still i not able to understand clearly,can you explain again on port values

That isn't surprising. I count 32 4GL servers in use, in a port range of 2000 ports. The broker attempts to use port numbers sequentially, from minport to maxport. In this case, the highest port number it used was 13473. There was no need to try to use a higher number than that.
 

Attachments

Rob Fitzpatrick

ProgressTalk.com Sponsor
#15
Still i not able to understand clearly,can you explain again on port values

That isn't surprising. I count 32 4GL servers in use, in a port range of 2000 ports. The broker attempts to use port numbers sequentially, from minport to maxport. In this case, the highest port number it used was 13473. There was no need to try to use a higher number than that.
A database connection broker uses a single TCP port, specified by its -S parameter. It also spawns some number of servers, up to a maximum number specified by its -Mpb parameter (or by the primary broker's -Mn parameter, in the absence of an -Mpb parameter). Each server spawned by a connection broker binds to a TCP port; the port number is assigned to it by the broker. The broker has a limited range of port numbers that it can use for this purpose. The extent of the range is given by the value of the connection broker's -minport and -maxport parameters. Therefore if a connection broker has an -Mpb value of 40, it must have at least 40 port numbers between -minport and -maxport inclusive. If the broker actually has thousands of port numbers at its disposal, it has far more than it needs and it is highly likely that the higher ones will never be used. As I have discussed, this is somewhat complicated by the fact that 4GL brokers cannot use ports that are currently in use (i.e. some other process has bound to that port) or ports that are defined in /etc/services, whether or not they are currently in use.

This is why it is important to take the time to assign individual port ranges to each connection broker on each database on a given machine, being careful to avoid ranges already defined in /etc/services and ranges used by other databases or applications. Doing so will ensure that small minport/maxport ranges can be used, similar in size to the associated -Mpb values. When database connection brokers are so configured, (1890) error messages can be avoided.

Please find attached screenshot for the error we received during application connection
This time the error is different: a (748). This likely means that -n is too low, i.e. not enough concurrent database connections are possible given the number attempted.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
#18
Can i set -Mpb parameter to 50, so 50 ports can be used and i will reduce maxport and minport to 13000 to 13500.
I wouldn't suggest making arbitrary changes without proper justification. If your minport/maxport range were 501 ports as you suggest it would still be ten times more than you would need with -Mbp 50. Remember, -Mpb is the maximum number of servers for the broker. Each server requires one port. There is no need to be wasteful if you plan properly.

I suggest you first deal with the errors you have been seeing, (1890) and (748). Both are completely preventable with properly configured connection brokers.

My suggestion, at a high level:
  • Determine how many concurrent self-service connections you need.
  • Determine how many concurrent remote ABL connections you need. That number should be equal to or a little less than the product of -Mpb and -Ma.
    • Decide their values based on desired number of clients per server and available resources to run servers. High -Mpb and low -Ma means servers are more responsive to clients, but more resources are required to run the larger number of servers. It also means more ports required in the range of -minport/-maxport.
  • Determine how many concurrent SQL connections you need. The SQL broker has its own values for client/server parameters, including -Mpb, -Ma, -minport, -maxport.
  • Set the primary broker's -Mn value to at least the sum of (4GL broker -Mpb) + (SQL broker -Mpb) + (number of secondary brokers). Double-check that each broker's -minport/-maxport range size is at least as large as its -Mpb and that those ports are not and will not be in use by any other process. If you do this correctly you will eliminate (1890) errors.
  • Add up the total number of concurrent database connections you will need (not just clients), including 4GL clients, SQL clients, utilities, monitors, page writers, watchdog, online backup, etc., but excluding servers and brokers. Set -n on the primary broker to at least this number. Better yet, add 10 or 20 to it for a margin of error. If you do this correctly you will eliminate (748) errors.
When doing this on a multi-database machine, you need to be careful not to overlap the configuration of one broker with another. But that situation is also one in which planning is most important. I have been in the situation of diagnosing such errors on a machine with 80 databases and 160 brokers, where they all have either default -minport/-maxport or they all have a large common range, like -minport 30000 -maxport 31000. It's no fun. Plan, execute, and put these errors behind you.
 
Top