Financial Appserver went down with the error Connection failure for host 127.0.0.1

[rajeev.babu]

Dear All,

Its been long time I opened a thread. I have faced an issue and I would like to have your valuable comments on the issue I have faced.

Steps:

1. I have changed the propath of production appserver in ubroker.properties. Please note that I didn't restart the appserver to effect the changes.

2. The change was just a line added ie., /prod/apps/qxo/runtime for the qxtend path needs to be added.

3. I did the changes and left it for weekly offline backup with restarting the production appserver

4. After 10 minutes it has been identified that financial appserver got down with the below error

[14/01/24@01:09:11.834-0800] P-017804 T-2735152624 1 AS -- Connection failure for host 127.0.0.1 port 51440 transport TCP. (9407)

[14/01/24@01:09:12.833-0800] P-021734 T-344124912 1 AS -- Connection failure for host 127.0.0.1 port 57772 transport TCP. (9407)

5. Now all of them thinks that the appserver went down due to the changes done by me in ubroker.properties.

Requesting you to help on this issue.

1. Does a small line adding to propath of ubroker.propeties may bring down appserver? Again note that I havent copied anything to ubroker.properties as I know there is a chances of ubroker.properties got corrupted. I typed those lines in ubroker.

2. What does the error means:
[14/01/24@01:09:11.834-0800] P-017804 T-2735152624 1 AS -- Connection failure for host 127.0.0.1 port 51440 transport TCP. (9407)

Awaiting your reply for me to sit and answer to management in a RoundTable

 

[Cringer]

What happens when you revert ubroker.properties to the previous version? Might as well eliminate that as a cause...

 

[rajeev.babu]

Thanks Cringer for the reply.

The environment started working without issues after restarting the financial appserver and production appserver with the propath changes I made.

 

[RealHeavyDude]

When you change anything in the ubroker.properties file, depending on the version you are running - which you don' tell, but I think it is some V9 - some changes will be picked up on-the-fly. Whenever then the AppServer broker attempts to start a new agent then these changed setting will become efffective. The AppServer broker might stop idle agents after the idle time out and will then start new agents on demand. I think that was what happened to you.

Manually editing the ubroker.properties file without proper testing whether this changes will is always risky. There is always a potential that you accidentally screw it up. You say that you added a new line to add an additional entry to the PROPATH. In the ubroker.properties file the PROPATH, like any other property, is always just one line. Depending on the editor you used some automatic line wrapping applied by the editor may have fooled you to think that you can just add a line where you should have appended your change to the existing line.

Heavy Regards, RealHeavyDude.

 

[saurabhbane1983]

Dear All,

Its been long time I opened a thread. I have faced an issue and I would like to have your valuable comments on the issue I have faced.

Steps:

1. I have changed the propath of production appserver in ubroker.properties. Please note that I didn't restart the appserver to effect the changes.

2. The change was just a line added ie., /prod/apps/qxo/runtime for the qxtend path needs to be added.

3. I did the changes and left it for weekly offline backup with restarting the production appserver

4. After 10 minutes it has been identified that financial appserver got down with the below error

[14/01/24@01:09:11.834-0800] P-017804 T-2735152624 1 AS -- Connection failure for host 127.0.0.1 port 51440 transport TCP. (9407)

[14/01/24@01:09:12.833-0800] P-021734 T-344124912 1 AS -- Connection failure for host 127.0.0.1 port 57772 transport TCP. (9407)

5. Now all of them thinks that the appserver went down due to the changes done by me in ubroker.properties.

Requesting you to help on this issue.

1. Does a small line adding to propath of ubroker.propeties may bring down appserver? Again note that I havent copied anything to ubroker.properties as I know there is a chances of ubroker.properties got corrupted. I typed those lines in ubroker.

2. What does the error means:
[14/01/24@01:09:11.834-0800] P-017804 T-2735152624 1 AS -- Connection failure for host 127.0.0.1 port 51440 transport TCP. (9407)

Awaiting your reply for me to sit and answer to management in a RoundTable

Hey Rajeev,

I think the host which the logfile is referring relates to the local host of the machine . Try checking the local host
May be hosts file had a reference to the machine name and internal address on the same line as the local host reference.

Try having them on separate lines.

Thanks

 

[rajeev.babu]

When you change anything in the ubroker.properties file, depending on the version you are running - which you don' tell, but I think it is some V9 - some changes will be picked up on-the-fly. Whenever then the AppServer broker attempts to start a new agent then these changed setting will become efffective. The AppServer broker might stop idle agents after the idle time out and will then start new agents on demand. I think that was what happened to you.

Manually editing the ubroker.properties file without proper testing whether this changes will is always risky. There is always a potential that you accidentally screw it up. You say that you added a new line to add an additional entry to the PROPATH. In the ubroker.properties file the PROPATH, like any other property, is always just one line. Depending on the editor you used some automatic line wrapping applied by the editor may have fooled you to think that you can just add a line where you should have appended your change to the existing line.

Heavy Regards, RealHeavyDude.

 

[rajeev.babu]

Thanks for the reply RealHeavy Dude.

The version we use is OpenEdge Release 10.2B0727. Do you really think that changing something in propath will pick up on-the-fly in OE10.2 ? I do remember we have done lot of these kind of changes and leave it for offline backup without taking any system outage in the weekends.

The changes I made in ubroker.properties was already tested in both, our development and testing environment and I know that there is a big chance of corrupting the ubroker.properties file while editing online. So I took utter care and typed manually the line.

Saurabh: The line specified in the /etc/hosts is as below
127.0.0.1 localhost.localdomain localhost

Where can we check how the ports are allocated in the specified in the error "Connection failure for host 127.0.0.1 port 51440 transport TCP"

 

[RealHeavyDude]

Does the port 51440 belong to a database the AppServer wants to connect to?

Heavy Regards, RealHeavyDude.

 

[rajeev.babu]

Hi RHD,

I am not able to find any port connecting to 51440. Where can i check for the port usages and configuration?

 

[RealHeavyDude]

What happens at startup of an AppServer agent is also defined in the ubroker.properties file. The proerty srvrStartupParam=... will either point to a parameter file ( -pf ) or directly has the parameters for the agent startup. Usually this would include database connection parameters.

The most prominent reasons as to why an AppServer agent cannot start:

1. The database connection parameters specified are invalid or the datbase they are pointing to is down.
2. The PROPATH is not configure correctly so that it can not find the startup procedure.
3. The working directory is either invalid or the agent can not write to it.
4. Ditto for the temporary directory if it is different from the working directory.
5. The path to the log files is either invalid or the broker / agent can not write to it ( properties brokerLogFile and srvrLogFile ).

Never had any other issue when an AppServer start failed on me or somebody else I know.

Heavy Regards, RealHeavyDude.