Has anyone enabled LargeAddressAware on prowin32.exe or _proapsrv.exe ?

[Dan4gl]

We are having issues with memory leaks and having a really hard time finding them. It could be there is just more data needed for the application. We are trying to use a stop gap measure in the meantime to add LargeAddressAware to the prowin32.exe and _proapsrv.exe . We were wondering if anyone else has tried this and if there are any pitfalls to doing so with these executables? We are trying to stop our users from getting OOM expceptions and crashing out. We used editbin prowin32.exe /largeaddressaware to allow the application to use more than 1.5-2 GB of memory . We have just tested this operation internally and we do see that it stops the OOM exceptions. Any help or advice on this topic or around catching the culprit of memory leaks or using largeaddressaware for 32 bit applications.

We have already enabled appserver disconnect logging that shows us an procedure, object, buffer or other leak checks offered by the ABL.

 

[Rob Fitzpatrick]

I have never enabled LargeAddressAware on a 32-bit executable.

If you are able to reproduce this memory leak in a non-production environment, try enabling some log entry types to capture additional data to the client log (or server log). You can use 4GLTrace to capture run statements, triggers, etc. so you have an idea of what code is running and when; note that this can grow the log file quite substantially. It may help to narrow down which code you need to look at more closely. If you use temp-tables, you could be leaking those; use the temp-table log entry type to find TT creation/deletion. Enable DynObjects logging if you are creating objects dynamically. The Debugging and Troubleshooting manual has lots of info on client logging.

Look in the knowledge base, there are several articles about finding memory leaks.

Also, you didn't mention your OpenEdge release. It is possible that some part of OpenEdge itself, as opposed to your application code, has a memory leak bug. So consider installing the latest update of your release, if you are behind.

 

[Dan4gl]

We are running 11.7.5.009 and we dont have the luxury of enabling client logging on our users systems as that would slow things down and we dont know what sessions are leaking the memory. Its not a workflow that we are able to reproduce.

 

[Rob Fitzpatrick]

we dont know what sessions are leaking the memory

Aren't you able to see this from an OS perspective, e.g. with Task Manager or Process Explorer? Without enabling -logentrytypes, you could still see what code is running by enabling Client Statement Cache and sampling the data in _Connect, either by writing your own code or by using ProTop.

We are running 11.7.5.009

That update is over four years old. The current update is 11.7.17. You should consider updating your client side, at a minimum.

 

[Dan4gl]

We are slated to move to PASOE 12.2.12 in the near future but until then we are on this stable version. We are not able to upgrade to .17 as we would need to do that on over 2500 servers for many different customers. We deploy this as an external application to our customers.

We see it in the OS but are not able to catch the moment when its about to crash or catch the exception. We cant do the client statement cache as much of the crashes are on the client side and there are hundreds of user sessions logged in.

We need either to enable the largeaddressaware or have an automated may to find what is leaking the data in the client.

 

[TomBascom]

I'm a bit confused... why are you using prowin32.exe in the first place? Is that just a bad habit from days of yore? Or do you really _need_ to be using a 32 bit client for some reason?

Also - the "OOM" terminology is usually specific to Linux and its "OOM killer". Are you just kind of generically speaking of being out of memory? If so - then what are the actual errors that you are running into? Or is this maybe a mixed environment with Windows clients and a Linux database and the errors are actually occurring on the Linux side of things?

You mention moving to PASOE in the future and the way you mention that makes it sound like it is expected to solve the problem. But moving to PASOE would only change the appserver layer, not the client so if your errors are with prowin32.exe, PASOE isn't going to make things any better. But if you are in a modern enough situation that you are looking at something like PASOE it brings me back around to: why are you running a 32 bit client?

Lastly, I don't follow why you "cant do" the client statement cache. It works just fine with hundreds of clients and it could give you a ton of insight into what those clients are running just before they crash. Although you would need to do some work to collect that data. Or download ProTop to do it for you

Ok, one more "lastly" - you say that "most" of the crashes are on the client side? So some crashes are on the server side? It sounds like you have a fairly fragile environment that could use a big picture review. We are all happy to help out here in the forum as time permits etc but you might want to look into getting some professional services involved. Either directly from Progress or from some of the independent organizations that can help you out...

 

[Dan4gl]

We have a very large legacy application with 32 bit OCX controls and DLLs that will take a very large effort to replace and test.

OOM = Out of memory exception messages from Windows.

PASOE reference will solve only half the problem.

We have tried to use... I believe it was called the clientstamentcache in promon but created performance issues and there was too much data to sift through to find anything helpful. The challenge we have is finding what is leaking memory and not really DB queries.

 

[Rob Fitzpatrick]

You have said before that you can't enable more detailed logging as it would affect performance. It doesn't have to be all or nothing. One possible approach is to alter the startup configuration of a handful of clients and enable logging for them. If there is no evidence of memory leaks, change their configuration back to default and enable detailed logging for a different subset of users. Sampling in this way, you will eventually find users with memory leaks, and you will be able to see what code they are running. This will tell you where to look in your application to start fixing the code.

If anything, I think moving from App Server to PASOE would make your life worse rather than better. It tends to exacerbate memory-leak problems. You should try to eliminate those problems before you migrate to 12.x.

 

[TomBascom]

It might, none the less, be helpful if you posted an actual error message or 3 or 4.

That kind of memory usage _might_ be a "leak" but it could also be a logic problem where you are just using much more memory than expected. I have, for instance, seen situation where someone was processing XML documents that used far, far more memory than they imagined would be used. Nothing "leaked" per se but they certainly ran out of memory in a hurry.

Memory leaks usually have the benefit of occurring somewhat slowly over time. That kind of thing can be traced externally with stuff like the sysinternals tools. Once you have your eyes on a process that it leaking you can start to narrow down the cause.

You are correct that CSC won't show memory leaks. But it does give you a lot of insight into who is doing what which can very often get you into the neighborhood of where things are going wrong. But, ok, you'd rather not do that. So your next best bet is to start with some client logging. As Rob says, that doesn't need to be all or nothing.