Swap space?

[ron]

Thanks Rob. I'd already seen that one (and several others, by the way).

Allocating swap doesn't mean that anything actually "gets" swapped -- it's just that the space in swap maps the whole virtual memory space so every process has to "own" a section of swap. But actual swapping would only happen when there is not enough physical memory to hold all currently active processes.

Yes -- as you say -- swapping doesn't happen at the level of a process, but at the level of a page -- but I was glossing over that. I was focusing on the essential bit.

I still think all running processes have to be mapped onto a block of virtual memory -- and that's in swap.

(Unfortunately all the articles I can find that try to explain virtual memory jump straight into so much detailed blaa that it's hard to get the gist of what they're trying to explain. )

 

[TomBascom]

Ron has described the old style pessimistic swap reservation algorithm. To a degree this still used on Solaris, HPUX, and AIX. But Linux uses an “optimistic” reservation algorithm that assumes you won’t ever actually need much of that disk space. Linux is correct, you won’t.

Rob’s points about paging vs swapping are also very much on the money.

(To make things even more fun the IO subsystem also operates via the paging mechanism.)

You can, sometimes, make problems temporarily go away by increasing swap space but in my experience problems that present themselves as running out of swap usually have a root cause that is better addressed in another way. So, in a way, I agree with the SysAdmin. Although I very much disagree that -B is the root cause.

Getting to the bottom of these sorts of issues tends to take a lot of very careful reading and analysis. You might start by verifying that all of the running processes are using only as much (private) memory as they should be and that what is running should be running.

I would also carefully check that nothing huge is being written to any tmpfs file systems. That could include invisible Progress temp files if you’re pointing -T at such a file system without -t.

 

[TomBascom]

Are there lots of Java processes running? Those are also famous memory hogs.

 

[Rob Fitzpatrick]

You could also check the kernel's swappiness setting to see if someone set it to be too aggressive.

 

[TomBascom]

There is a good rabbit hole to dive down here: Linux: find out what process is using all the RAM?

 

[ron]

Thanks Rob and Tom!

I will keep researching, although it's hard to spare the time researching problems on a Dev system when I have many Production issue to keep me busy. The OOM-killer always targets two brokers when it hits -- the broker for the TEST and UAT environments and affects quite a few people ... so I can't ignore it.

 

[TomBascom]

This is a very useful tool:

Install and use smem–Linux Memory Reporting/Statistics Tool | LinTut

smem is a tool that can give numerous reports on memory usage on Linux systems. In this article, we will show you how to install and use smem memory reporting tool.

lintut.com

The "USS" column is "Unique Set Size" - IOW, memory use *without* counting any shared bits.

 

[Rob Fitzpatrick]

Wow, smem is new to me. Great tool!