Question New Server Spec

[TheMadDBA]

Most (if not all) of the Dell systems are going to be Numa once you get past 8 cores. Depending on your workload you might be better off having one server for the DB and one or more for the appservers to avoid the numa hell.

 

[Cringer]

Hmmm this is an issue as we're seriously strapped for rack space.

 

[TheMadDBA]

That doesn't sound good

When somebody complains about their new system being slower than they expected (or slower than the old one)... my first question used to be what the startup parameters are and how they compared to the old box, then more questions about the setup. Now my first question is whether the new box is NUMA or not.

All that being said... it can "work" depending on...

- How many reads you do and how big your -B is going to be
- How much faster the new cores/disks are than the old system
- What your current bottlenecks are

But it is more brute force of newer hardware (that is running slower than it should) compared to ancient hardware.

 

[Cringer]

It's the unfortunate side-effect of a move to managed/shared office space. We used to have our own building and two decent server rooms with as much rack space as was necessary. We now have a caged area in the single server room with just enough rack space for what we need. Very little for expansion though. The .Net guys are already bagsying the empty space as their requirements are growing rather rapidly too.
Ah well - report has been submitted to management. See what they come up with.

 

[Rob Fitzpatrick]

Hmmm this is an issue as we're seriously strapped for rack space.
We now have a caged area in the single server room with just enough rack space for what we need.

If you have legitimate requests for rack space being turned down then you don't really have just enough space for what you need.

Needs change. And whoever doles out the space needs to recognize that.

I can't see management authorizing budget for new servers and OS licenses but refusing you a place to put the new kit.

 

[TheMadDBA]

There is always the option to use vmware to split the physical box into multiple logical boxes that don't span NUMA areas. It isn't easy to get right though and there are many horror stories of people doing it wrong.

With all of the management issues you are up against it might be better to take a step back and re-evaluate what your real performance issues are. Sounds like it isn't going to be easy to keep buying hardware and you need to nail it the first time.

 

[cj_brandt]

Windows Perfmon is able to track disk stats now without setting some flag and rebooting the server. I would highly recommend tracking the disk queue on the storage system with perfmon.
Avoiding NUMA architecture is hard to do for large servers. I was recently bit by NUMA, the effects were reduced by lowering the memory buffers - in our case from 160gb down to around 90gb. I guess it is similar to raid 5 popping up in larger disk systems.
I

 

[Cringer]

I'd tend to agree. In an ideal world I'd want to upgrade the current box to 64 bit OS and add in 64 bit Progress. Then tune that up as much as we can, leaving us more of an idea of where the bottlenecks are. The trouble is it seems like a nightmare to get downtime now because it has to be done out of office hours, and management don't seem to be prepared to pay overtime for it. Obviously I'm not going to do it all for free.

 

[Rob Fitzpatrick]

So what is the business driver for the change? Is it a change you want to make personally? Are users complaining about application performance? Is management complaining about efficiency?

 

[Cringer]

Yes users are complaining about performance, and so they should be. The other driver for it is that the server has 3TB of physical disk (effectively 1.5TB due to RAID 10). The disk enclosure is at maximum capacity meaning we can't add direct attached disks and as far as I know we can't get bigger disks easily. We'll be out of space on the server by the end of the year at current growth rates. Therefore we need a new server that is able to service bigger disks and SSDs.

 

[Rob Fitzpatrick]

Well, if management isn't prepared to pay for after-hours work to be done to get onto a new server and keep the business moving forward, something has to give eventually. That will be a business-hours outage (bad) or DB disks fill to capacity and you crash (bad) or management relents and plans a server migration to avert business disruption.

Doesn't take an MBA to figure out the right path.

 

[Rob Fitzpatrick]

Just to be clear Cringer, that last dig was directed at your management, not you. I'm sorry if it came across otherwise. I was a little cranky this morning.

 

[Cringer]

No worries Rob - completely taken the right way.

 

[Cringer]

Well you probably saw this coming. Apparently we have ordered servers for all the different applications (not all Progress) that we have. It has been decided that for ease they will all be identical in terms of spec, will be virtualised, and the main storage will be on a SAN. So pretty much every recommendation has been ignored. The only saving grace is that they will only have 2 multi-core processors so NUMA is out of the question.
I really give up.

 

[Rob Fitzpatrick]

All is not lost yet. Fight for RAID 10 on the SAN for your DBs. Disks are dirt-cheap, especially if you're not buying SSDs, so there shouldn't be a compelling reason to go with RAID 5. How large are your Progress DBs?

 

[Cringer]

I think the plan is for a flash based SAN. And yes it will be RAID10.
Our main DB is ~350GB at the moment and growing. Once we have a chunk of contiguous disk space a dump and load should bring that down significantly as we have some tables in a really wrong RPB configuration.

 

[Rob Fitzpatrick]

If I recall correctly you had LOBs in your DB as well, and were thinking of eliminating them. Is that correct?

 

[Cringer]

Would love to eliminate them, but we can't agree on a strategy to do so. The main issue with them is though that they're not in their own area and they're in a massive data area with 64 RPB. My plan is to at least zip them up before we store them. Just got to implement that.

 

[Rob Fitzpatrick]

Once we have a chunk of contiguous disk space a dump and load should bring that down significantly as we have some tables in a really wrong RPB configuration.

If "wrong" means too low then that will definitely help. I saw a case with a customer DB where a table with many small records was put in the wrong area, so the RPB was too low. Dumping and loading to an area with appropriate RPB gave us a 5x reduction in disk usage for the same data. The change also considerably reduced physical I/O and improved cache efficiency when reading that data, and reduced memory pressure on other data, all of which improve application performance.

A server migration is a great time to dump and load. And if the new box is much faster than the old, you can use it for both the dump and the load.

 

[Cringer]

So the new servers and SAN are being installed and configured by our vendor at the moment. I had the chance to chat with him yesterday about the architecture that's being pushed on us and it doesn't sound too bad.
The servers have a BIOS switch to turn off NUMA (bonus!). I've made sure that this is going to be turned off.
He has experience of Progress applications (he has no Progress knowledge, but has clients who use Progress and they're happy). A positive at least.
As far as I understand it, the SAN where the DBs will be stored writes at RAID10 and reads RAID5. Will be interesting to see how that pans out in practise.
Managed to persuade him that VMware snapshots are not a valid backup strategy. Phew.
We've tested our application in a 64 bit Server 2012 environment and it works fine.
Heading int eh right direction at least!

Are there any gotchas with vmware that I should be wary of?