EMC vs. HP or Seagate

[rhi]

I am considering returning to a traditional disk sub-system made up of a bunch of small (18 or 36 gig) HP or Seagate disks instead of continuing to operate with our EMC Symmetrix system.

Our configuration:
HP-UX 11.0
Progress 8.3D (24 gb database, 80 - 300mb exts.)
MFG/PRO 9.0
N4000 with 6 CPU's
4 ultrafast wide SCSI ports
8 gig RAM
400 - 450 users - all local self-serv clients
EMC Symmetrix - 1 gig of cache

Our biggest performance problem is disk I/O.
When running Glance, we consistently see disk I/O between 90% to 100%. User complaints confirm this.

We've had EMC technicians look at their side of the system a half dozen times. They've identified some "slightly hot" areas, and suggested moving a couple of file systems, but stated over all I/O is fairly evenly distributed accross all 4 channels running from the HP server to the EMC box. From the EMC side, nothing is at 100%, and everything is performing very well!!!

The RAM usage of our databases is very close (1.72gig) to the 1.75 gig ceiling of RAM consumption for 32-bit apps on HP-UX. (This was confirmed by an HP performance expert).

My experience:

Before we used EMC for our disk sub-system, we used a bunch of plain old mirrored 18 gig HP disks and stripped the disks while creating one large volume group, and then distributed the database extents evenly across multiple LV's within the large volume group. (BI & AI got their own disks).

Whenever I would do a dump & load (whether dictionary, binary or bulkload) I could complete the whole thing in 24 hours or less.

When we went to EMC, it would take me 48 hours or more to do the same thing, and index rebuilds were excruciatingly painful !!!

Recently, while doing a dump & load, I dumped to, and loaded from, a large vol. group of 6-18 gig HP disks. I also used this same vol group as my sort area for the index rebuilds.

The database that I was dumping from/loading to was on EMC, but most of the heavy I/O was occurring against the HP disks. The whole dump & load time was back down to a little over 24 hours.

Our HP performance consultant stated that he has had other clients complain that dump & loads are MUCH, MUCH slower on EMC vs disks such as HP or Seagate.

Apparent facts:

- Dump & Loads are almost 100% disk I/O.
- Our biggest performance problem is disk I/O.
- Dump & Loads are MUCH faster on "just a buch of disks" than on EMC.
- HP disks are very, very cheap.
- EMC is very, very, expensive.

Conclusion:

I should get rid of EMC and get a bunch of cheap disks (mirrored of course, maybe even 3-ways).

Any thoughts/suggestions/experiences from anyone about this situaiton would be greatly appreciated!!

 

[ChrisPaulson]

How have you got the database set up?

-B
apws
directio

etc


Why do you do so many dump/loads?


The system sounds brill spec so you shouldn't have performance problems.

Does the code use efficient indexes etc? (Old legacy code written by cobol minded people)

 

[rhi]

Here are the startup params for our main database:

-L 250000 -B 93750 -c 250 -n 450 -spin 4000 -bibufs 150 -Mf 25 -H rhin4000 -S hwrusa -N tcp -tablebase 1 -tablelimit 364 -aibufs 225 -bithold 890 -bistall

The dump & loads I am referring to took place over a period of about 5 years. We do about one per year, usually after archiving.

Coding is always a problem, but the performance difference that I notice personally is via dump & loads. Because of this, I can not say for sure that bad coding is our biggest problem. That is why I am investigating dropping EMC in favor of other disk systems.

 

[ChrisPaulson]

-L 250000 -B 93750

Is this the correct why around, if so why only 93 meg to the db when you have soooo much memory?
(Assuming 1K block size as you did not say)

Why sooo many locks?

If this is the right way round you should give loooooads more memory to the DB (like a gig or so more).

Have you set the bi block size and cluster size. Big performance gain can be got from this.

I think 16K block size
8 meg cluster size

This can really help on stopping checkpointing.

 

[rhi]

Sorry - db dblocksize is 8192 - therefore -B =750Mb
Also, have 3 other databases on the system, for a total of
1.4 Gb of RAM allocated via "-B".

BI cluster size is 4096, and BI Blocksize is 8192.

-L is large because QAD writes code that will actually lock that many records.

I also pre-format BI clusters after truncating the BI, so I/O doesn't get spent by users growing the BI during the day.

 

[toby.Harman]

Ah - the joys of EMC.

I am sure they are telling you the truth about the bandwidth in the EMC array, and that you are not fully utilising it.

I wonder whether you are using the available number of I/O operations per second per disk?

The problem I have encountered on EMC is that you cannot see under the hood, and they will not tell you enough or let you reconfigure the disks.

I would recommend getting the EMC out of there as fast as you can. Don't get me wrong - they are GREAT when they are set up correctly. They are a pain the rest of the time.

Your comments on dump&load and Index rebuilds are the most telling. As you say, these are almost pure I/O, and give a very good indication of what you can expect.

If the EMC is there for a while, get the engineer to talk to you about the physical disks (as opposed to the logical disks). I suspect your problem relates to the number of I/O's rather than the volume that are going to each disk. Remember that Progress I/O operations are generally pretty small.

Toby Harman