Index Rebuild Perfomance

anoobbhaskar

New Member
Hi,

I want to perform an index rebuild on 70GB data base in HP Unix Machine with 10.2B05 ITANIUM Loaded. The server has 4 core CPU and 16 GB Memory now. Currently we are specifying the parameters such as -TB as 31 , -TM as 32 , -B 100000 in the idxbuild command. Could you please tell me the parameters are optimum or still we can increase to get better performance on indexrebuild. I heard that service pack 05 has some advanatge on index rebuild where we can specify upto 62 for -TB. We are planning to increase our Memory (32GB) and Cpu (8 Core). How do I know that what is the optimum/maximum value for these parameters?

Thanks in Advance
 
You didn't mention which RDBMS license you have. If you you have Enterprise you can take advantage of the multi-threaded index rebuild (this is the default behaviour).

As you said, SP05 introduced some new tuning options for index rebuilds; they are in the release notes, but not in the product docs. The maximum for -TB (sort block size) has been increased from 31 to 64.

There is a new -TF (memory usage factor) parameter, which lets you specify the percentage of system memory used for sorting index keys. The default is 50 (i.e. 50%), and 0 is the value to get pre-SP05 behaviour. I haven't done much testing with this yet, so I can't make a recommendation, other than to do your own testing before adjusting it.

There is also a new -rusage parameter; this causes the rebuild process to report resource-usage statistics like disk I/O read/write and CPU utilization.

The guidance I have been given on -B for an index rebuild is not to use a very large setting as you might be tempted to in production. Test with a smaller value like 10000 to 20000.

You could try adjusting -SG (sort groups); the default is 48 and max is 64. Note that a larger value uses more memory and file handles.

I specify -pfactor 80 (index packing factor) in index rebuilds. This is the target percentage that the rebuild will shoot for when filling index blocks with keys. The default is 100%. Using 80 leaves empty space in blocks for the future addition of index keys. If you use 100 your blocks will be more tightly packed, but key insertions will more often result in block splits which are expensive.

If you can, specify a temp directory (-T <directory>) that is on a different disk from your database. This will reduce I/O contention. Alternatively you can create a sort file (dbname.srt) containing multiple directory names and sizes to be used for the disk sort. See the Database Admin manual page 14-46 (Maintaining and Monitoring Your Database | Managing Performance | Index Use | Rebuilding indexes) for more details. Also see Reference | PROUTIL utility | PROUTIL IDXBUILD qualifier in the same manual for more details.

When the idxbuild asks if you have enough disk space, if you say "yes" (assuming you do) you'll get better performance from a disk sort than from a memory sort (saying "no"). Counter-intuitive, but that is the guidance I have received.

In answer to your question "How do I know that what is the optimum/maximum value for these parameters?": maximums are easy; they are documented. Optimum values are specific to your database on your server and storage hardware, with your OS. What works well for me may be non-optimal for you. All I can advise is to test with some baseline recommended settings, and document the results. Then tweak a setting, re-test, document the results; lather, rinse, repeat. It is valuable to watch top/htop/topas/sar (or equivalent, I'm not familiar with HP-UX) while the rebuild is running to get a sense for OS resource usage and how it is affected by parameter values. I like htop as it can show per-process I/O read/write/total as well as CPU utilization.

Lastly, and this is probably common sense but bears repeating: make sure you have a backup of your database after your last shutdown, before you start your index rebuild. If it crashes, you may have to restore from backup.

There is some more information in this KB article: What performance gain using idxbuild multi-threads?

I'm sure Tom will weigh in soon with some valuable perspective as well.
 
The new features in 10.2B05 have been reported to be troublesome. 10.2B06 is said to have improvements on those issues. I would upgrade first.
 
You didn't mention which RDBMS license you have. If you you have Enterprise you can take advantage of the multi-threaded index rebuild (this is the default behaviour).

As you said, SP05 introduced some new tuning options for index rebuilds; they are in the release notes, but not in the product docs. The maximum for -TB (sort block size) has been increased from 31 to 64.

There is a new -TF (memory usage factor) parameter, which lets you specify the percentage of system memory used for sorting index keys. The default is 50 (i.e. 50%), and 0 is the value to get pre-SP05 behaviour. I haven't done much testing with this yet, so I can't make a recommendation, other than to do your own testing before adjusting it.

There is also a new -rusage parameter; this causes the rebuild process to report resource-usage statistics like disk I/O read/write and CPU utilization.

The guidance I have been given on -B for an index rebuild is not to use a very large setting as you might be tempted to in production. Test with a smaller value like 10000 to 20000.

You could try adjusting -SG (sort groups); the default is 48 and max is 64. Note that a larger value uses more memory and file handles.

I specify -pfactor 80 (index packing factor) in index rebuilds. This is the target percentage that the rebuild will shoot for when filling index blocks with keys. The default is 100%. Using 80 leaves empty space in blocks for the future addition of index keys. If you use 100 your blocks will be more tightly packed, but key insertions will more often result in block splits which are expensive.

If you can, specify a temp directory (-T <directory>) that is on a different disk from your database. This will reduce I/O contention. Alternatively you can create a sort file (dbname.srt) containing multiple directory names and sizes to be used for the disk sort. See the Database Admin manual page 14-46 (Maintaining and Monitoring Your Database | Managing Performance | Index Use | Rebuilding indexes) for more details. Also see Reference | PROUTIL utility | PROUTIL IDXBUILD qualifier in the same manual for more details.

When the idxbuild asks if you have enough disk space, if you say "yes" (assuming you do) you'll get better performance from a disk sort than from a memory sort (saying "no"). Counter-intuitive, but that is the guidance I have received.

In answer to your question "How do I know that what is the optimum/maximum value for these parameters?": maximums are easy; they are documented. Optimum values are specific to your database on your server and storage hardware, with your OS. What works well for me may be non-optimal for you. All I can advise is to test with some baseline recommended settings, and document the results. Then tweak a setting, re-test, document the results; lather, rinse, repeat. It is valuable to watch top/htop/topas/sar (or equivalent, I'm not familiar with HP-UX) while the rebuild is running to get a sense for OS resource usage and how it is affected by parameter values. I like htop as it can show per-process I/O read/write/total as well as CPU utilization.

Lastly, and this is probably common sense but bears repeating: make sure you have a backup of your database after your last shutdown, before you start your index rebuild. If it crashes, you may have to restore from backup.

There is some more information in this KB article: What performance gain using idxbuild multi-threads?

I'm sure Tom will weigh in soon with some valuable perspective as well.


What could be the possible reason for index rebuild fail/DB crash due to index rebuild?
 
First, if an index rebuild crashes the database is not usable until you successfully complete the index rebuild. Usually, when it crashes you solve the issue and just start it again.

Some common issues:


  • Not enough disk space for sorting.
  • Power loss.
  • Somebody kills the process because she/he thinks it is not doing anything ...


Heavy Regards, RealHeavyDude.
 
Back
Top