You didn't mention which RDBMS license you have. If you you have Enterprise you can take advantage of the multi-threaded index rebuild (this is the default behaviour).
As you said, SP05 introduced some new tuning options for index rebuilds; they are in the release notes, but not in the product docs. The maximum for -TB (sort block size) has been increased from 31 to 64.
There is a new -TF (memory usage factor) parameter, which lets you specify the percentage of system memory used for sorting index keys. The default is 50 (i.e. 50%), and 0 is the value to get pre-SP05 behaviour. I haven't done much testing with this yet, so I can't make a recommendation, other than to do your own testing before adjusting it.
There is also a new -rusage parameter; this causes the rebuild process to report resource-usage statistics like disk I/O read/write and CPU utilization.
The guidance I have been given on -B for an index rebuild is not to use a very large setting as you might be tempted to in production. Test with a smaller value like 10000 to 20000.
You could try adjusting -SG (sort groups); the default is 48 and max is 64. Note that a larger value uses more memory and file handles.
I specify -pfactor 80 (index packing factor) in index rebuilds. This is the target percentage that the rebuild will shoot for when filling index blocks with keys. The default is 100%. Using 80 leaves empty space in blocks for the future addition of index keys. If you use 100 your blocks will be more tightly packed, but key insertions will more often result in block splits which are expensive.
If you can, specify a temp directory (-T <directory>) that is on a different disk from your database. This will reduce I/O contention. Alternatively you can create a sort file (dbname.srt) containing multiple directory names and sizes to be used for the disk sort. See the Database Admin manual page 14-46 (Maintaining and Monitoring Your Database | Managing Performance | Index Use | Rebuilding indexes) for more details. Also see Reference | PROUTIL utility | PROUTIL IDXBUILD qualifier in the same manual for more details.
When the idxbuild asks if you have enough disk space, if you say "yes" (assuming you do) you'll get better performance from a disk sort than from a memory sort (saying "no"). Counter-intuitive, but that is the guidance I have received.
In answer to your question "How do I know that what is the optimum/maximum value for these parameters?": maximums are easy; they are documented. Optimum values are specific to your database on your server and storage hardware, with your OS. What works well for me may be non-optimal for you. All I can advise is to test with some baseline recommended settings, and document the results. Then tweak a setting, re-test, document the results; lather, rinse, repeat. It is valuable to watch top/htop/topas/sar (or equivalent, I'm not familiar with HP-UX) while the rebuild is running to get a sense for OS resource usage and how it is affected by parameter values. I like htop as it can show per-process I/O read/write/total as well as CPU utilization.
Lastly, and this is probably common sense but bears repeating: make sure you have a backup of your database after your last shutdown, before you start your index rebuild. If it crashes, you may have to restore from backup.
There is some more information in this KB article:
What performance gain using idxbuild multi-threads?
I'm sure Tom will weigh in soon with some valuable perspective as well.