Dumping with Multiple Threads

ron

Member
RHEL 9.2 Linux + OE 12.8.3

To recover a lot of "deleted" space after purging, I face having to reclaim the storage on a few large databases (1TB to 2TB), and of course the down-time needed is a major factor. I am looking at various ways to tackle this, one being a D/L. I have done many D/Ls before, but not with DBs of this size and if a D/L is finally chosen as the way to do it, dumping using multiple threads would be especially helpful ... but I have not used this functionality before.

I understand the way it works: splitting tables into ranges of rowids for each thread. My questions are:

Is there a way to know when any particular thread is finished (ie, available for loading)?
Is there any better way to load-back the dump files other than running each of them separately in parallel?

Ron.
 
I've never done it that way. I've always run different tables in each thread. Essentially you decide how many threads you want, and then split up the batch files so each one is essentially doing a similar amount of work. I just used record count, but you could also add mean record size to the equation I suppose. You want File1 to start with a big table and get smaller, and you want File2 to start small and get bigger. File3 starts big, File4 starts small, etc. If the script then moves each dump file to a new location you know that any files in that location are ready for loading.
Start the new DB without integrity checking and have all indexes deactivated. If you screw up you're starting again, but it'll be orders of magnitude faster than doing it with integrity on.
Once data is loaded restart with Integrity enabled and build all the indexes. In 12.8 there's lots of switches for improving this performance as I'm sure you're aware.
 
Yes, I have done D/Ls previously like that with several parallel dumping streams (likewise for loading) -- but the ultimate result was that the end-to-end time for the full process is (more or less) the time to dump then reload the very largest table. And that's why I'm looking to be able to carve-up the biggest handful of tables using dump streams.
 
Hi Ron,
Depending on your system , you will be able or not to use the following trick.
I dump my "big" tables in a filesystem which use an "old" jfs ( limited to 2Gb) .
So each time the dump file reach the 2Gb, it creates automatically a new dump file .
So i am able to dump and load the same table at the same time (dumping in file number X, loading the file X-1).
With this trick the load ends few time after the dump.

If you are able to have more space for few time,
There is also the "online dump-load" of Mike Furgal based on Tablepartitioning .
This allow you to "tablemove" (on line with some activity on it) a table in another partition which could be in another area .
At the end of the process, you should have no table/partition in the old area so you could remove them.

kind regards
Patrice
 
Of course the optimum strategy will depend on the profile of the database ... how many big tables? big records, etc.

FWIW, I have done a dump by recid ranges ... a painful experience with hardware damage and inadequate backups, but I did manage to recover something like 97% of the data.
 
Max number of the dump threads is defined by the number of the keys in the root block of the index used for dump. In other words, the number of dump threads depends on the chosen index. The larger table more likely its root blocks have just the small numbers of index keys.

Dbanalys does not report the number of the keys per the root blocks. One of the methods is to dump the root blocks. I have the script that can dump the blocks even on online database.

You can force Progress to create the multi-volume dumps: create a copy of _proutil without the setuserid bit. The executable will use the current ulimit. Progress will switch to the new dump file when the current one reaches ulimit. You can load a dump volume as soon as you see its successor.
 
Take care of LOB if you use the ulimit .

Defect # OCTA-31889 : "Binary dump fails with errors (9201) and (14624) when ulimit hit on AIX." (Status : Ready To Develop ) when dumping a table with a Clob or Blob.
 
The threaded dump can be very effective but, as George mentions, it will depend on your indexes. And you might benefit from doing an idxcompact first to clean up your post purge situation.
 
Thank you -- as always, some very interesting and helpful comments.

I confess that I don't understand George's remarks about indexes. A multi-thread dump does't use any index (as far as I know) -- it just deals with a block of records within separate ranges of recid ... or have I missed something?

I have found that I can know when each thread has compled dumping -- it is reported in the log
 
Ron,

Are you thinking of “dump specified” where you supply the range for each “thread”? That’s a very different beast from using the -threads proutil option.
 
No, Tom, I am only testing using -thread 1. I wasn't aware that a range can be specified -- and I don't see why I would consider that. From (limited) testing using 4 threads it looks like Progress divides-up the table into reasonably equal numbers of records.
 
I confess that I don't understand George's remarks about indexes. A multi-thread dump does't use any index (as far as I know) -- it just deals with a block of records within separate ranges of recid ... or have I missed something?
Binary dump does use an index. That is how it finds the records and determines their order in the dump file(s), and thus in the resulting table in the target. You use the -index n option to specify the dump index.

1757164484235.png

Source: https://docs.progress.com/bundle/openedge-database-management/page/PROUTIL-DUMP-qualifier.html

Example: dumping customer from sports2020 via the primary index (CustNum):

$ proutil s2k20 -C dump customer dump -thread 1
OpenEdge Release 12.7 as of Fri Apr 21 08:45:41 EDT 2023

Threads number is not specified. Maximum number of threads running is 2. (14774)
Using index CustNum (13) for dump of table customer. (17813)
Thread 140132717238592 dumped 222 records for bracket 2. (17816)
Thread 140132718250304 dumped 895 records for bracket 1. (17816)
Binary dump started 2 threads. (14776)
Dumped 1117 records. (13932)
Binary Dump complete. (6254)

This explains George's comment on how the Progress code chooses the thread count. The idxblockreport provides a textual depiction of the index B-tree. We can see that for the index Customer.CustNum, the root block in the tree contains two entries:

Code:
$ proutil s2k20 -C idxblockreport customer.custnum
OpenEdge Release 12.7 as of Fri Apr 21 08:45:41 EDT 2023


BlockSize = 8192  Block Capacity = 8100
                Number  Length  On      Length  Delete
                of      of      Delete  of      Chain           Percent
DBKEY   Level   Entries Entries Chain   Size    Type            Utilized
1664    1       2       21      0       0       root            0
1728    2       896     8092    0       0       leaf            99
1696    2       222     2010    0       0       leaf            24
Index Block Report completed successfully.

The entries in the root block, dbkey 1664, point to the two leaf blocks with dbkeys 1728 and 1696. So one thread dumps records pointed to by the leaf-level entries in the left side of the tree, and the other thread dumps the records for the right side of the tree. George's point is that the B-tree can have more than two paths from the root block, so a binary dump with -thread 1 might choose a different number of threads, depending on how many blocks there are at the second level, i.e. how many entries are in the root block (the first level).

Note also that in this case, the threads did not do equal work. One dumped 222 records and the other dumped 895 records.

Side note: leaf blocks are the blocks in the bottom level of the B-tree. They contain the pointers to the records. In the example above, the index had only two levels: the root block and the leaf level, because this is a very small table. Large tables will have more levels (typically three to five) in their B-trees. The number of B-tree levels in an index is shown in the Index Block Summary of proutl dbanalys/idxanalys:

Code:
Table                      Index  Fields Levels         Blocks    Size  % Util  Factor
PUB.Customer
  CustNum                     13       1      2              3    9.9K    41.3     2.2

For dumping without an index, you may be thinking about -index 0. This follows the table's cluster chain to find the records in their per-cluster physical order, so it is only possible for a table in a Type 2 area. But a consequence is that the dump order of the records is indeterminate, which may not be desirable.

$ proutil s2k20 -C dump customer dump -index 0
OpenEdge Release 12.7 as of Fri Apr 21 08:45:41 EDT 2023

Performing table scan for dump of table customer. (14653)
Dumped 1117 records. (13932)
Binary Dump complete. (6254)

A simple visual depiction of a B-tree:
1757165177618.png

Note that in this B-tree, the root block (19) has two paths: to block 7 and its child nodes, on the left; and to block 37 and its child nodes, on the right.
Source:

Again, this example shows two child nodes below the root, but the kind of B-tree used in Progress indexes (Prefix B+ tree, I think?) can have multiple children below a parent node.
 

Attachments

  • 1757163410402.png
    1757163410402.png
    12.2 KB · Views: 2
“Dump specified” predates threaded dumps. It allows you do do a partial dump (perhaps the current year if you have a date field indexed) or, pre -thread, break the data into groups
 
If you after ultimate speed and “logical scatter” isn’t a big factor for you then index 0 can be wicked fast.
 
The original problem was stated as "recover a lot of "deleted" space after purging". And you have decided that the correct solution is purge/dump & load; specifically multi-threaded D&L.

Why are you purging? Is it because you don't have the storage capacity you need? Is it because you are writing a lot of transient data that only has short-term value?

One could also ask: do you need to retain all of the historic data that is currently in your database? Are you retaining it to comply with corporate policy, or to meet regulatory or legal requirements? Or because it is all required for the application and business to run properly? Or because it gives someone a warm and fuzzy feeling that they will be able to help, on the off chance that someone asks them to run a report on very old data? Or did the data just grow organically because no one thought to institute data-retention policies in the early days?

And if your database is just going to regrow back to the size it was before the purge, then isn't that size your new normal? What is the benefit of recovering the space? Is it worth the time and money you are spending regularly to recover that space temporarily? Could that money be better spent; say, to buy more storage? Or to implement a completely different strategy?

Other options to consider:

Table reorg
Code:
proutil <db-name> -C tablereorg [<owner-name>.]<table-name>
     [ info ] |
     [ [ resume | resume-numrecs <n> | resume-rowid <n> ] | [ nosmartscan ]
       [ restrict [ EQ <value> ] |
                  [ LT | LE <high-value> ] |
                  [ GT | GE <low-value> [ AND LT | LE <high-value> ] ] ]
       [ useindex <index-name> ] [ recs <n> ]
       [ searchdepth <n> ] [ reusepercent <n> ] ]
     [ tenant <tenant-name> | group <group-name> |
       partition <partition-name> | composite initial ]

Table partitioning
https://www.progress.com/resources/papers/improve-performance-for-increased-concurrency
Table Partitioning for Improved Availability and Manageability

Table Partitioning is an RDBMS add-on product. But if you already have the Advanced Enterprise Edition (AEE) RDBMS bundle then there is nothing extra for you to buy. With Table Partitioning, you can define policies for large, fast-growing tables. The table defines a partitioning key; in your case, you might want to use a date field. That policy determines which partition of a table each new record is written to. So for example, you could write your 2025 data in a large table to one partition, your 2026 data to another, and so on. Each partition is a separate storage object, and can so can be written to a different storage area if desired. And when that data ages out, according to your data-retention policy, you can remove the partition in one operation, rather than programmatically (FOR EACH... DELETE).
 
If we stay with D&L… There are the reasons not to use the multi-threaded load. The load will be the longest phase. Then the best choice, IMHO, is to dump and load in parallel using the multi-volume dump. Bonus - we will use the less disc space.
 
Back
Top