[Progress Communities] [Progress OpenEdge ABL] Forum Post: RE: Bulk processing taking off slow on a restart

  • Thread starter Thread starter Valeriy Bashkatov
  • Start date Start date
Status
Not open for further replies.
V

Valeriy Bashkatov

Guest
Hi Dmitry, This is because in older versions, when the program is restarted, the loader reads all the records again from scratch and skips them based on the counter. What version of Pro2 are you using? In Pro2v5.0.4, they changed the procedure, as far as I can see. Now the Bulk_Max_Cache property is used to indicate the number of last cached ROWIDs in order to start processing from them. This cache is stored in replcontrol at loading time. For example: for each replcontrol where groupid = "BULKCOPY" no-lock. displ replcontrol.codeval3 format "x(120)" end. ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │Value3 │ │────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│ │Count=400000;RowidCache=0x00000000000d372d,0x00000000000d372c,0x00000000000d372b,0x00000000000d372a,0x00000000000d3729,0│ I also use Pentaho Data Integration solutions instead of the standard Bulk-loading. This solution also appeared in Pro2v5.0.4. With it, the primary synchronization speed is increased many times over. For comparison, a table with 10 000 000 records via Pro2 Bulk-Loading is loaded in 13-16 minutes. But with PDI it takes only about ~500 seconds.

Continue reading...
 
Status
Not open for further replies.
Back
Top