AI Circular

lab7

New Member
I would like to set up files (AI) circular one for our base mfpgro. If somebody can help me

Thanks


 

lab7

New Member
hye,
We've currently a base mfgpro, which turns on 1 file (AI), and if this file exceeds the 2Go the base stop. I believe that with mfgpro we can make several files (AI) and to safeguard them on another disk or tape in order to ensure the intégriter base !!!
Thanks to help me for my project !!!
 

TomBascom

Curmudgeon
What version of Progress? What operating system?

This is all covered in more detail in the database administration guides but in brief here is what you need to do:

1) You need to add AI extents to your database. I suggest that you have at least 4 AI extents. If you're running a relatively modern release of Progress (version 9 or better) then you should use variable length AI extents.

2) You need a method to switch extents and archive old extents. One good way to do this is with "cron" jobs. A generally reasonable method is to have one job mark force an extent switch every 15 minutes and have a second job sweep full extents into an archive directoy one or two minutes later. (Adjust "15 minutes" to suit your needs.)

To force an extent switch:

$DLC/bin/_rfutil $DBNAME -C aimage new

Archiving is trickier (in 10.1A it is automated). To archive you need to loop through the list of full extents, copy & rename them with OS commands and then mark the originals as being available. You can get a list of full extents with:

$DLC/bin/_rfutil $DBNAME -C aimage extent list

(You will need to parse the output for "FULL".)

Once the full extents have been copied and renamed you mark them as being available again with:

$DLC/bin/_rfutil ${DBNAME} -C aimage extent empty $EXTENT

3) You should document and regularly test the recovery process. You don't want to be figuring out how to recover for the first time when you're under pressure.
 

lab7

New Member
Thanks

- version progress 8.5E
- OS: Unix True64 V5.1
I work with System Administration guide book, and your recommandation

Best regards
 

ron

Member
Tom -- hello!

Curious to understand why you suggested using all variable AI extents for Progress 9 or 10?

We use 9,1D09 on Solaris (250GB DB). When we switched-on AI (in 2003) we did many tests and found variable extents hit performance rather badly -- so we made all extents "fixed".

The downside of fixed, however, is that you chew-up lots of disc space for the AI archive -- and lots of bandwidth if (like us) you have to send them to another site over the network.

But, this problem was solved when the new rfutil "aimage extract" command was provided.

Regards,
Ron.
 

TomBascom

Curmudgeon
1) Because fixed extents chew up lots of disk space and take a lot longer to transfer.

2) Because it is simpler to implement time based extent switching with variable extents. (You don't necessarily have to cover the case where multiple extents may have filled in the interval.)

It may be that I'm lucky in terms of the disk subsystems that I work with (although I prefer to think of that as good planning on my part ;) ) but I have not found a situation where there was an appreciable performance drag caused by variable AI extents.
 

refurbished

New Member
Tom,

Just to solidify my understanding...so when your cron job runs to archive an AI extent, does it just copy the oldest extent or can you set some upper boundry on variable extents which allows you to check for "FULL"? I thought variable extent meant that it will fill up to the size of the disk. Just want to clarify how those variable extents are being checked for archiving, etc. And I'm not a guru on AI by any means, so thank you for bearing with me on this.

Thanks!
Adam
 

TomBascom

Curmudgeon
In a nutshell my time based AI process is:

1) Mark the current extent full.

2) Archive all full extents.

3) Mark all full extents empty.
 

TomBascom

Curmudgeon
I'm a bit dense this morning...

The key to all of this is to create a shell script to parse the output of:

Code:
rfutil dbname -C aimage extent list
 

refurbished

New Member
OK more half stupid questions for the newbie admins out there (Me). Please bear with me here as this is so far the most informative AI thread on the board atm.

1. In this thread you mention 2 ways to call rfutil, "_rfutil" and "rfutil". Are these one in the same (ie: they do the same thing) or does it depend on what you're trying to do as to which of the two you call?

2. Also, I never really got a grasp on the marking an extent full that is variable size. Is there some way to set this upper boundry on the size of the extent so it is marked automagically or is there a command to mark them as full?

3. If you mark an extent as full using a command (while in your archive script) and archive it while using another extent, why do you suggest using 4 extents? Wouldn't the same method work with 2 extents?

Thanks for all this help Tom, I'm sure several of the anonymous readers will benefit from it as well.
 

TomBascom

Curmudgeon
1) _rfutil is the executable which the script rfutil calls. The script is just a wrapper that doesn't (IMHO) add a whole lot of value so I often call the executable directly. (Much of $DLC/bin works this way).

2) A variable extent has no explicit size limit. It is full when you say "enough" via "rfutil dbname -C aimage new" (or when the OS says "no more!").

3) The number of extents needed is up to you. I suggest a minimum of 4 extents. 1 is busy, 1 is available to be used at the drop of a hat (or when an extent switch is forced for some reason, like an online backup, before the full extent has been returned to service) and 1 is (occasionally) full and being archived. The 4th one makes it an even number which allows the extents to be arranged on 2 disks and the busy and full extents never contend for IO.

I actually implement however many I need in order to protect myself from various mishaps. For instance -- if you're doing extent switches every 15 minutes then 4 extents gives you an hour to notice that something has gone wrong before that 4th extent starts filling beyond its usual size. Personally I think an hour is a bit optimistic so I usually shoot for at least 3 hours of padding (14 extents).

Yes, you could use a smaller number of extents and let them get very big. But I'd rather have more extents so that I can try various strategies to fix a problem if one occurs. Since many of those strategies will result in a need to switch extents it is helpful to have some empty extents available.

A related configuration strategy is how many backups and archived ai files should be kept. I like to have enough ready to hand (IOW on disk) to be able to recover from multiple bad backups -- at least a weeks worth and ideally a months worth.

And I like to have them on more than one server. At least one of which is in a different building and better yet a different time zone.

When things go wrong lots of things go wrong... tapes turn out to be unreadable, disk drives and controllers go sour at just the wrong moment, servers go belly up all at once. And so on and so forth. Having multiple paths for recovery has turned out to be a life saver more than once.
 

ron

Member
Tom - I agree with almost everything you have said. My only qualm is about using variable AI extents. As I mentioned before, we did a number of performance tests and found that variable extents had a major performance impact. But, we also faced the problem that if we used fixed -- the files were far too big and sending them over the network was a problem.

For about 18 months we "worked around" this by having 50 "small" fixed AI extents -- and switching every hour.

But, when Progress released 9.1D09 -- our prayers were answered. They introduced a new variant of rfutil: "aimage extract". That allows all extents to be fixed (good performance) -- but with no problem at all with the fixed being excessively large.

Now we only have 10 "big" AI fixed extents (500 MB) -- and switch files every ten minutes. It works very well indeed.

I have to say that now that Progress have solved the extent size problem -- I would never recommend using variable extents. They offer no advantage at all -- and one big disadvantage: poor performance.

Regards,
Ron.
 

TomBascom

Curmudgeon
I've heard about it but I have never actually seen poor performance with variable extents. While I'm sure that it happens for some people it is certainly not a universal phenomena. I'd like to understand better the circumstances under which poor performance occurs. I'd also like to see some hard data on what the impact is when it occurs.

I've had very good results using variable extents on many different configurations of gear. The flexibility of variable extents makes them my preferred choice. Of course if I ever do run into poor performance that can be laid at the feet of the variable extents I'll be sure to change that ;)
 

ron

Member
FIXED vs VARIABLE AI EXTENTS.
========================

Hi Tom,

You asked "what" poor performance we had with variable AI extents - so the following is what we found.

First, to explain the environment, we are a water utility company. One very important requirement almost every night is to produce customer bills -- many, many thousands of them. It's rather time critical, because the bill files are sent overnight to a printing company to have them printed and delivered the next day. Since the job of producing bills is "important" -- and involves a lot of update activity -- and we do it almost every day -- we used it as our performance benchmark.

The platform used for all tests was a Sun V480 with 4 x CPU and 16 GB memory. All discs are Ultra160 SCSI. All discs are mirrored and striped via software. The AI file extents are all on a dedicated disc drive (also mirrored).

Over a number of tests, the following are the average results for the number of bills we could process in one minute:

8.3E (no AI) -- 518
9.1D05 (no AI) -- 379
9.1D05 with AI (var ext) -- 119
9.1D06 (no AI) -- 590
9.1D06 with AI (var ext) -- 303
9.1D06 with AI (fixed ext) -- 566

There was a significant performance problem with 9.1D05 under Solaris -- which is why we moved to service pack 06.

With SP 06, as you see, with AI enabled and using fixed AI extents, the performance loss from using AI was about 5% -- which was quite acceptable to us. But with variable extents, the hit was over 45% -- which we just couldn't tolerate.

However, as I mentioned in a prior post, we did NOT want to use fixed extents -- because that would consume a lot of disc space (for the saved files) and would create a much larger data volume for transmission to our remote backup site. We were forced to accept the (considerable) difficulty that the fixed extents imposed -- because that was the lesser of the evils. All we could do is lessen the impact of the bloated files size problem by switching AI files every hour, which was less often than we would like (every ten minutes). Also we created "lots" of small fixed AI extents (50).

We lived with this compromised situation for about 18 months until we discovered that in SP 09 Progress had released a new feature in rfutil which allowed a fixed extent to be copied such that only active data was copied -- thus stopping the "bloated file" problem. We moved to SP 09 and found that performance was not measurably changed but because of the new rfutil "aimage extract" function we could reduce the number of AI extents from 50 to 10 -- but increase to size of each to 500MB -- and switch AI the the next extent every 10 minutes -- with no problems at all.

Our conclusion, from the tests we performed, was that using variable AI extents in a heavy-update environment was shear disaster, from a performance aspect. And, now that the "bloated file" problem is solved there just doesn't seem to be any advantage at all for using variable extents. (Unless, perhaps, one is critically short of disc space.)

Ron.
 

TomBascom

Curmudgeon
Thanks Ron! That's an excellent analysis.

I've not seen similar degradation with any of my customers but that is exactly the kind of thing that I would be looking for and those are the sorts of results that would get me to use fixed extents.

It's a minor point but the main advantage to variable extents that I think you are still overlooking is that they are variable. Unless your fixed exents are filling (or very nearly filling) the filesystem that they are on variable extents provide you with a higher upper limit on how long you can go when unusual things happen.

But I can see why you're doing what you're doing and I certainly agree that it is right for you.

I'm curious -- what sort of filesystem are you using for the AI files? Any special mount options?

Also, do you (or did you) have compression and copy/archive activities going on on the same disks while new notes were being added? Or did you have the alternate disks stratgey going on?
 

ron

Member
Hello, again, Tom.

No special file system options at all -- straight UFS under Solaris.

I wouldn't say that I was "overlooking" the fact that variable extents have the advantage of size-flexibility. I agree that that is "good". It's just that the performance issue dominates.

We do not have alternate AI files on separate drives -- they are all on one disc. I understand the idea behind using two discs -- and we may do that when we next reconfigure. However, we swap every 10 minutes during business hours -- and about every 30 minutes we archive any filled AI extents during batch processing (the fixed extents are 300MB). The total elapsed time for the swap process, from go-to-whoa -- including copying, marking empty and compressing -- is typically 5 to 9 seconds during business hours -- and maximum of 90 seconds during batch work. So, I wouldn't expect a big gain by using alternate discs. Some, though, certainly.

Ron.
 

rbaggetta

New Member
Below is neat script I wrote so that when the AI extent does get rather large, it will perform the switching and verify all extents are marked as empty and one 1 busy. Its a 2 part process. What you need to watch for is if you are also doing online backup's of the database during the day. An online backup will mark the extent as full and automatically switch, so do testing and really know your system if its even possible for both programs to run at the same time...don't really know what would happen if online back started, and programs below ran at the exact same instance. Personally, I don't think its a problem if both ran.:awink:

This part is just a cron job that would run this script every 10 min and see if the active AI image is larger than 750 MG

[FONT=r_ansi]#!/bin/csh
#
# This checks the AI file size and sends an email to cell phone if
# tolerance level is reached
#
setenv JITAI /aijit
cd $JITAI
foreach FFILE (/aijit/jitprod.a*)
set FILE=$FFILE
set FSIZE=`ls -l $FILE |awk '{print $5}'`
set FNAME=`basename $FILE`
if ( $FSIZE > 750000999 ) then
echo "AI file" $FNAME "is" $FSIZE >> $HOME/jitai.lst
echo `date` >> /home/rico/jitai.lst
mailx -s "AI switching in Process" username < $HOME/jitai.lst
/scriptsjit/progress_scripts/JIT_switchAInow
endif
end
if ( -e $HOME/jitai.lst ) then
rm -f $HOME/jitai.lst
endif

====================
2nd phase is to do the actual switching of AI's. Everything is captured to a log file for review.
====================
[FONT=r_ansi]#!/bin/sh
#
swAI_JIT="/scriptsjit/backup_scripts/backup_logs/swAI_JIT."`date +%d%b%I:%M:%S`
echo " "
echo "After Image switch initiated: `date`" >$swAI_JIT
DLC=/apps/dlc91d;export DLC
PATH=$DLC/bin:$PATH;export PATH
PROMSGS=$DLC/promsgs;export PROMSGS
PROTERMCAP=$DLC/protermcap;export PROTERMCAP
#
(
cd /prodjit1
aifile="zzz"
while [ "$aifile" != "Database has no full extents. (3804)" ]
do
aifile=`_rfutil jitprod -C aimage extent full`
sleep 4
if [ "$aifile" != "Database has no full extents. (3804)" ]
then
echo " "
cp $aifile /backupjit/cpjitai
echo " "
echo "Copied AI File:"$aifile "to /backupjit/cpjitai."
rfutil jitprod -C aimage extent empty $aifile
echo "Empty the AI File:"$aifile
fi
done
echo "Checking again for Full AI to copy before emptying it".
echo "Creating a new extent, then copy current to backup folder"
echo "---------------------------------------------------------"
rfutil jitprod -C aimage new
aifile=`_rfutil jitprod -C aimage extent full`
cp $aifile /backupjit/cpjitai
echo "Copied AI File:"$aifile "to /backupjit/cpjitai."
rfutil jitprod -C aimage extent empty $aifile
echo "Empty the AI File:"$aifile
#
if test -e /prodjit1/lastjit
then
rm -f /prodjit1/lastjit
echo "Cleaned up lastjit file as well"
fi
)>>$swAI_JIT 2>&1
[/FONT]
[/FONT]
 
Top