Replication After Image both server and agent stays Locked status

D

donopena

New Member
Hi I am trying to setup a Replication Proof of Concept on OE 11.7.21. Does anyone know why all the After Image (fix extent) stays in Locked status and not being cleared or Emptied and caused error AIStalled and halted everything. As per Replication documentation After Image supposed to be Emptied once the AI applied to the Target Database.

Source:

]# rfutil dbx1 -C aimage list|grep -i status

Status: Locked

Status: Locked

Status: Locked

Status: Locked

Status: Locked

Status: Busy

Target:

# rfutil dbx1 -C aimage list|grep -i status

Status: Locked

Status: Locked

Status: Locked

Status: Locked

Status: Busy

Status: Empty
 
That would suggest that replication isn't actually working.

rfutil <dbutil> -C monitor

Do that on both source and target as a first instance. They should both say Normal Processing. You should also have Replication related messages in the Source DB logfile.

I'd expect something along these lines.

Code: 
(10436) The source database central and the target database xxx on host yyy are synchronized.
(10508) Beginning Fathom Replication synchronization for the Fathom Replication Agent agent1.
(11251) The Replication Server successfully connected to all of its configured Agents.
 
donopena said:
Does anyone know why all the After Image (fix extent) stays in Locked status and not being cleared or Emptied and caused error AIStalled and halted everything.
Click to expand...
I prefer not to use fixed-length AI extents, for a few reasons. In a case like this where your AI extents are filling up, having fixed extents means you have a fixed amount of time (remaining AI extent space / AI write rate) left to resolve the problem. Whereas with variable-length extents, they can grow and give you time to resolve the issue that is preventing the re-use of your AI extents, without the pressure of a looming deadline.

As @jdpjamesp said, your source and target are likely not in sync. An AI extent in "Locked" status means that it is full and its AI notes have not yet been replicated to all configured targets, so it cannot be emptied and re-used. There could be a variety of reasons for this, e.g.:
  • target is offline;
  • repl agent (target side) is not running;
  • repl server (source side) is not running;
  • network issue;
  • source and target are not in sync.
There are probably other possible causes as well.

Also note that OE 11.7 will retire in April of this year, so you should be working on upgrading to OE 12.8.
 
Thank you for your responses. Can someone confirm. Does the replication really auto emptied the AI extent content once replicated to the target Database (fix extent btw)? Hey Rob yes it does work without AI stalling if I set the AI to variable extent my only concern is, it grow the AI extent to a very large size and doesn't auto truncate. I see that the replication is working by monitoring both source and target with promon option 5, the size of my database is about 1 TB and we are looking to replicate also a 3TB database to serve as Disaster Recovery option.
 
donopena said:
Does the replication really auto emptied the AI extent content once replicated to the target Database (fix extent btw)?
Click to expand...
Once the notes in the Locked extent have been sent to the target (or targets), the database does change its status from Locked to Full. But it does not empty the extent.

You should use the AI File Management Daemon (AIMD), also known as the AI Archiver, to manage the switching and archiving of your AI extents on a regular basis. The only other alternative is to create your own process for doing so, using scripts with rfutil commands, which I do not recommend.

donopena said:
if I set the AI to variable extent my only concern is, it grow the AI extent to a very large size and doesn't auto truncate.
Click to expand...
If you don't monitor what happens with your AI extents, bad things can happen. Aspects of AI extents that you should monitor are their sizes, their statuses, how many extents there are in each status, whether extent switches are happening, whether there is sufficient space in your AI archive directory, etc.

It is important to use AI. OE Replication requires AI. AI necessitates AI monitoring. There is no avoid this. So you need to either develop your own monitoring capability or use an existing solution. [Disclaimer: I work for White Star Software, maker of ProTop. You can check my signature for a link to our site if you're interested in learning more.]

As for "auto-truncate", I assume what you mean is that when an AI extent transitions from Locked to Full and then to Empty after it is archived, you want its file size to be reduced to a nominal value. The AIMD will do this for you.

More info: https://docs.progress.com/bundle/openedge-database-management-117/page/AI-File-Management-utility.html
 
Rob Fitzpatrick said:
Once the notes in the Locked extent have been sent to the target (or targets), the database does change its status from Locked to Full. But it does not empty the extent.

You should use the AI File Management Daemon (AIMD), also known as the AI Archiver, to manage the switching and archiving of your AI extents on a regular basis. The only other alternative is to create your own process for doing so, using scripts with rfutil commands, which I do not recommend.
Click to expand...
Thank you for confirming. I know what you mean we do have the process of emptying full AI extents. Let me try this option.
 
Last edited:
You cannot empty the extents while their status is LOCKED. You will need to fix that first.

The information provided so far doesn’t help to identify why they are locked beyond the obvious “replication is not currently synchronized”. To get to the bottom of that some dsrutil status / monitor (from both source and target) screenshots might help. There may also be messages of interest in the db log files.
 
Hi Tom yes that is exactly the issue I am having, and you are right the LOCKED status need to be fixed first. When I run a test batch job on the source (server) start to fill up the AI extent until all the status of the AI become locked, the same situation with the target database is it filled all the AI extent via update from source and set the status of the AI extent to locked. I am expecting that once the notes is applied to the target the AI status should turn to full.
 
Without giving us details of what's going on, like the Replication Status and errors/warnings from the source DB log file we can't actually help you.
 
To me it sounds like you have done “something” that you hope has resulted in replication being enabled.

But whatever you did has not worked. Replication is obviously not functioning. Finding out why should be your focus.

Start by explaining the steps that you took to enable it and please provide any replication related messages in the db logs.

You might also benefit from running through your procedure with a test database such as “sports”. The big advantage to that is that a re-baseline is trivial. Whereas rebaselining a real db can be quite painful.
 
Thank for all your help.

Look like it is working now by trimming the extra agent2 on the property file.
But i have new issue on the target, now i can not extract the AI and empty like i did in the source using cron script process.

Here is the error on Target.

[root@ifdbat3 tmp]# cat fixdba31.AiStorMgr.log
CLIENT=fixdba31
envid run for fixdba31: client=fix galaxy=dba31
envset run for fix and dba31: LABEL=fixdba31 DLC=/opt/progress/dlc
CLIENT=fixdba31
envid run for fixdba31: client=fix galaxy=dba31
envset run for fix and dba31: LABEL=fixdba31 DLC=/opt/progress/dlc
You are not licensed to access the database. (10830) <--------------------------- Error
ERROR: aimage extract failed /fixdb/fixdba31/db/INVESTORDB.a1 to /fixdb/fixdba31/ai_backup/ai_tmp.


[root@ifdbat3 db]# rfutil INVESTORDB -C aimage extract -a INVESTORDB.a1 -o /fixdb/fixdba31/ai_backup/INVESTORDB.a1
OpenEdge Release 11.7.21 as of Wed Oct 23 14:05:45 EDT 2024

You are not licensed to access the database. (10830)

Here is the log.

Source: Host:ifdbat2
[2025/02/19@09:27:55.489-0500] P-1332307 T-140153506756160 I RPLS 5: (10507) The Fathom Replication Server has successfully connected to the Fathom Replication Agent agent1 on host 192.168.41.35.
[2025/02/19@09:27:55.490-0500] P-1332307 T-140153506756160 I RPLS 5: (11251) The Replication Server successfully connected to all of its configured Agents.
[2025/02/19@09:27:55.492-0500] P-1332307 T-140153506756160 I RPLS 5: (10439) The Fathom Replication Agent agent1 has been properly configured and the target database /fixdb/fixdba31/db/INVESTORDB has been properly sourced.
[2025/02/19@09:27:55.492-0500] P-1332307 T-140153506756160 I RPLS 5: (10508) Beginning Fathom Replication synchronization for the Fathom Replication Agent agent1.
[2025/02/19@09:27:55.594-0500] P-1332307 T-140153506756160 I RPLS 5: (10436) The source database INVESTORDB and the target database /fixdb/fixdba31/db/INVESTORDB on host ifdbat3 are synchronized.
[2025/02/19@09:27:55.595-0500] P-1332307 T-140153506756160 I RPLS 5: (11805) Unlocking after-image file 1 and locking ALL FULL after-image files beginning with file 2.

Target: Host: ifdbat3
[2025/02/19@09:27:54.490-0500] P-105912 T-140077434914624 I RPLA 5: (18549) Involve multiple targets during transition (replication-set) changed to : No.
[2025/02/19@09:27:54.490-0500] P-105912 T-140077434914624 I RPLA 5: (10392) Database /fixdb/fixdba31/db/INVESTORDB is being replicated from database /fixdb/fixdba22/db/INVESTORDB on host 192.168.41.26.
[2025/02/19@09:27:55.492-0500] P-105912 T-140077434914624 I RPLA 5: (10489) Fathom Replication Agent agent1 is beginning Startup Synchronization at block 0.
[2025/02/19@09:27:55.594-0500] P-105912 T-140077434914624 I RPLA 5: (10668) The Source and Target databases are synchronized.
[2025/02/19@09:27:55.594-0500] P-105912 T-140077434914624 I RPLA 5: (10490) Fathom Replication Agent agent1 is beginning normal processing at block 0.
[2025/02/19@09:27:55.594-0500] P-105912 T-140077434914624 I RPLA 5: (10765) Access to the target database is disabled.
[2025/02/19@09:27:55.594-0500] P-105912 T-140077434914624 I RPLA 5: (10766) You need Replication Plus for the Read-Only access.
[2025/02/19@09:27:55.594-0500] P-105912 T-140077434914624 I RPLA 5: (-----) Enabled limited logins.
[2025/02/19@09:27:59.098-0500] P-106195 T-139885252941632 I Usr 12: (452) Login by root on /dev/pts/0.
[2025/02/19@09:27:59.115-0500] P-106195 T-139885252941632 I Usr 12: (7129) Usr 12 set name to Dbdes.
[2025/02/19@09:27:59.116-0500] P-106195 T-139885252941632 I Usr 12: (7129) Usr 12 set name to root.
[2025/02/19@09:27:59.116-0500] P-106195 T-139885252941632 I Usr 12: (453) Logout by root on /dev/pts/0.


Source: properties
[root@ifdbat2 db]# cat INVESTORDB.repl.properties
[server]
control-agents=agent1
database=db1
agent-shutdown-action=recovery
defer-agent-startup=5
transition=manual
transition-timeout=5
[control-agent.agent1]
name=agent1
database=db0
host=ifdbat3
port=6600
replication-method=async
critical=0
connect-timeout=120

[transition]
#replication-set=1
database-role=reverse
transition-to-agents=agent1
restart-after-transition=1
start-secondary-broker=0
normal-startup-arguments=-S 51876
source-startup-arguments=-S 51876 -DBService replserv
target-startup-arguments=-S 51876 -DBService replagent

[agent]
name=agent0
database=db1
listener-minport=4387
listener-maxport=4500

Target: Properties

[root@ifdbat3 db]# cat INVESTORDB.repl.properties
[server]
control-agents=agent1
database=db1
agent-shutdown-action=recovery
defer-agent-startup=5
transition=manual
transition-timeout=5

[control-agent.agent1]
name=agent1
database=db0
host=localhost
port=6600
replication-method=async
critical=0
connect-timeout=120

[transition]
replication-set=1
database-role=reverse
transition-to-agents=agent1
restart-after-transition=1
start-secondary-broker=0
normal-startup-arguments=-S 51877
source-startup-arguments=-S 51877 -DBService replserv
target-startup-arguments=-S 51877 -DBService replagent
[agent]
name=agent1
database=db1
listener-minport=4387
listener-maxport=4500

Source: AI status
[root@ifdbat2 db]# rfutil INVESTORDB -C aimage list|grep -i status
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Locked
Status: Busy
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty
Status: Empty

Target AI status
[root@ifdbat3 db]# rfutil INVESTORDB -C aimage list|grep -i status
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Full
Status: Busy
 
donopena said:
But i have new issue on the target, now i can not extract the AI and empty like i did in the source using cron script process.
Click to expand...
This is a new one on me. What is the point, i.e. the business objective, of this process involving rfutil -C aimage extract? Is this an attempt at saving disk space?
 
Yes to save space and you are correct no point of extracting AI content on target since i have already copy on the source. I just want to have a clean target Database that auto purge AI when its full and not to worry about filling up the disk. Thanks Rob.
 
Have you considered simplifying your life by simply adding enough disk not to have to worry about it? They're cheap these days.
 
You must log in or register to reply here.
Back
Top