Answered Invisible File by Windows OS

Hello Guys,

We had a strange issue recently. We don't know if is a progress side issue or on windows side.
There was an invisible file on a end user working directory that continue to grow endlessly around 60Gb and more. The issue was resolved by restarting the server.
On this working directory we generate and modify HTML file by using "OUTPUT TO" .

One of our idea is that a prowin32 encounter an issue and looped in a writting procedure. But it's a supposition.

Did you ever come accross this kind of issue ? If yes how did you solve it ?

Best Regards,
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
What do you mean by "invisible"? Do you mean hidden (a file system attribute) or something else?

What was the evidence for its existence and size? If you found the file somehow, do you know its name? Where was it located?

Did you check which process(es) had handles to the file?
 
By invisible:
- I mean that the folder c:\temp\progress\ weigthed around 60Go and more
- When you cliked on the folder to enter it, there was no such file or sufficient number of small file to attaint this weight
- They have tried we different tools to analyzed disk space and no such file was visible

The file wasn't found at all.

Because they weren't able to found the file they weren't able to found the process that created it.
 
This was unable too and the file wasn't there.
As I said earlier I'm not sure that progress was the root of it. I'm just exploring the path and saerching if it happend to one of you ?
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
I'm not sure that progress was the root of it
Are any OpenEdge components running on this machine, apart from GUI clients? For example a database or App Server?

If the client applications connect to a database, is the database TDE-enabled? In that case Progress creates client temporary files in a different way. Although I wouldn't expect one to grow to 60 GB in size.
 
They are only GUI client and no database or apps server.
We have a server for each task:
A database server
A RDS serveur for production application
And local install for othe user.

Same on this side.
Also it's the first time we had that.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
If you aren't using TDE and you had File Explorer showing hidden files when the space was being used and the directory c:\temp\progress was 60 GB in size and you still couldn't see a large file, then then it seems likely that whatever was using your disk space wasn't an OpenEdge client temporary file.

How did you determine the size of this one directory? By looking at its properties in File Explorer?

In case this happens again: if a file is growing quickly, that means one or more processes are doing a lot of disk write I/O. You could find them with Process Explorer, by displaying the appropriate I/O columns (e.g. I/O Writes, I/O Write Bytes, I/O Delta Writes, I/O Delta Write Bytes) and sorting on them. Run as Administrator to see I/O from all processes. You can see the file handles of the threads in a process by selecting the process and hitting Ctrl+h for the handle view and then sorting by Name and looking for the appropriate path.

If you need more detail, e.g. the amount of data being written to each file by that process then run Process Monitor, filter on that process' PID, and capture some data. Then stop capturing and select Tools, File Summary. Select the By Path tab and then click the Write Bytes column to sort on that metric. The Path column contains the fully-qualified paths of the files written to by the filtered process(es) during the trace.

Bonus tip: if you install the Sysinternals Suite (which includes Process Explorer and Process Monitor) from the Windows store, you won't have to re-download the utilities as they are updated. They will be kept up to date.

1670333659046.png

If you don't have Store access, you can also use winget.
 
Hello @Rob Fitzpatrick ,
Thank you for your support on this matter. I'm glad in a way that progress seems not to be the root of it. Also what you tell me will help us to put down a procedure to follow if this issue where to happen again .
Thank you again . I will marked it answered
 
Thank you @TomBascom .
After checking some kb and info from progress knowledge base, I know that for my version the limit size of DBI file is 1Tb.
I also know that the growth could be caused by non using "NO-UNDO" on temp-table definition.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
I also know that the growth could be caused by non using "NO-UNDO" on temp-table definition.
Not multiple GB of data. That amount of growth is caused by someone creating huge amounts of temp-table data. For example reading a large database table and caching it in a temp-table. I saw this kind of growth a while back with developers who were new to ProDataSets and used them improperly.
 
The application that was causing that run on arround 300 hundred computer 7days a week 24hours a day.
And we had two cases of this growing file causing by this app in less than a year but without a way to know the exact reason of this growth. I'm pretty sure that the "NO-UNDO" was the root of it. But I don't know what really happened that caused that.
We are not using prodataset as of today and I don't master it at all.
I have placed some trakker to try discover the exact caused but not sure to be able to found.
 

TomBascom

Curmudgeon
It is very unlikely that NO-UNDO per se is your problem.

MUCH more likely is that you have some form of a "memory leak" via temp-tables. Especially if this occurs slowly and after a long run time. You might, for instance, be dynamically creating them but never deleting them. Or passing them around between procedures that make deep copies and then never clean up.

Or it could be as simple as creating lots of records that just never get deleted.

It is possible to see what temp tables are active and how large they are in a session but you will need to add some code to your application to do that (you cannot get the data from outside of a running session). This PPT might be useful: https://pugchallenge.org/downloads2017/RAPID.pptx, the temp-table stuff starts on slide 32.
 
Top