Using ChilkatZip to compress in-memory data?

corby-fellas

New Member
Hi,

Has anybody had any experience with ChilkatZip and Progress?

We're using 9.1E and have had success using it to Zip files in the past, and would like to be able to Zip in-memory data, i.e. a MEMPTR. Has anyone successfully been able to do this?

Basically we have two sites and would like to do the following...

Site A:
  • Builds XML Document using database records (using X-DOCUMENT, X-NODEREF etc)
  • Saves the XML to a MEMPTR
  • Compresses the MEMPTR using ChilkatZip
  • Write the Zipped data back to the same (or another) MEMPTR
  • The MEMPTR will then be broken down into a BLOB temp-table to be passed to Site B via an AppServer
Site B:
  • The AppServer reads the BLOB temp-table back into a MEMPTR.
  • Uncompresses the Zipped data using ChilkatZip
  • Writes the uncompressed data back to the same (or another MEMPTR)
  • Loads the X-DOCUMENT from the uncompressed MEMPTR
  • Processes the data in the XML document
Are there any ideas on how this can be done?

The problem seems to be with the AppendData method, the COM Object Viewer gives the following syntax...

[ Com-Handle-Var = ] <com-handle>: AppendData (
Character-FileName,
<anytype>-data ).

Full documentation for ChilkatZip can be found here... http://www.chilkatsoft.com/refdoc/xChilkatZip2Ref.html

TIA
Andrew
 

joey.jeremiah

ProgressTalk Moderator
Staff member
Hello Corby,

Sorry for jumping in so late. I would also be interested in similar functionality for a future project. I'm certainly planning on researching the subject, hopefully, I'll have some code, whitepapers etc. to post in the future. I'm also looking for a cross platform solution.

Have you looked at zlib http://www.zlib.net/ ?

I'm not all that familiar with the formats but, I think, it even comes with an example for handling zip archives called minizip. You can use zlib for compressing/uncompressing memptrs. zlib is free for general purpose and can be used in commercial applications. There are 4GL applications that use zlib, like, PDFInclude. I've also written a small library that makes use of zlib, if you're interested I'll send you a copy.

I'll be interested to hear what you came up with. Good luck !
 

corby-fellas

New Member
Hi Joey,

Many thanks for your reply.

As it happens, my good friend Cecil was kind enough to point me to zlib a couple of days ago and send me some sample code.

Once I've adapted it to suit our requirements I'll post some sample code back here.

Thanks again

Andrew
 

joey.jeremiah

ProgressTalk Moderator
Staff member
Thanks, but I'm specifically interested in ZIP.

Another thing I remembered is, I think, the AppServer, in some cases, compresses when marshaling/transfering data. I think, I've heard of some very impressive results especially with XML.

So that's something you might want to look into, in the documentation or @PEG. It could be that that functionality is already built-in and this may be an unnecessary optimization.
 

corby-fellas

New Member
Hi Joey,

I think you've hit the nail on the head with the AppServer compression!

I had a look on the Knowledge Center and found the following...

ID: P69169
Title: "Message Compression (-mc) available on Progress 9.1D08 service pack"

And...
ID: P80779
Title: "Is it required to specify -mc in the AppServer startup parameters to enable compression"

BUT, there are a couple of important points to note:

9.1D08 Client is only compatitble with 9.1D08 AppServer, as stated in the 9.1E Product Update Bulletin, later versions are compatible with each other.

ID: P74083
Title: "Using AppServer compression causes error 5565 when transferring raw data using HTTP or HTTPS protocols"

We've run a test this morning with a 1.5MB .d file sent over a 64Kb ISDN line in a BLOB temp-table. Before adding the parameter we timed it at 224 sec, with the parameter add the time was reduced to 38 sec. That's near 7 times faster!

I think this will do the job for us for now, but we may revisit the zip/zlib options later and run a comparison between the methods. I'll keep this thread updated if and when we do.

Once again, thanks for your help Joey.

Andrew
 

joey.jeremiah

ProgressTalk Moderator
Staff member
Hi Corby,

Wow, sounds great !

If you're interested, I'll send you a small zlib library I wrote a while back, later on today or tomorrow.

It should be really simple to test and the results could be interesting since text files can compress as much as 20x.
 

corby-fellas

New Member
Hi Joey

Yes, that zlib library does sound interesting. It would most likely save me some time in my testing. :)

Thanks

Andrew
 
I've been looking for some -mc figures, as we upgrade (at last!) soon to something that can use it.

It would be very nice to say it will make a difference - but I've only found before one previous reference to performance improvement, and I would have expected to see much more references (PSDN/Peg/wherever) if it was generally significant.

Thanks for posting that figure though Andrew.
 
but if the compression is done at the message size level theres a chance for a significant improvement

if the compression is done at the object or file level instead.

Sorry, don't understand. If the appserver uses zlib already, how do you improve compression by using it again?
 

joey.jeremiah

ProgressTalk Moderator
Staff member
What I meant is that if the compression is done in smaller chunks some of the compression potential could be lost if the file was otherwise compressed as a whole. For example, try compressing a 1 megabyte text file and a 1 kilobyte snippet from that file, usually the bigger the text file the higher the compression ratio is. I think, it's also one of the shortcomings of the ZIP format, that the compression is done on each file separately and not for the entire data as a whole.

But from a few tests I ran the results that came back were very similar, upwards of 7x times smaller, but I would still be very interested to hear the results you got back. I've attached a small zlib library from an old I/O library I wrote, have a look at the zlib_compressFile and zlib_uncompressFile procedures. I have to be running now, I'll be back soon. HTH
 

Attachments

  • zlib.zip
    33.6 KB · Views: 38

corby-fellas

New Member
I didn't realise -mc used zlib!

I can see what your saying Joey, I will try compressing the file using zlib first when I get a chance.

Thanks for the library, although I don't know if it's just me, but I seem to be having trouble extracting it.

Andrew
 

joey.jeremiah

ProgressTalk Moderator
Staff member
try extracting the attachment with firefox.

the internal procedures i noted show you how to compress data in-memory, they're pretty simple and strait forward. good luck !
 

corby-fellas

New Member
Yep, that's got it! Firefox to the rescue as per normal! :)

I'll give it a look and run some test when I can, and let you know the results.

Thanks

Andrew

try extracting the attachment with firefox.

the internal procedures i noted show you how to compress data in-memory, they're pretty simple and strait forward. good luck !
 
What I meant is that if the compression is done in smaller chunks some of the compression potential could be lost if the file was otherwise compressed as a whole. For example, try compressing a 1 megabyte text file and a 1 kilobyte snippet from that file, usually the bigger the text file the higher the compression ratio is.

I thought that's what you meant, and it sounds reasonable, though I don't know how often it would be practicable in a bog-standard business app passing temp-tables back and forth across the Appserver.
 
I've been looking for some -mc figures, as we upgrade (at last!) soon to something that can use it.

It would be very nice to say it will make a difference - but I've only found before one previous reference to performance improvement, and I would have expected to see much more references (PSDN/Peg/wherever) if it was generally significant.


Here's some possibly relavent info from the Help entry:

Message compression can improve message throughput by reducing network overhead within the lower communication layers. This is more common on slower topologies, such as dial-up and WAN configurations.

Progress recommends that you proceed with caution on faster, more optimized networks where network performance is higher. The high per-packet processing cost of enabling compression can increase network latency as it improves throughput.

The net result could degrade the overall performance of the application.


Anyway, I will experiment with it when we upgrade and let you know if there seems to be noticeable differences over both internal and external networks.

In the meantime, please keep the figures coming, but Joey, I don't drink.
 

joey.jeremiah

ProgressTalk Moderator
Staff member
i also think that using the built in messaging compression (-mc) is a much better solution. if the differences weren't substantial i wouldn't even bother with it, the added complexities wouldn't be worth the trouble and like you said in a lan the compression overhead could decrease performance. though it wouldn't be especially hard to build it into a gateway procedure.

still it would be very interesting to know the difference if any between compression done at the message buffer and object level, if thats even the case and if theres reason it should. i think, the test should include different and small message buffer sizes with considerable size text objects to emphasize any differences, for example, 1k buffer size and 5m xml file.

i think, it was also mentioned that passing the same data as temp-tables return results that are much worse ?! if more smaller messages were sent that could explain the results. so thats also something that would be interesting to play around with. it would be helpful to start with some detail about the appserver messaging mechanism.
 

corby-fellas

New Member
...and the results are in!

I've run a batch of tests with three different files, a small ascii file, a medium sizes jpeg and a larger ascii file. The results are in the attached pdf.

Depending on the file your sending, compressing the whole thing first before splitting it up into the temp-table does improve the performance. In this case the 3rd file goes from 213sec uncompressed, to 40sec with -mc, to 27sec with zlib.

I'm more of a vodka me myself, but then, I wouldn't have bet against Joey's prediction anyway! :)

Andrew
 

Attachments

  • test-results.pdf
    37.7 KB · Views: 19
Top