K
Kumar Vikesh
Guest
In today’s data-driven world, cloud storage is no longer optional—it’s essential. Organizations rely on platforms like Amazon S3 for durability, scalability and cost efficiency. With support for objects up to 5 TB, S3 has become the backbone for data exchange, analytics and ETL workflows.
But moving large files between systems and S3 can be challenging. Network interruptions, timeouts and memory limits often stand in the way of smooth transfers. That’s where the Progress DataDirect Autonomous REST Connector (ARC) steps in.
AWS S3 is more than just storage; it is a strategic layer for:
The Progress DataDirect Autonomous REST JDBC Connector had existing support for file upload and download to S3, but transfers were previously limited to 2 GB due to memory constraints. With our latest enhancements, DataDirect ARC supports multipart upload and download, making large file transfers seamless and more reliable.
You can configure this feature using:
The Connector affords organizations like yours:
By bridging REST APIs and cloud storage, Progress DataDirect ARC strengthens its position as a versatile connector for modern data architecture. Learn more.
Also, watch this webinar for more on how to connect any REST Data Source using the Progress DataDirect Autonomous REST Connector.
Continue reading...
But moving large files between systems and S3 can be challenging. Network interruptions, timeouts and memory limits often stand in the way of smooth transfers. That’s where the Progress DataDirect Autonomous REST Connector (ARC) steps in.
Why AWS S3 Matters
AWS S3 is more than just storage; it is a strategic layer for:
- Data pipelines: Acting as an intermediary for bulk data movement
- Analytics workflows: Storing large datasets for BI and ETL processes
- OEM integrations: Enabling scalable, secure data exchange between systems
What’s New in the Progress DataDirect Autonomous REST JDBC Connector?
The Progress DataDirect Autonomous REST JDBC Connector had existing support for file upload and download to S3, but transfers were previously limited to 2 GB due to memory constraints. With our latest enhancements, DataDirect ARC supports multipart upload and download, making large file transfers seamless and more reliable.
Key Features
- Multipart Upload: Automatically splits files larger than 100 MB into parts and uploads them in parallel, following AWS best practices
- Multipart Download: Uses byte-range requests to download files in chunks, enabling efficient retrieval of objects up to 5 TB
- Configurable Chunk Size: Fine-tunes performance using the new connection property, FileTransferPartSize
- Robust Error Handling: Retries logic and detailed logging for operational visibility
How It Works
You can configure this feature using:
Connection Property: FileTransferPartSize
New Stored Procedures:
getObjectSize – Retrieves the size of an object in S3
uploadFile – Uploads a file from the local system to S3
downloadFile – Downloads a file from S3 to the local system
What This Means for You
The Connector affords organizations like yours:
- Scalability: Moves files up to 5 TB without memory bottlenecks
- Reliability: Reduces the risk of failure with the allowance of retries via multipart operations
- Integration: Leverages S3 for robust data exchange in OEM solutions that embed DataDirect ARC
By bridging REST APIs and cloud storage, Progress DataDirect ARC strengthens its position as a versatile connector for modern data architecture. Learn more.
Also, watch this webinar for more on how to connect any REST Data Source using the Progress DataDirect Autonomous REST Connector.
Continue reading...