Question Processing Chunked data from XML socket response

[Potish]

I have a socket connection that is receiving an XML response from a host. The host sends the data chunked (Transfer-Encoding: chunked). I have a procedure that is reading the response and saving to file (.dat). I am able to identify and remove the http header details but have not been able to comeup with an efficient process to remove the length indicators that get passed in the data. I need to remove these so I can save teh contents to an XML file. The procedure to read the response looks as follows. I am trying to write a procedure to read the contents under c:/somefile.dat, identify the chunk length indicators and remove these from the content. Suggestions are appreciated.

PROCEDURE getResponse:
    DEFINE VARIABLE vcWebResp    AS LONGCHAR NO-UNDO.
    DEFINE VARIABLE viBytesAvail AS INTEGER  NO-UNDO.
    DEFINE VARIABLE mResponse    AS MEMPTR  NO-UNDO.
 
    ASSIGN viBytesAvail = vhSocket:GET-BYTES-AVAILABLE().
   
    IF viBytesAvail = 0 THEN RETURN.
                   
    SET-SIZE(mResponse) = 0.
    SET-SIZE(mResponse) = viBytesAvail.
   
    vhSocket:READ(mResponse,1,viBytesAvail,READ-EXACT-NUM).
   
    COPY-LOB FROM OBJECT mResponse TO vcWebResp.
   
   
    ASSIGN vcFinalResp = vcFinalResp + vcWebResp.

    SET-SIZE(mResponse) = 0.
END PROCEDURE.


COPY-LOB FROM OBJECT vcFinalResp TO FILE "c:/somefile.dat".

 

[Marian EDU]

before going into moody waters of socket programming in 4GL may I ask if this is a HTTP POST or GET request that you do to get that XML... do you need to pass parameters to the request, does it have to be POST, does it require authentication?

there might just be an easier way to accomplish what you are trying to do although having fun with sockets and HTTP specs is always fun and could be instructive

 

[Potish]

It is a HTTP POST and it does require authentication. I do not need to pass any parameters to in the request. The host does recommend using 'Accept-Encoding: gzip,deflate' but I am not currently using this as I don't get have a a way to uncompress the respond in progress. The http header for the request looks as follows.

POST https://WebServiceURL HTTP/1.1
Content-Type: text/xml;charset=UTF-8
SOAPAction: ""
Authorization: Basic username:password
Content-Length: >>>9
Host: hostURL
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

 

[Marian EDU]

I see, was thinking you might just use one of the XML parsers (dom, sax) to read that... it does support HTTP URLs for 'file' mode but unfortunately HTTPS protocol is not supported, and the request is sent as GET of course.

Back to the fun part... what you might want to do is to make two methods/procedures: readLine and readBytes. First reads byte by byte from the socket till it get's a full line break (CRLF - \13\10) in it's read buffer, when it does returns the line from buffer. Second will just read in a buffer (should use memptr to be safe but longchar works as well if you're sure you won't get any binary data) a given number of bytes.

Start by reading the headers section, use readLine in a loop till the line is empty (end of headers is CRLF + CRLF)... one piece of info you need from header in Content-Length which will tell you how much to expect in the body buffer (Content-Type might also be found interesting). Then call readBytes using the value of content length and this is what are you after I think

Of course, always use is-connected in the loop and check read-bytes after each read (0 means connection was dropped)... or use read-response callback if you prefer.

For authorization (I assume basic) you'll need to base64-encode the "user : password" string.

 

[Cecil]

Hi

I don't know if you got a solution to your problem yet, but today I was having the same issue with chunked encode data and wrote this class object to fix my problem. Hope this is what you are looking for. Sorry that it has little to zero commentary it's still in development.

I must confess that I did use a piece of JAVA source code as my muse to develop an OpenEdge ABL equivalent. Also Julian lyndon-smith wrote a http client object to also handle chunk encoded data but some reason it was not working for me.

httpUnChunkData.cls

CLASS src.httpUnChunkData:

    CONSTRUCTOR httpUnChunkData():
 
    END CONSTRUCTOR.

    METHOD PRIVATE INTEGER HexToInt (INPUT chHEX AS CHARACTER):
   
        DEFINE VARIABLE ININTEGER   AS INTEGER NO-UNDO INITIAL 0.
        DEFINE VARIABLE inj         AS INTEGER NO-UNDO.
   
        chHEX = CAPS(chHEX).
        DO inj = 1 TO LENGTH(chHEX):
            IF CAN-DO("0,1,2,3,4,5,6,7,8,9", (SUBSTRING(chHEX, inj, 1))) THEN
                ININTEGER = ININTEGER + INT(SUBSTRING(chHEX, inj, 1)) * EXP(16, (LENGTH(chHEX) - inj)).
            ELSE
                ININTEGER = ININTEGER + (KEYCODE(SUBSTRING(chHEX, inj, 1)) - KEYCODE("A") + 10) * EXP(16, (LENGTH(chHEX) - inj)).
        END.

        RETURN ININTEGER.
    END METHOD.

    METHOD PRIVATE INTEGER strpos(INPUT mpDataSource AS memptr,
                                  INPUT chTarget AS CHARACTER):

        RETURN THIS-OBJECT:strpos(INPUT mpDataSource,
                                  INPUT chTarget,
                                  INPUT 1).
    END METHOD.

    METHOD PRIVATE INTEGER strpos(INPUT mpDataSource AS memptr,
                                  INPUT chTarget     AS CHARACTER,
                                  INPUT inStartingPos  AS INTEGER):

        DEFINE VARIABLE inLoop       AS INTEGER     NO-UNDO.
        DEFINE VARIABLE inStringPos  AS INTEGER     NO-UNDO.
        DEFINE VARIABLE inSourceSize AS INTEGER     NO-UNDO.

        inSourceSize = GET-SIZE(mpDataSource).
       
        IF inStartingPos GT inSourceSize THEN
            RETURN ?.

        STRING-LOOP:
        DO inLoop = inStartingPos TO inSourceSize:

            IF GET-STRING(mpDataSource, inLoop, LENGTH(chTarget)) EQ chTarget THEN
            DO:
                inStringPos = inLoop.
                LEAVE STRING-LOOP.
            END.
        END.

        RETURN inStringPos.
    END METHOD.

    METHOD PUBLIC MEMPTR Filter(INPUT mpDataSource AS MEMPTR):

        DEFINE VARIABLE inStartingAtPos         AS INTEGER     NO-UNDO INITIAL 1.
        DEFINE VARIABLE inChunkRemaining AS INTEGER     NO-UNDO.
        DEFINE VARIABLE inDelimiterPos      AS INTEGER     NO-UNDO.
        DEFINE VARIABLE chChunkedData    AS CHARACTER   NO-UNDO.
        DEFINE VARIABLE chChunkHEXSize    AS CHARACTER   NO-UNDO.
        DEFINE VARIABLE mpDataTemp       AS MEMPTR      NO-UNDO.
        DEFINE VARIABLE mpDataTarget     AS MEMPTR      NO-UNDO.
        DEFINE VARIABLE mpDataSwap       AS MEMPTR      NO-UNDO.
       
        SET-SIZE(mpDataTarget) = 0.

        UNCHUNK-FILTER:
        DO WHILE inStartingAtPos LT GET-SIZE(mpDataSource):

            IF inChunkRemaining EQ 0 THEN
            DO:
                inDelimiterPos = strpos(INPUT mpDataSource,
                                        INPUT CHR(13) + CHR(10),
                                        INPUT inStartingAtPos).

                chChunkedData    = GET-STRING(mpDataSource, inStartingAtPos, (inDelimiterPos - inStartingAtPos) ).
                chChunkHEXSize   = TRIM(chChunkedData).
                inChunkRemaining = THIS-OBJECT:HexToInt(INPUT chChunkHEXSize).

                MESSAGE inStartingAtPos inDelimiterPos chChunkHEXSize inChunkRemaining 'bytes' ROUND((inDelimiterPos / GET-SIZE(mpDataSource) * 100), 2)'%'.

                IF inChunkRemaining EQ 0 THEN
                    LEAVE UNCHUNK-FILTER.

                inStartingAtPos = inDelimiterPos + 2.
            END.

            SET-SIZE(mpDataTemp) = 0.

            COPY-LOB FROM OBJECT mpDataSource STARTING AT inStartingAtPos FOR inChunkRemaining TO OBJECT mpDataTemp.

            inStartingAtPos = inStartingAtPos + GET-SIZE(mpDataTemp).

            IF GET-SIZE(mpDataTemp) EQ inChunkRemaining THEN
                inStartingAtPos = inStartingAtPos + 2.

            inChunkRemaining = inChunkRemaining - GET-SIZE(mpDataTemp).
           
            /** Hack to to append data to an exisintg MEMPTR **/
            SET-SIZE(mpDataSwap) = 0.
            SET-SIZE(mpDataSwap) = GET-SIZE(mpDataTarget) + GET-SIZE(mpDataTemp).
           
            COPY-LOB FROM OBJECT mpDataTarget TO OBJECT mpDataSwap OVERLAY AT 1.
            COPY-LOB FROM OBJECT mpDataTemp   TO OBJECT mpDataSwap OVERLAY AT GET-SIZE(mpDataTarget) + 1.

            SET-SIZE(mpDataTarget) = 0.
            SET-SIZE(mpDataTarget) = GET-SIZE(mpDataSwap).
            COPY-LOB FROM OBJECT mpDataSwap TO OBJECT mpDataTarget.

            SET-SIZE(mpDataSwap) = 0.
            SET-SIZE(mpDataTemp)  = 0.

        END.
       
        RETURN mpDataTarget.

    END METHOD.
END CLASS.

 

[Potish]

Thank you the suggestion. I was not able to find a good solution so hopefully this will resolve the problem.