Progress Collation

[Hugues]

Hello,

We receive orders with name with internatioal caracter. ex: "Jesus Nińo".

When I check direct on progress table, the save information is "Jesus Ni?o"

some caracter is replaced by ?.

How to solve this ? I want the information saved correctly

thanks

 

[TomBascom]

You almost certainly have mismatched code pages somewhere between where the orders come from and the database.

You can check the code pages being used by the db by searching the .lg file for messages containing "-cp".

The program that put the data into the database *might* have used something different - to check that you need to carefully parse the startup of that program (don't forget $DLC/startup.pf) or get a stack trace (protrace) while it is running (use proGetStack for that).

If the source of the data documents the expected code page that would also be helpful to know.

In legacy applications this stuff can be mind numbingly difficult to untangle. The modern solution is to use UTF-8 at every layer.

 

[Rob Fitzpatrick]

We receive orders with name with internatioal caracter. ex: "Jesus Nińo".

Receive them from whom, and how?

How does the sending party think they are encoding the data? (i.e. encoded with which code page?) Is there a specification that dictates how text should be encoded?

 

[Hugues]

The message is a JSON message encoded in UTF-8

ex: "Name": "Jesus Ni\u0144o"

The chartset of the database is : ISO8859-1

 

[Rob Fitzpatrick]

The message is a JSON message encoded in UTF-8

ex: "Name": "Jesus Ni\u0144o"

The chartset of the database is : ISO8859-1

That may be a problem, if there are no application-level restrictions on what is considered valid data. There's a lot of data that can be represented in UTF-8 that you could potentially receive that has no valid mapping into ISO8859-1. And even for characters that are in both character sets, they may have different encodings. So another variable is whether the client that is receiving the UTF-8 data and writing the ISO8859-1 data into the database knows that the input data is UTF-8 and is mapping it somehow, or whether it is just assuming that everything it sees is ISO8859-1, in which case you have data loss.

 

[andre42]

So in this example you actually have "ń" (small letter n with acute)?
ISO8859-1 does only contain "ñ" (small letter n with tilde).
If you actually have to support all kinds of international letters than you have no other option than to convert your database and application to UTF-8.
If you think you could get by with replacing it by a plain "n" that you would have to include that conversion into your input routine.
Of course "ń" is included in ISO8859-2 but since you'd miss out on other characters it wouldn't help to switch to that code page.

 

[Stefan]

You can verify that your input data cannot be converted by using the following snippet:

def var lcjson as longchar no-undo.

define temp-table tt no-undo
   field Name as char
   .
  
lcjson = '~{"tt":[~{"Name":"Jesus Ni\u0144o"}]}'.

temp-table tt:read-json( "longchar", lcjson ).

find first tt.
message tt.name view-as alert-box.

If you start your session with -cpinternal utf-8 then the name will be displayed.
If you start your session with -cpinternal iso8859-1 then you will get error 15363 (Unable to convert JSON to native data type for field 'Name' in temp-table 'tt'.).