When is AnsiString(String(ansstr)) <> ansstr?

Remember the good old times when strings were composed of 8-bit characters? Those were simpler times. When we needed a data buffer, we just used an AnsiString. An 8-bit character is just a byte, surely?

Then Unicode arrived and suddenly all strings were based on 16-bit characters. The code broke all around us. At the company I'm working for we needed about one year to port all our applications to Unicode. (And by “we” I mean myself. I allocated one day per week to work on Unicode issues and slowly worked through the code. Today I'd probably use AI for the task. :) )

We completed this port in Delphi 2010 times. And then we spent the next 10-ish Delphi versions fixing bugs related to this change. And when I thought we had fixed them all (hope, hope), Microsoft did something unexpected.

Some time ago one of our applications started experiencing occasional data corruption — but only on the newest Windows Server 2025. We needed quite some time to find the reason, namely that the new OS version uses UTF-8 for the system code page by default.

Why is that a problem? Well, as it turned out, our code still wasn't completely converted to Unicode. There was still some data buffer that was declared as an AnsiString. It was sent through various functions, was converted to a Unicode string at one location and back to AnsiString at another. And that was enough to destroy the data.

In short, the code did this:

var ansstr := SomeData();
Process(AnsiString(string(ansstr)));

You may think to yourself — so what? Data got converted to Unicode according to the system locale and then it got converted back. Surely that should not destroy the data?

Let me introduce a simple application that points directly to the source of the problem.

program TestAnsiBuffer;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils;

function DumpBuffer(const astr: AnsiString): string;
begin
  Result := Format('[%d] ', [Length(astr)]);
  for var ch in astr do
    Result := Result + Format('%.2x', [Ord(ch)]);
end;

procedure Check(const astr: AnsiString);
begin
  Write('Original: ', DumpBuffer(astr));
  Writeln(', Converted: ', DumpBuffer(AnsiString(string(astr))));
end;

begin
  Check(#$41#$42#$43);
  Check(#$01#$02#$03);
  Writeln('---');
  Check(#$e2#$28#$a1);
  Check(#$e2#$82#$28);
  Check(#$c3#$28);
end.

[The code is also available at https://github.com/gabr42/GpDelphiCode/blob/master/TestAnsiBuffer/TestAnsiBuffer.dpr]

On my development computer this produces:

Original: [3] 414243, Converted: [3] 414243
Original: [3] 010203, Converted: [3] 010203
---
Original: [3] E228A1, Converted: [3] E228A1
Original: [3] E28228, Converted: [3] E28228
Original: [2] C328, Converted: [2] C328

In short, all is well. All converted buffers are the same as the originals.

On a computer with the problematic option enabled, I get this:

Original: [3] 414243, Converted: [3] 414243
Original: [3] 010203, Converted: [3] 010203
---
Original: [3] E228A1, Converted: [7] EFBFBD28EFBFBD
Original: [3] E28228, Converted: [4] EFBFBD28
Original: [2] C328, Converted: [4] EFBFBD28

While the first two buffers are just fine, the latter three are all corrupted.

If you look carefully, you'll notice that in all bad buffers we get the same sequence: EF BF BD. Search for it on the Internet and you'll see that this is the UTF-8 representation of �, also known as the replacement character.

Now it gets clearer. While all five test sequences represent valid binary data, they don't all represent valid UTF-8 encodings. In fact, the latter three were specially constructed so that they represent short invalid UTF-8 sequences. Somewhere in the conversion process (and I didn't bother searching for the specific place) Windows says “oh, this is not a valid UTF-8 sequence; let's replace it with a replacement character.”

Fixing this problem is easy. Just remove the incorrect casting. (And while doing that, change the buffer from AnsiString to something more appropriate.) Finding the problem — well, that was hard.

If you'd like to test your applications, you can find this setting at Settings > Time & language > Language & region > Administrative language settings > Change system locale. Then check Beta: Use Unicode UTF-8 for worldwide language support.

Or, if you have a relatively new Windows 11 setup, you'll find this option directly on the Language & region page.

Enabling this option on my Windows 11 machine, however, produces a different result than on the server:

Original: [3] 414243, Converted: [3] 414243
Original: [3] 010203, Converted: [3] 010203
---
Original: [3] E228A1, Converted: [5] C3A228C2A1
Original: [3] E28228, Converted: [6] C3A2E2809A28
Original: [2] C328, Converted: [3] C38328

Maybe there's a good reason why this setting is still in beta?