WndProc/AddInputCharacter issue in Slovak layout.

sneakyevil commented

2023-09-04 13:43:14 +00:00

(Migrated from github.com)

Version/Branch of Dear ImGui:

Version: 1.89.8
Branch: master

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_win32.cpp + imgui_impl_dx11.cpp
Compiler: MSVC (Multi-byte Character Set)
Operating System: Windows 10

My Issue/Question:

I got some issue with WndProc and I couldn't find anything at google or anyone having similar issue besides the russian input posted here as issue like year ago, but this issue is kinda weird. So first at all I have loaded correct font with glyph range that can display the characters I'm trying to type and I even tried to manually insert the utf-8 encoded characters that I tried to type and they display correctly. The issue is when I press "4" on top of my keyboard which translates to "č" in Slovak layout it is passed in WndProc as wparam of 232 which instead displaying as "č" it display as "è" even after MultiByteToWideChar conversion. When I add primitive if check before io.AddInputCharacter call to check if wch is 232 and change it to 269 it will always properly display the "č" character without any issue.

Here are some characters typed on the Slovak layout on top keyboard from "1, 2, ... 0":

In ImGui: "+¾šèžýáíé"
Outside ImGui: "+ľščťžýáíé"

I have no idea how to even solve this issue besides manually doing conversion to it and I still have no clue if this issue is present only on my system or anyone else.

**Version/Branch of Dear ImGui:** Version: 1.89.8 Branch: master **Back-end/Renderer/Compiler/OS** Back-ends: imgui_impl_win32.cpp + imgui_impl_dx11.cpp Compiler: MSVC (Multi-byte Character Set) Operating System: Windows 10 **My Issue/Question:** I got some issue with WndProc and I couldn't find anything at google or anyone having similar issue besides the russian input posted here as issue like year ago, but this issue is kinda weird. So first at all I have loaded correct font with glyph range that can display the characters I'm trying to type and I even tried to manually insert the utf-8 encoded characters that I tried to type and they display correctly. The issue is when I press "4" on top of my keyboard which translates to "č" in Slovak layout it is passed in WndProc as wparam of 232 which instead displaying as "č" it display as "è" even after MultiByteToWideChar conversion. When I add primitive if check before `io.AddInputCharacter` call to check if wch is 232 and change it to 269 it will always properly display the "č" character without any issue. Here are some characters typed on the Slovak layout on top keyboard from "1, 2, ... 0": - In ImGui: "`+¾šèžýáíé`" - Outside ImGui: "`+ľščťžýáíé`" I have no idea how to even solve this issue besides manually doing conversion to it and I still have no clue if this issue is present only on my system or anyone else.

ocornut commented

2023-09-04 13:59:01 +00:00

(Migrated from github.com)

č is U+010D == 269 so that value is right.
https://www.compart.com/en/unicode/U+010D

Your issue has to do with Win32 API supports of Unicode and UTF+8.

I reckon you have the same issue as #5725.
As you are compiling in MBCS mode, you need to be calling CreateWindowW() when creating your window. CreateWindow() will call CreateWindowA().

See our solution
ocornut/imgui@a229a7f39f
and ocornut/imgui@0a7054c7e4

Compiler: MSVC (Multi-byte Character Set)

Why not switching to Unicode?

č is U+010D == 269 so that value is right. https://www.compart.com/en/unicode/U+010D Your issue has to do with Win32 API supports of Unicode and UTF+8. I reckon you have the same issue as #5725. As you are compiling in MBCS mode, you need to be calling `CreateWindowW()` when creating your window. `CreateWindow()` will call `CreateWindowA()`. See our solution https://github.com/ocornut/imgui/commit/a229a7f39fece2ec40308e47684792a9ceb519cd and https://github.com/ocornut/imgui/commit/0a7054c7e448457cad37a6904dff1d044aeb3b2b > Compiler: MSVC (Multi-byte Character Set) Why not switching to Unicode?

sneakyevil commented

2023-09-04 15:11:46 +00:00

(Migrated from github.com)

Well this does indeed fix the issue by using CreateWindowW even in MBCS mode, but is there any solution for Ascii?
The reason why I can't switch to unicode is due to that I working on modification for game that uses CreateWindowExA.

Well this does indeed fix the issue by using CreateWindowW even in MBCS mode, but is there any solution for Ascii? The reason why I can't switch to unicode is due to that I working on modification for game that uses CreateWindowExA.

sneakyevil commented

2023-09-04 15:33:33 +00:00

(Migrated from github.com)

I think there could be some implementation done in ImGui for the ascii version, but I have no idea how to properly get this thing to even work. Changing the codepage to 1250 for MultiByteToWideChar will properly fix the issue of conversion, but at the same time this is hardcoded. Calling GetACP returns for me 1252 and same value I get when using GetLocaleInfoA for some odd reason. Using that codepage will break the conversion same way as before. There is wndproc msg WM_INPUTLANGCHANGE that could be used to somehow implement the codepage fetch properly and change it for the conversion.

I think there could be some implementation done in ImGui for the ascii version, but I have no idea how to properly get this thing to even work. Changing the codepage to `1250` for `MultiByteToWideChar` will properly fix the issue of conversion, but at the same time this is hardcoded. Calling `GetACP` returns for me `1252` and same value I get when using `GetLocaleInfoA` for some odd reason. Using that codepage will break the conversion same way as before. There is wndproc msg `WM_INPUTLANGCHANGE` that could be used to somehow implement the codepage fetch properly and change it for the conversion.

ocornut commented

2023-09-04 15:35:23 +00:00

(Migrated from github.com)

I don't know, you will need to dig (also check linked issues and other issues linked to it and to its commit).
If you find a solution that's not too complex I'm open to including it in imgui_impl_win32.

I don't know, you will need to dig (also check linked issues and other issues linked to it and to its commit). If you find a solution that's not too complex I'm open to including it in imgui_impl_win32.

sneakyevil commented

2023-09-04 16:07:48 +00:00

(Migrated from github.com)

No idea if this is proper way of getting code page, but it seems to work for me.

The code for getting codepage:

HKL m_KeyboardLayout = GetKeyboardLayout(0);

LCID m_KeyboardLCID = MAKELCID(HIWORD(m_KeyboardLayout), SORT_DEFAULT);
    
unsigned int m_CodePage = CP_ACP;
GetLocaleInfoA(m_KeyboardLCID, LOCALE_RETURN_NUMBER | LOCALE_IDEFAULTANSICODEPAGE, (LPTSTR)&m_CodePage, sizeof(m_CodePage));

Using this with wndproc WM_INPUTLANGCHANGE will return proper codepage when switching layout.
When switching from Slovak to English layout the code page is 1252, switching from English to Slovak the code page is 1250.

No idea if this is proper way of getting code page, but it seems to work for me. The code for getting codepage: ```cpp HKL m_KeyboardLayout = GetKeyboardLayout(0); LCID m_KeyboardLCID = MAKELCID(HIWORD(m_KeyboardLayout), SORT_DEFAULT); unsigned int m_CodePage = CP_ACP; GetLocaleInfoA(m_KeyboardLCID, LOCALE_RETURN_NUMBER | LOCALE_IDEFAULTANSICODEPAGE, (LPTSTR)&m_CodePage, sizeof(m_CodePage)); ``` Using this with wndproc WM_INPUTLANGCHANGE will return proper codepage when switching layout. When switching from Slovak to English layout the code page is 1252, switching from English to Slovak the code page is 1250.

👍 1

ocornut commented

2023-09-07 12:11:05 +00:00

(Migrated from github.com)

Closed by #6785

PathogenDavid commented

2023-09-29 00:08:22 +00:00

(Migrated from github.com)

I've had in my back of my head that I wanted to look into this more since https://github.com/ocornut/imgui/pull/6785 didn't quite pass the sniff test for me (for reasons similar to Omar's concerns raised on the PR.)

I'm not confident using GetKeyboardLayout is the technically correct thing to do here.

WM_CHAR is defined as follows for non-Unicode windows:

[...] the system provides characters in the current process code page

Is the keyboard layout always the same as the current process code page? My instinct says no, but maybe that's the default?

If it's only the default, I would be concerned the adopted fix will end up breaking apps manifested as UTF-8 since in those apps all the old-school ANSI stuff will be UTF-8 but it's unclear if that'd extend to the keyboard layout. (Would need to investigate further.)

I could not find a 100% definitive API for getting the "process code page", I suspect that should actually probably read "thread code page", IE: CP_THREAD_ACP.

I'd be interested in hearing whether using the old code with CP_THREAD_ACP instead of CP_ACP fixes this issue. (I'm not currently set up to easily reproduce this bug, otherwise I'd check for myself.)

It'd also be worth checking if there's a disagreement between CP_ACP and CP_THREAD_ACP when this bug occurs. You can check with GetCPInfoExW:

CPINFOEXW info = { };
::GetCPInfoExW(CP_ACP, 0, &info);
wprintf(L"       CP_ACP: '%s'\n", info.CodePageName);
::GetCPInfoExW(CP_THREAD_ACP, 0, &info);
wprintf(L"CP_THREAD_ACP: '%s'\n", info.CodePageName);

I've had in my back of my head that I wanted to look into this more since https://github.com/ocornut/imgui/pull/6785 didn't quite pass the sniff test for me (for reasons similar to Omar's concerns raised on the PR.) I'm not confident using `GetKeyboardLayout` is the technically correct thing to do here. [`WM_CHAR`](https://learn.microsoft.com/en-us/windows/win32/inputdev/wm-char) is defined as follows for non-Unicode windows: > [...] the system provides characters in the current process code page Is the keyboard layout always the same as the current process code page? My instinct says no, but maybe that's the default? If it's only the default, I would be concerned the adopted fix will end up breaking [apps manifested as UTF-8](https://learn.microsoft.com/en-us/windows/win32/sbscs/application-manifests#activeCodePage) since in those apps all the old-school ANSI stuff will be UTF-8 but it's unclear if that'd extend to the keyboard layout. (Would need to investigate further.) I could not find a 100% definitive API for getting the "process code page", I suspect that should actually probably read "thread code page", IE: `CP_THREAD_ACP`. I'd be interested in hearing whether using the old code with `CP_THREAD_ACP` instead of `CP_ACP` fixes this issue. (I'm not currently set up to easily reproduce this bug, otherwise I'd check for myself.) It'd also be worth checking if there's a disagreement between `CP_ACP` and `CP_THREAD_ACP` when this bug occurs. You can check with [`GetCPInfoExW`](https://learn.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getcpinfoexw): ```cpp CPINFOEXW info = { }; ::GetCPInfoExW(CP_ACP, 0, &info); wprintf(L" CP_ACP: '%s'\n", info.CodePageName); ::GetCPInfoExW(CP_THREAD_ACP, 0, &info); wprintf(L"CP_THREAD_ACP: '%s'\n", info.CodePageName); ```

ocornut commented

2023-09-29 08:47:31 +00:00

(Migrated from github.com)

I did back then try to run GetCPInfoEx() on all meaningful defines including CP_THREAD_ACP and didn’t get any difference.

sneakyevil commented

2023-10-02 07:27:30 +00:00

(Migrated from github.com)

It'd also be worth checking if there's a disagreement between CP_ACP and CP_THREAD_ACP when this bug occurs. You can check with GetCPInfoExW:

This has been discussed in the pull request that it doesn't change anything. Also I mention there that I fixed it by changing the language for non-unicode programs in advanced language settings in windows settings, which seems weird to even use that approach if you want for example change keyboard layout on the fly and type in different language.

> It'd also be worth checking if there's a disagreement between `CP_ACP` and `CP_THREAD_ACP` when this bug occurs. You can check with [`GetCPInfoExW`](https://learn.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getcpinfoexw): This has been discussed in the pull request that it doesn't change anything. Also I mention there that I fixed it by changing the language for non-unicode programs in advanced language settings in windows settings, which seems weird to even use that approach if you want for example change keyboard layout on the fly and type in different language.

PathogenDavid commented

2023-10-02 20:42:43 +00:00

(Migrated from github.com)

This has been discussed in the pull request that it doesn't change anything.

Oops, I don't know how I missed/forgot that considering it was right at the start. Sorry about that!

> This has been discussed in the pull request that it doesn't change anything. Oops, I don't know how I missed/forgot that considering it was right at the start. Sorry about that!

Rows
Columns

WndProc/AddInputCharacter issue in Slovak layout. #6782