ecraven commented

(Migrated from github.com)

Using imgui in a game, we pass the pathnames as utf-8 to it. Under windows, it seems ImLoadFileToMemory uses fopen, which does not deal with utf-8 correctly, but just uses the "default locale". Would it be possible to just make every pathname in imgui work correctly with utf-8? Otherwise cross-platform support is even harder :-/

Thanks for a great library!

Using imgui in a game, we pass the pathnames as utf-8 to it. Under windows, it seems ImLoadFileToMemory uses fopen, which does not deal with utf-8 correctly, but just uses the "default locale". Would it be possible to just make every pathname in imgui work correctly with utf-8? Otherwise cross-platform support is even harder :-/ Thanks for a great library!

ocornut commented

2016-11-22 14:50:02 +00:00

(Migrated from github.com)

Pinging @dougbinks as he had the same problem last week.
What would be the correct way to open files with UTF-8 path under Windows?

Pinging @dougbinks as he had the same problem last week. What would be the correct way to open files with UTF-8 path under Windows?

Wohlstand commented

2016-11-22 15:00:08 +00:00

(Migrated from github.com)

on Windows fopen reads ANSI-only. To have support of UTF-8 paths you must convert your string into wide characters (UTF-16) string via MultiByteToWideChar and them use _wfopen (then use same fread/fwrite, fprintf, fscanf, etc. as is).
Look to an example to force UTF8 paths support on the Windows:

std::string filePath = "C:/Папка Васи/Что-то.txt";

#ifndef _WIN32
    FILE *f = fopen(filePath.c_str(), "rb");
#else
    std::wstring filePathW;
    filePathW.resize(filePath.size());
    int newSize = MultiByteToWideChar(CP_UTF8, 0, filePath.c_str(), filePath.length(), const_cast<wchar_t *>(filePathW.c_str()), filePath.length());
    filePathW.resize(newSize);
    FILE *f = _wfopen(filePathW.c_str(), L"rb");
#endif

fread(&xxx, 1, xxx_size, f);

on Windows fopen reads ANSI-only. To have support of UTF-8 paths you must convert your string into wide characters (UTF-16) string via MultiByteToWideChar and them use _wfopen (then use same fread/fwrite, fprintf, fscanf, etc. as is). Look to an example to force UTF8 paths support on the Windows: ```C++ std::string filePath = "C:/Папка Васи/Что-то.txt"; #ifndef _WIN32 FILE *f = fopen(filePath.c_str(), "rb"); #else std::wstring filePathW; filePathW.resize(filePath.size()); int newSize = MultiByteToWideChar(CP_UTF8, 0, filePath.c_str(), filePath.length(), const_cast<wchar_t *>(filePathW.c_str()), filePath.length()); filePathW.resize(newSize); FILE *f = _wfopen(filePathW.c_str(), L"rb"); #endif fread(&xxx, 1, xxx_size, f); ```

mooskagh commented

2016-11-22 15:00:36 +00:00

(Migrated from github.com)

Windows'es native encoding is UTF-16, so I cannot think out anything better than somehow to convert filename from utf-8 to utf-16 (using MultiByteToWideChar()) and then use _wfopen().
It's ugly though as requires allocating buffer etc.

Windows'es native encoding is UTF-16, so I cannot think out anything better than somehow to convert filename from utf-8 to utf-16 (using MultiByteToWideChar()) and then use _wfopen(). It's ugly though as requires allocating buffer etc.

dougbinks commented

2016-11-22 15:04:59 +00:00

(Migrated from github.com)

I converted the filenames to ASCII with GetShortPathName for use in imgui, see https://github.com/dougbinks/RuntimeCompiledCPlusPlus/blob/Avoyd/Aurora/RuntimeCompiler/FileSystemUtils.h#L528-L544

Wohlstand commented

2016-11-22 15:08:53 +00:00

(Migrated from github.com)

converted the filenames to ASCII with GetShortPathName for use in imgui, see https://github.com/dougbinks/RuntimeCompiledCPlusPlus/blob/Avoyd/Aurora/RuntimeCompiler/FileSystemUtils.h#L528-L544

I think, is not good call MultiByteToWideChar twice while it internally doing lot of works. Instead while you converting UTF8 into UTF16, just make buffer equal to UTF8 string, then cut tail when you got the resulted size. When converting UTF16 to UTF8, just make buffer of double size, then same - cut tail. Danger is exists only on converting UTF32 strings to UTF8 (where value of character is bigger than 65535 and requires more than two UTF8 bytes (three or four)). Therefore buffer must have a quad size.
Short filenames are helpful in cases when need use libraries with fopen and closed source (impossible to implement native UTF8 support) or just to don't modify it

> converted the filenames to ASCII with GetShortPathName for use in imgui, see https://github.com/dougbinks/RuntimeCompiledCPlusPlus/blob/Avoyd/Aurora/RuntimeCompiler/FileSystemUtils.h#L528-L544 I think, is not good call MultiByteToWideChar twice while it internally doing lot of works. Instead while you converting UTF8 into UTF16, just make buffer equal to UTF8 string, then cut tail when you got the resulted size. When converting UTF16 to UTF8, just make buffer of double size, then same - cut tail. Danger is exists only on converting UTF32 strings to UTF8 (where value of character is bigger than 65535 and requires more than two UTF8 bytes (three or four)). Therefore buffer must have a quad size. Short filenames are helpful in cases when need use libraries with fopen and closed source (impossible to implement native UTF8 support) or just to don't modify it

dougbinks commented

2016-11-22 15:17:38 +00:00

(Migrated from github.com)

Converting all fopen to _wfopen and supporting UTF8 to UTF16 is certainly the right way to approach this, I used the GetShortPathName solution since I needed a solution for more than just imgui this was the fastest way to get this with no code change needed for imgui.

Wohlstand commented

2016-11-22 15:19:46 +00:00

(Migrated from github.com)

In my case I just used a custom function (on non-Win32 platforms just a macro to regular fopen) which accepts same arguments as fopen, but inside I converting a string into UTF16 and then passing _wfopen

#ifdef _WIN32
FILE*  my_fopen(char* filepath, char*mode);
#else
#define my_fopen fopen
#endif

In my case I just used a custom function (on non-Win32 platforms just a macro to regular fopen) which accepts same arguments as fopen, but inside I converting a string into UTF16 and then passing _wfopen ```C++ #ifdef _WIN32 FILE* my_fopen(char* filepath, char*mode); #else #define my_fopen fopen #endif ```

ecraven commented

2016-11-23 16:30:56 +00:00

(Migrated from github.com)

hm.. so this boils down to either imgui converting internally (thus providing an actual utf-8 interface), or me converting externally, and imgui pretending it supported utf-8 in pathnames, while I only pass it ascii on windows?

ocornut commented

2016-11-27 14:27:01 +00:00

(Migrated from github.com)

I could probably implement the utf-8 to utf-16 conversion + _wfopen() under Windows.
How about OSX, Linux, etc?
Would it make sense and be natural for your app to obtain and pass those filenames in utf-8 in the first place?

I could probably implement the utf-8 to utf-16 conversion + _wfopen() under Windows. How about OSX, Linux, etc? Would it make sense and be natural for your app to obtain and pass those filenames in utf-8 in the first place?

dougbinks commented

2016-11-27 15:14:17 +00:00

(Migrated from github.com)

I think that for OSX and Linux nothing needs to change for dear imgui since on both fopen accepts UTF-8 - though there is a slight complexity on Linux as although most systems are default set up to have a UTF-8 locale not all do. However the app can handle everything required (using iconv for example) since the interface to fopen is always the same, it's how the buffer is interpreted which is changed by the locale.

ocornut commented

2016-11-27 15:28:12 +00:00

(Migrated from github.com)

Thank you Doug. Would MultiByteToWideChar() work with non-MSVC Windows enviroment, or should I test for _MSC_VER?

Thank you Doug. Would MultiByteToWideChar() work with non-MSVC Windows enviroment, or should I test for `_MSC_VER`?

dougbinks commented

2016-11-27 15:36:26 +00:00

(Migrated from github.com)

Sadly this depends on the non-MSVC environment, it's a bit of a mess. I would use _MSC_VER and await contributions from anyone developing with MinGW or PellesC etc (Cygwin should go through a POSIX interface so be like Linux I would hope).

ocornut commented

2016-11-27 15:38:29 +00:00

(Migrated from github.com)

I will use the UTF-8>wchar functions internal to imgui for the _wfopen() path + test with _MSC_VER for now.

ocornut commented

2016-11-27 15:44:17 +00:00

(Migrated from github.com)

Should be done now.
@ecraven @dougbinks @Wohlstand let me know if that looks all ok to you.

Should be done now. @ecraven @dougbinks @Wohlstand let me know if that looks all ok to you.

dougbinks commented

2016-11-27 16:17:21 +00:00

(Migrated from github.com)

Just tested this on Windows 10, VS2015 with a Sanskrit directory for my assets (fonts) and this worked wonderfully. Many thanks!

Wohlstand commented

2016-11-28 11:43:43 +00:00

(Migrated from github.com)

I think, you need to use _WIN32 macro than _MSC_VER, because many custom compilers like MinGW will still have this bug

👍 1

ocornut commented

2016-11-28 14:01:39 +00:00

(Migrated from github.com)

@Wohlstand Doesn't MinGW comes with its own standard library? How does fopen() perform there?

Wohlstand commented

2016-11-28 14:16:57 +00:00

(Migrated from github.com)

@ocornut, MinGW != CygWin, and it's purpose to build native Windows apps, therefore most of things are dealing with Windows's stuff.
Yea, it has own runtime, but in result it is more same as MSVC's by standard WinAPI and fopen / _wfopen / etc. functions. MinGW is able to build application which will work even on Windows 95 (in case of avoiding usage impossible functions on it) while latest versions of MSVC are hates to keep backward compatibility with older Windows versions. MinGW has no fmemopen function like CygWin and unix systems, for example. Main difference between MinGW and MSVC compiler that MinGW faster receiving a most modern C++ standard features (MinGW itself is a native port of GCC for Windows which specifying to it) while MSVC usually doing that with a big delaying. For example, while was MSVC 2013, MinGW in same time was able do a lot of things that MSVC 2015 is almost able to do now.

@ocornut, MinGW != CygWin, and it's purpose to build native Windows apps, therefore most of things are dealing with Windows's stuff. Yea, it has own runtime, but in result it is more same as MSVC's by standard WinAPI and fopen / _wfopen / etc. functions. MinGW is able to build application which will work even on Windows 95 (in case of avoiding usage impossible functions on it) while latest versions of MSVC are hates to keep backward compatibility with older Windows versions. MinGW has no fmemopen function like CygWin and unix systems, for example. Main difference between MinGW and MSVC compiler that MinGW faster receiving a most modern C++ standard features (MinGW itself is a native port of GCC for Windows which specifying to it) while MSVC usually doing that with a big delaying. For example, while was MSVC 2013, MinGW in same time was able do a lot of things that MSVC 2015 is almost able to do now.

dougbinks commented

2016-11-28 14:46:22 +00:00

(Migrated from github.com)

@Wohlstand

See my earlier comment about my advice to @ocornut - if you have a working tested solution to fopen on MinGW then a contribution would be the recommended path.

My own experience with MinGW is that there is sufficient difference in behaviour (especially of win32 POSIX APIs), that testing is needed, especially as the documentation is poor. Many POSIX APIs behave more like the their Unix variants than the Windows VS ones.

@Wohlstand [See my earlier comment](https://github.com/ocornut/imgui/issues/917#issuecomment-263128693) about my advice to @ocornut - if you have a working tested solution to fopen on MinGW then a contribution would be the recommended path. My own experience with MinGW is that there is sufficient difference in behaviour (especially of win32 POSIX APIs), that testing is needed, especially as the documentation is poor. Many POSIX APIs behave more like the their Unix variants than the Windows VS ones.

Wohlstand commented

2016-11-28 15:01:58 +00:00

(Migrated from github.com)

@dougbinks, you seems have very old MinGW version. I using always fresh version (4.9 or 5+) (even 4.7 is same). In all my cases fopen accepts ANSI string for locale and dislikes UTF-8. Only converting into UTF-16 and using _wfopen allows me open file by right UTF-8 path.

My working MinGW example:

INI Processing - INI parser with API like QSettings, but in pure STL

https://github.com/WohlSoft/IniProcessing/tree/master/src
I.e.:
https://github.com/WohlSoft/IniProcessing/blob/master/src/ini_processing.cpp#L338

(included test application is requires a Qt 5, but IniProcessing class itself must don't require any Qt API). All source codes I saving into UTF-8 with no BOM. (MSVC reads UTF-8 code only if BOM presented. MinGW is reads UTF-8 code always).

If you want real test, just make a folder "Вася" (Cyrillic), then iside a folder "κάτι εκεί" (greek), and file "那裡的東西.txt" (traditional Chinese), then try to open it by string "Вася/κάτι εκεί/那裡的東西.txt". Source code file must be saved in UTF-8.

Very small unicode paths test example

https://github.com/Wohlstand/UTF8-fopen
and pre-built Windows binary which built by MinGW and prints a contents of that text file: https://github.com/Wohlstand/UTF8-fopen/releases/tag/1.0

@dougbinks, you seems have very old MinGW version. I using always fresh version (4.9 or 5+) (even 4.7 is same). In all my cases fopen accepts ANSI string for locale and dislikes UTF-8. Only converting into UTF-16 and using _wfopen allows me open file by right UTF-8 path. # My working MinGW example: ## INI Processing - INI parser with API like QSettings, but in pure STL https://github.com/WohlSoft/IniProcessing/tree/master/src I.e.: https://github.com/WohlSoft/IniProcessing/blob/master/src/ini_processing.cpp#L338 (included test application is requires a Qt 5, but IniProcessing class itself must don't require any Qt API). All source codes I saving into UTF-8 with no BOM. (MSVC reads UTF-8 code only if BOM presented. MinGW is reads UTF-8 code always). If you want real test, just make a folder "Вася" (Cyrillic), then iside a folder "κάτι εκεί" (greek), and file "那裡的東西.txt" (traditional Chinese), then try to open it by string "Вася/κάτι εκεί/那裡的東西.txt". Source code file must be saved in UTF-8. ## Very small unicode paths test example https://github.com/Wohlstand/UTF8-fopen and pre-built Windows binary which built by MinGW and prints a contents of that text file: https://github.com/Wohlstand/UTF8-fopen/releases/tag/1.0

dougbinks commented

2016-11-28 15:25:44 +00:00

(Migrated from github.com)

@Wohlstand I think you're misunderstanding me.

I don't have an old MinGW version since I don't have a MinGW version installed at all currently. Hence the request for you to check your proposed solution with imgui and submit a pull request or at least let @ocornut know it works.

My apologies to everyone if I have confused issues.

@Wohlstand I think you're misunderstanding me. I don't have an old MinGW version since I don't have a MinGW version installed at all currently. Hence the request for you to check your proposed solution with imgui and submit a pull request or at least let @ocornut know it works. My apologies to everyone if I have confused issues.

Wohlstand commented

2016-11-28 15:32:30 +00:00

(Migrated from github.com)

@dougbinks, oh, okay, wait a sec (I'll try SDL2 demo and will try to load custom font with locale mixed name)

Wohlstand commented

2016-11-28 15:57:17 +00:00

(Migrated from github.com)

@dougbinks , @ocornut , I just built SDL2 OpenGL2 example by MinGW, and bug still be presented, how I many times noticed before I built this (it will assert on attempt open font in the non-ASCII folder).
But, when I replaced _MSC_VER with _WIN32 in the ImOpenFile function, everything works fine (try ImguiExample-failed.exe (built as-is) and then ImguiExample-fixed.exe (built with my fix))

http://wohlsoft.ru/docs/Software/imgui-mingw-example-449c47c.7z

@dougbinks , @ocornut , I just built SDL2 OpenGL2 example by MinGW, and bug still be presented, how I many times noticed before I built this (it will assert on attempt open font in the non-ASCII folder). But, when I replaced _MSC_VER with _WIN32 in the ImOpenFile function, everything works fine (try ImguiExample-failed.exe (built as-is) and then ImguiExample-fixed.exe (built with my fix)) http://wohlsoft.ru/docs/Software/imgui-mingw-example-449c47c.7z

Flix01 commented

2016-11-28 16:12:12 +00:00

(Migrated from github.com)

I think using _WIN32 is the way to go, and then we can add exceptions for other Windows compilers on request if necessary (and AFAIK _WIN64 defines _WIN32 too).

P.S. @Wohlstand: just tested https://github.com/Wohlstand/UTF8-fopen on Linux: fopen in Mingw behaves the same as cl.exe (=> no UTF8). (However the source code in that repository now uses _WIN32 so it works).

I think using **_WIN32** is the way to go, and then we can add exceptions for other Windows compilers on request if necessary (and AFAIK _WIN64 defines _WIN32 too). **P.S.** @Wohlstand: just tested https://github.com/Wohlstand/UTF8-fopen on Linux: fopen in Mingw behaves the same as cl.exe (=> no UTF8). (However the source code in that repository now uses _WIN32 so it works).

Wohlstand commented

2016-11-28 16:14:13 +00:00

(Migrated from github.com)

@Flix01, _WIN32 is always defined even Windows build for Win64, _WIN64 means build is targeted to pure x64.

Flix01 commented

2016-11-28 16:36:49 +00:00

(Migrated from github.com)

@Wohlstand, yep!

Another (slightly related) potential problem is that AFAIK a lot of the stb_xxx files use plain f_open(...) internally. So maybe imgui calls some of these methods (in stb_truetype ?).

@Wohlstand, yep! Another (slightly related) potential problem is that AFAIK a lot of the stb_xxx files use plain f_open(...) internally. So maybe imgui calls some of these methods (in stb_truetype ?).

ocornut commented

2016-11-28 19:31:15 +00:00

(Migrated from github.com)

Should be ok now.

Flix01 commented

2017-01-13 18:35:24 +00:00

(Migrated from github.com)

As I've posted here, as far as I can see in imgui_internals.h:

IMGUI_API FILE*         ImOpenFile(const char* filename, const char* file_open_mode);

it's declared but never defined (in imgui.cpp it's called ImFileOpen).

That makes it inaccessible from imgui_internals.h.

As I've posted [here](https://github.com/ocornut/imgui/commit/104b3810237d4a959ff421fea14c6fc0fd8d0b79), as far as I can see in _imgui_internals.h_: ```c++ IMGUI_API FILE* ImOpenFile(const char* filename, const char* file_open_mode); ``` it's declared but never defined (in _imgui.cpp_ it's called **ImFileOpen**). That makes it inaccessible from _imgui_internals.h_.

ocornut commented

2017-01-14 12:49:28 +00:00

(Migrated from github.com)

Correct, sorry for the mismatching declaration. The correct name is ImFileOpen() and I have renamed ImLoadFileToMemory to ImFileLoadMemory for consistency. (It might affect some code using those internals but the renaming will be obvious)

Flix01 commented

2017-01-14 15:23:58 +00:00

(Migrated from github.com)

Thanks!

Rows
Columns

Windows's fopen() Assertion if path contains utf-8 #917

My working MinGW example:

INI Processing - INI parser with API like QSettings, but in pure STL

Very small unicode paths test example