# What is state of the art: Unicode file names on Windows

**URL:** <https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821>\
**Category:** Development\
**Created:** [September 25, 2019, 2:23pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821 "2019-09-25T14:23:23Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 16, 2019, 1:58am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/41 "2019-10-16T01:58:44Z")

</div>

Are there places in the public API of VTK where char\* is specified instead of a vtkDataArray or void\* parameter?

---

<div class="post-metadata">

**Author:** ![dgobbi](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dgobbi/32/18_2.png) [@dgobbi](https://discourse.vtk.org/u/dgobbi)\
**Post date:** [October 16, 2019, 5:24am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/42 "2019-10-16T05:24:00Z")

</div>

The only place I’m aware of where “char \*” is used for data is the interface to vtkCharArray. I don’t know if vtkCharArray is used much outside of legacy code.

The Python wrappers check UTF8 validity via `PyUnicode_FromStringAndSize()`, which raises a Python exception if the input isn’t valid.

> [@toddy](#):
>
> Why are Python byte arrays passed through string parameters at all?

Python byte arrays are how Python handles strings in other encodings. For example, if someone is using VTK-Python on a Windows box with CP1252 as the active code page, then right now the only way to specify a filename with accented characters is to use Python to generate a bytes() object with CP1252 encoding and pass that bytes() object to SetFileName().

If VTK goes full UTF8, then this will no longer be necessary, since VTK will always use the Unicode versions of the Windows APIs.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 16, 2019, 5:34am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/43 "2019-10-16T05:34:59Z")

</div>

Great. So there is no impediment to switching to a requirement that all string type parameters are UTF-8 encoded.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [January 30, 2020, 11:13pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/44 "2020-01-30T23:13:39Z")

</div>

@efahl Finally I can invite you to try this again using utf-8 text for file/folder names and building VTK from master. If you can think of any other test cases please try them. Let us know how it goes. 😄

---

<div class="post-metadata">

**Author:** ![efahl](https://discourse.vtk.org/user_avatar/discourse.vtk.org/efahl/32/1087_2.png) [@efahl](https://discourse.vtk.org/u/efahl)\
**Post date:** [February 3, 2020, 8:04pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/45 "2020-02-03T20:04:24Z")

</div>

@toddy, tried it with several files, works great, thanks!

```auto
vtkPNGReader (000002508818E8E0)
...
  FileName: 三维图片.png
...
  Internal File Name: 三维图片.png

```

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [February 11, 2020, 12:06am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/46 "2020-02-11T00:06:31Z")

</div>

> [@toddy](#):
>
> b) they need to be specifically converted for some function calls (e.g. UTF-16 for the Windows API)

FYI, since Windows10 May 2019 Update, the [process code page can be set to UTF-8 in the application manifest](https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page).

We can use plain `fopen` API with UTF-8 file names on Windows the same way as on Mac and Linux. There is no need to convert filenames to UTF-16 (`WCHAR`) and use `_wfopen` anymore or use vtksys::SystemTools::Fopen. This is convenient because we don’t need to make hundreds of changes in VTK and invasive patches in third-party libraries.

I’ve tested it and works well. I only had to add the manifest file to the list of source files in CMake `add_executable` (in CTK, I’ve added [`add_executable_utf8` macro](https://github.com/commontk/CTK/pull/902/files#diff-9fa77245970f38e9202992bc68b23649R13-R19) for this).

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [February 11, 2020, 1:12am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/47 "2020-02-11T01:12:47Z")

</div>

@lassoan The necessary changes have already been made in VTK.

How does the manifest code page setting work for Windows \< 10?  
 ![image](https://discourse.vtk.org/uploads/default/original/2X/e/e250ba83fd9e1b57993a6439940f8039b7bcfeac.png)

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [February 11, 2020, 1:44am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/48 "2020-02-11T01:44:02Z")

</div>

Code page setting in the manifest has no effect on pre-Windows10. So, the application still works well except it cannot read/write files that have special characters in their names. Fortunately for us, Microsoft took on the task of migrating users to current Windows versions, so we don’t need to worry about supporting earlier versions.

> [@toddy](#):
>
> The necessary changes have already been made in VTK.

Thanks for your efforts. Since UTF-8 is not the default code page on Windows applications yet, your changes are still useful, as they make things a bit simpler for VTK users. Maybe in a couple of years, when UTF-8 will become the default code page on Windows and then we don’t need to use a custom compatibility layer anymore (similarly to how vtkStdString served its time and is now ready to be retired).

Has everything been merged into VTK master? I still see lots of plain `fopen` in VTK master.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [February 11, 2020, 2:10am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/49 "2020-02-11T02:10:47Z")

</div>

Everything has been merged into master. No changes were made to 3rd party libraries.

It is great that MS are finally addressing this issue, but it’s not a silver bullet for legacy systems. 26% of users are still on Windows 7 and I note that the UK national health service was substantially affected by the WannaCry virus in 2018 because so many of their machines were running Windows XP.

---

<div class="post-metadata">

**Author:** ![dr\_ppetrov](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dr_ppetrov/32/2121_2.png) [@dr\_ppetrov](https://discourse.vtk.org/u/dr_ppetrov)\
**Post date:** [September 22, 2020, 11:11am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/51 "2020-09-22T11:11:11Z")

</div>

Hello Mr A.Lasso,  
I managed to find the macro and the actual manifest file. It seems 1-to-1 with the mSDN link you provided except for the schema version. (yours is: xmlns=“urn:schemas-microsoft-com:asm.v3”  
while on MSDN is xmlns=“urn:schemas-microsoft-com:asm.v1”

not sure if that’s causing the following warning I get in VisualStudio:

Severity Code Description Project File Line Suppression State  
Warning 81010002 Unrecognized Element “activeCodePage” in namespace “[http://schemas.microsoft.com/SMI/2019/WindowsSettings](http://schemas.microsoft.com/SMI/2019/WindowsSettings)”.

Any idea why I am getting this!? I am using VS 2017 and latest cmake+vtk9.0.1.

p.s. Slicer3D is not supporting unicode on win when I use the installer

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [September 22, 2020, 3:52pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/52 "2020-09-22T15:52:21Z")

</div>

> [@dr\_ppetrov](#):
>
> Any idea why I am getting this!? I am using VS 2017 and latest cmake+vtk9.0.1.

You need to use a more recent windows SDK & toolchain version.

> [@dr\_ppetrov](#):
>
> Slicer3D is not supporting unicode on win when I use the installer

“UTF8 everywhere” switch happened in Slicer-4.11 version, so you need to use a Slicer Preview Release.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [September 23, 2020, 1:29am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/53 "2020-09-23T01:29:55Z")

</div>

If you’re using VTK 9 you shouldn’t need the macro at all.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [September 23, 2020, 3:51am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/54 "2020-09-23T03:51:57Z")

</div>

Good point, this may still be useful for applications that must run on older Windows versions.

On current Windows versions, using the UTF8 code page allows making system calls or use any libraries (not just VTK) without worrying about encodings.

---

<div class="post-metadata">

**Author:** ![albert.liu](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/a/edb3f5/32.png) [@albert.liu](https://discourse.vtk.org/u/albert.liu)\
**Post date:** [November 9, 2020, 3:25pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/55 "2020-11-09T15:25:01Z")

</div>

I’m using vtk9.0.1,but the vtkSTLReader still can not handle chinese file name on win10-1903.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [November 9, 2020, 5:49pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/56 "2020-11-09T17:49:38Z")

</div>

I’ve just tested if vtkSTLReader works with arbitrary characters in the filename and I confirm that it works well. Probably you have forgot to enable UTF8 code page in your application manifest. You can check that the code page is set correctly as shown in [this test](https://github.com/Slicer/Slicer/blob/43300c7b9a3b27265c2bc736693cb881f05e3ea9/Libs/MRML/Widgets/Testing/qMRMLUtf8Test1.cxx#L86-L94).

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [November 9, 2020, 10:46pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/57 "2020-11-09T22:46:00Z")

</div>

@albert.liu Please post a sample of your code.

VTK-9 can read non-Ascii filenames in any version of Windows. Simply provide the filename as a utf8 encoded string.

@lassoan In VTK-9 the code page setting is irrelevant and not required.

---

<div class="post-metadata">

**Author:** ![albert.liu](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/a/edb3f5/32.png) [@albert.liu](https://discourse.vtk.org/u/albert.liu)\
**Post date:** [November 10, 2020, 1:53am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/58 "2020-11-10T01:53:01Z")

</div>

UINT activeCodePage = GetACP();  
I trace this line in my project, the activeCodePage is 936.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [November 10, 2020, 2:38am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/59 "2020-11-10T02:38:35Z")

</div>

> [@toddy](#):
>
> VTK-9 can read non-Ascii filenames in any version of Windows. Simply provide the filename as a utf8 encoded string.
> 
> @lassoan In VTK-9 the code page setting is irrelevant and not required.

For VTK9 library the application code page does not matter, but for the application it still does. Since in an application you need to use C runtime and other APIs, not just VTK (for example to get the filename from a command-line argument, console input, or GUI), you still need to set the application’s code page to UTF8 to avoid manual conversions between various string encodings.

> [@albert.liu](#):
>
> UINT activeCodePage = GetACP();  
> I trace this line in my project, the activeCodePage is 936

This means that you don’t use UTF8 code page. You either need to convert the filename from “simplified Chinese” (CP936) to UTF8 before you pass it to VTK; or change your application manifest to use UTF8 code page (because then you can already get the filename as UTF8).

---

<div class="post-metadata">

**Author:** ![liuzhongshu](https://discourse.vtk.org/user_avatar/discourse.vtk.org/liuzhongshu/32/6921_2.png) [@liuzhongshu](https://discourse.vtk.org/u/liuzhongshu)\
**Post date:** [September 7, 2023, 1:25am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/60 "2023-09-07T01:25:38Z")

</div>

I am testing [Activiz.NET](http://Activiz.NET) 9.x, and found objReader can not open file with non-ascii path, after setting the manifest with a UTF-8 activeCodePage, the problem is gone.

But the same obj file can be opened with [Activiz.NET](http://Activiz.NET) old version without activeCodePage setting.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [September 7, 2023, 1:57am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/61 "2023-09-07T01:57:38Z")

</div>

Can you be more specific? What is the path/file name? Which version is the “old” version.

VTK 8.2+ expects/handles utf-8 encoded text.  
For Activiz that means setting the UTF-8 codepage to do the UTF16 to UTF8 conversion on Windows.

[Previous page](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821.md?page=2)

[Next page](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821.md?page=4)