# What is state of the art: Unicode file names on Windows

**URL:** <https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821>\
**Category:** Development\
**Created:** [September 25, 2019, 2:23pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821 "2019-09-25T14:23:23Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [October 13, 2019, 1:55am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/21 "2019-10-13T01:55:48Z")

</div>

> [@toddy](#):
>
> I see the KWSYS branch in the repository. Am I supposed to make the SystemTools changes in that branch and then run **update.sh** to merge them into VTK master?

You’ll need to fork the main `kwsys` repository (linked in the MR) and make an MR for it. Brad will review it there. I suspect the changes aren’t as straightforward as they seem; that library supports dozens of platforms and I don’t know if `std::codecvt` stuff can be assumed to work on all of them. Once it is merged into `master` there, running the `update.sh` will bring the changes into your branch.

> [@lassoan](#):
>
> vtkUnicodeString should be deprecated and wherever it was used before, new APIs should be added that use plain strings instead.

Agreed. I’d like to see `vtkStdString` gone as well (though deprecation is OK, I don’t think we should keep it around; it’s use has been discouraged for years).

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 13, 2019, 3:38am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/22 "2019-10-13T03:38:17Z")

</div>

Hmm. In that case I’m going to introduce new methods WideToUTF8(), UTF8ToWide() and FileOpenUTF8()

---

<div class="post-metadata">

**Author:** ![efahl](https://discourse.vtk.org/user_avatar/discourse.vtk.org/efahl/32/1087_2.png) [@efahl](https://discourse.vtk.org/u/efahl)\
**Post date:** [October 13, 2019, 12:33pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/23 "2019-10-13T12:33:09Z")

</div>

> [@toddy](#):
>
> Great. I mostly just wanted to know which VTK class you were using.

These are the specific classes that started me on this quest:

```
for reader_class in vtk.vtkBMPReader, vtk.vtkJPEGReader, vtk.vtkPNGReader, vtk.vtkTIFFReader, vtk.vtkPNMReader:
     reader = reader_class()
     reader.SetFileName(file_name)
     reader.Update()
```

It appears that David Gobbi’s DICOM reader already handles Unicode properly, as including ‘vtkDICOMReader’ in the above list simply complains appropriately about invalid file contents and lists out the name as expected in the error message.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 14, 2019, 9:23am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/24 "2019-10-14T09:23:58Z")

</div>

Since images are all binary file types, I’m not worried about handling the contents just the file name.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [October 14, 2019, 12:47pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/25 "2019-10-14T12:47:13Z")

</div>

Many file formats that VTK supports are text-based. Ideally, for each file format you would check if there is any guidance on how to handle special characters.

If there is guidance (e.g., for DICOM and XML based formats), then you need to implement that.

If there is no guidance, then you can define a sensible behavior. You can write as UTF-8, or if that breaks commonly used parsers then force ASCII. For reading, you may use some heuristics, such as checking BOM and/or check if the content can be successfully interpreted as UTF-8. Some files are mixed text/binary/base64, so you may need to extract text components before you are trying to guess its encoding.

These should be all doable, but requires some effort. You cannot ignore the file content.

---

<div class="post-metadata">

**Author:** ![efahl](https://discourse.vtk.org/user_avatar/discourse.vtk.org/efahl/32/1087_2.png) [@efahl](https://discourse.vtk.org/u/efahl)\
**Post date:** [October 14, 2019, 3:52pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/26 "2019-10-14T15:52:44Z")

</div>

> [@toddy](#):
>
> Since images are all binary file types, I’m not worried about handling the contents just the file name.

Exactly, I’m assuming that the only bug to be resolved here is wrt the file paths themselves. Any text-based-image readers are already doing whatever is appropriate for the content, or they have bugs orthogonal to this issue.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [October 14, 2019, 4:32pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/27 "2019-10-14T16:32:38Z")

</div>

> [@efahl](#):
>
> Any text-based-image readers are already doing whatever is appropriate for the content, or they have bugs orthogonal to this issue.

VTK was not implemented with UTF-8 encoding in mind, so without doing any code review or testing, it is more reasonable to assume that things don’t work correctly with UTF-8 content.

Just a few examples of why things may break: File path manipulation methods (get filename from paths, etc.) may not work correctly for some UTF-8-encoded paths (as you may have ‘/’ character as part of a unicode character). XML files written by VTK don’t contain declaration, so readers can only guess what encoding is used. Filenames are often saved into file headers, therefore by using UTF-8 file names, you introduce UTF-8 content to files that did not have it before.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [October 14, 2019, 4:54pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/28 "2019-10-14T16:54:57Z")

</div>

> [@lassoan](#):
>
> File path manipulation methods (get filename from paths, etc.) may not work correctly for some UTF-8-encoded paths (as you may have ‘/’ character as part of a unicode character).

No? UTF-8 was designed specifically to _not_ have that problem. Specifically here, any ‘/’ byte in a UTF-8 encoded sequence is guaranteed to be the ‘/’ codepoint. I suspect VTK is actually very UTF-8 safe. The main issues come up with assuming byte length == display length and the like, splitting strings, and more. _Those_ are the operations that need to be looked at for UTF-8 safety.

---

<div class="post-metadata">

**Author:** ![dgobbi](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dgobbi/32/18_2.png) [@dgobbi](https://discourse.vtk.org/u/dgobbi)\
**Post date:** [October 14, 2019, 5:58pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/29 "2019-10-14T17:58:13Z")

</div>

Yes, ASCII will pass unchanged through UTF-8, and any non-ASCII octets in UTF-8 have the high bit set, so they are never mistaken for ASCII. This is in stark contrast to ISO-2022-JP, Shift-JIS, GB18030, etc etc which cause tons of headaches.

I’m hugely in favor of using UTF-8 everywhere.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [October 14, 2019, 6:19pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/30 "2019-10-14T18:19:23Z")

</div>

OK, that’s good that UTF-8 does not have issues specifically with path splitting (I remembered something like that but I could not find a reference quickly).

> [@ben.boeckel](#):
>
> I suspect VTK is actually very UTF-8 safe

This may or may not be the case. We should add tests to confirm.

It is awesome that this activity has started and even not everything will work correctly at first, the issues can be fixed as they are discovered.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 14, 2019, 9:05pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/31 "2019-10-14T21:05:55Z")

</div>

> [@lassoan](#):
>
> Some files are mixed text/binary/base64, so you may need to extract text components before you are trying to guess its encoding.

The only way that mixed content can be handled reliably is if the binary data is base64 encoded. Then the whole document is utf-8 encoded.

> [@lassoan](#):
>
> These should be all doable, but requires some effort. You cannot ignore the file content.

In general yes, but I was referring to the file readers listed by @efahl in my comment.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 14, 2019, 9:17pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/32 "2019-10-14T21:17:25Z")

</div>

@lassoan “/” has ASCII code 47 which is unchanged by utf8 encoding. That’s the beauty of utf8 with respect to altering/finding file paths and extensions etc. Anything within ANSI 128 is encoded as is. This was actually a request of american experts, so that pre-existing documents, written in plain english, would contain the exact same number of bytes and require no change in format. Also no need for a byte order mark, BOM, with utf8.

If VTK were not already mostly utf8 safe, I think you would have encountered many issues from Linux/Mac users long ago.

@ben.boeckel I looked through some of the string splitting and so far it appears to be a non-issue.

Text display issues (lengths etc) only need to be handled/converted close to where they are displayed/sorted. Otherwise they can just flow through VTK untouched.

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 14, 2019, 11:53pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/33 "2019-10-14T23:53:32Z")

</div>

I suspect the Python and Java wrappers cannot handle a std:string in the public API, so either **vtkStdString** needs to be retained or replaced with **const char** \* parameters.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [October 15, 2019, 12:10am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/34 "2019-10-15T00:10:36Z")

</div>

Seems to work fine for this call in `Filters/Modeling/Testing/Python/HyperScalarBar.py`:

```python
scalarBar.SetLabelFormat("%-#6.3f")`

```

given this:

```auto
Charts/Core/vtkAxis.h: virtual void SetLabelFormat(const std::string &fmt);

```

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [October 15, 2019, 12:11am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/35 "2019-10-15T00:11:31Z")

</div>

Hmm. That might not be the right `LabelFormat`. In any case, the wrappers _should_ support it if they don’t already. @dgobbi?

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 15, 2019, 12:48am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/36 "2019-10-15T00:48:10Z")

</div>

That’s actually a **const char** \* parameter  
from _vtkScalarBarActor.h_

```
  //@{
  /**
   * Set/Get the format with which to print the labels on the scalar
   * bar.
   */
  vtkSetStringMacro(LabelFormat);
  vtkGetStringMacro(LabelFormat);
  //@}

#define vtkSetStringMacro(name) \
virtual void Set##name (const char* _arg) \
```

---

<div class="post-metadata">

**Author:** ![dgobbi](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dgobbi/32/18_2.png) [@dgobbi](https://discourse.vtk.org/u/dgobbi)\
**Post date:** [October 15, 2019, 1:29am UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/37 "2019-10-15T01:29:30Z")

</div>

The Python and Java wrappers accept std::string, and they accept utf8 encoded strings. Can’t remember what they do with invalid strings, I’ll check and reply in this thread.

---

<div class="post-metadata">

**Author:** ![dgobbi](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dgobbi/32/18_2.png) [@dgobbi](https://discourse.vtk.org/u/dgobbi)\
**Post date:** [October 15, 2019, 1:53pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/38 "2019-10-15T13:53:13Z")

</div>

Python wrapper behavior with respect to utf8:

- Python str() objects are converted to utf8 when passed to VTK
- Python bytes() objects are passed as-is to VTK
- Each VTK std::string (and const char \*) is checked by the Python wrappers and,  
– If it is valid utf8, it is converted to Python str()  
– If it is not valid utf8, it is converted to Python bytes()  
– If a “const char \*” is nullptr, it is converted to Python “None”

In other words, when VTK uses utf-8 std::string, there is direct correspondence between std::string and str(). This assumes Python 3.x, see [Wrapping/Python/README.md](https://gitlab.kitware.com/vtk/vtk/blob/master/Wrapping/Python/README.md) for more info.

The Java wrappers convert Java strings to utf8 before passing them to VTK, and attempt to decode VTK strings as utf8 (with undefined behavior if they are not utf8).

---

<div class="post-metadata">

**Author:** ![toddy](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/t/edb3f5/32.png) [@toddy](https://discourse.vtk.org/u/toddy)\
**Post date:** [October 15, 2019, 11:11pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/39 "2019-10-15T23:11:53Z")

</div>

How is utf8 validity determined? Unless binary data (bytes) are converted to base64, I foresee a lot of problems handling that string data in VTK.

Why are Python byte arrays passed through string parameters at all? Shouldn’t they all be converted to utf8? I would have thought binary data parameters would be unsigned char.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [October 15, 2019, 11:20pm UTC](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821/40 "2019-10-15T23:20:23Z")

</div>

Binary data gets passed around as `vtkDataArray` or the like usually. Or `void*`.

Binary data should never be passed around as strings (one of the many problems with C’s `char*` representation for strings).

[Previous page](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821.md?page=1)

[Next page](https://discourse.vtk.org/t/what-is-state-of-the-art-unicode-file-names-on-windows/1821.md?page=3)