在字符串、u16string & 之间转换u32字符串

Convert between string, u16string amp; u32string(在字符串、u16string amp; 之间转换u32字符串)

问题描述

我一直在寻找一种在 Unicode 字符串类型之间进行转换的方法，结果遇到了 这个方法.不仅没有完全理解方法(没有评论)而且文章暗示将来会有更好的方法.

I've been looking for a way to convert between the Unicode string types and came across this method. Not only do I not completely understand the method (there are no comments) but also the article implies that in future there will be better methods.

如果这是最好的方法，请您指出是什么让它起作用，如果不是，我想听听关于更好方法的建议.

If this is the best method, could you please point out what makes it work, and if not I would like to hear suggestions for better methods.

推荐答案

mbstowcs() 和 wcstombs() 不一定转换为 UTF-16 或 UTF-32，它们转换为 wchar_t 和任何区域设置 wchar_t 编码.所有 Windows 语言环境都使用两字节 wchar_t 和 UTF-16 作为编码，但其他主要平台使用 4 字节 wchar_t 和 UTF-32(甚至非- 某些语言环境的 Unicode 编码).一个只支持单字节编码的平台甚至可以有一个一字节的 wchar_t 并且编码因地区而异.所以 wchar_t 在我看来对于可移植性和 Unicode 来说是一个糟糕的选择.*

mbstowcs() and wcstombs() don't necessarily convert to UTF-16 or UTF-32, they convert to wchar_t and whatever the locale wchar_t encoding is. All Windows locales uses a two byte wchar_t and UTF-16 as the encoding, but the other major platforms use a 4-byte wchar_t with UTF-32 (or even a non-Unicode encoding for some locales). A platform that only supports single-byte encodings could even have a one byte wchar_t and have the encoding differ by locale. So wchar_t seems to me to be a bad choice for portability and Unicode. *

C++11 中引入了一些更好的选项；std::codecvt 的新特化、新的 codecvt 类和一个新模板，可以非常方便地使用它们进行转换.

Some better options have been introduced in C++11; new specializations of std::codecvt, new codecvt classes, and a new template to make using them for conversions very convienent.

首先使用 codecvt 的新模板类是 std::wstring_convert.创建 std::wstring_convert 类的实例后，您可以轻松地在字符串之间进行转换:

First the new template class for using codecvt is std::wstring_convert. Once you've created an instance of a std::wstring_convert class you can easily convert between strings:

std::wstring_convert<...> convert; // ... filled in with a codecvt to do UTF-8 <-> UTF-16
std::string utf8_string = u8"This string has UTF-8 content";
std::u16string utf16_string = convert.from_bytes(utf8_string);
std::string another_utf8_string = convert.to_bytes(utf16_string);

为了进行不同的转换，您只需要不同的模板参数，其中之一是 codecvt facet.以下是一些易于与 wstring_convert 一起使用的新方面:

In order to do different conversion you just need different template parameters, one of which is a codecvt facet. Here are some new facets that are easy to use with wstring_convert:

std::codecvt_utf8_utf16<char16_t> // converts between UTF-8 <-> UTF-16
std::codecvt_utf8<char32_t> // converts between UTF-8 <-> UTF-32
std::codecvt_utf8<char16_t> // converts between UTF-8 <-> UCS-2 (warning, not UTF-16! Don't bother using this one)

使用这些的例子:

std::wstring_convert<std::codecvt_utf8_utf16<char16_t>,char16_t> convert;
std::string a = convert.to_bytes(u"This string has UTF-16 content");
std::u16string b = convert.from_bytes(u8"blah blah blah");

新的 std::codecvt 特化有点难使用，因为它们有一个受保护的析构函数.为了解决这个问题，您可以定义一个具有析构函数的子类，或者您可以使用 std::use_facet 模板函数来获取现有的 codecvt 实例.此外，这些特化的一个问题是您不能在 Visual Studio 2010 中使用它们，因为模板特化不适用于 typedef 类型，并且编译器将 char16_t 和 char32_t 定义为 typedef.下面是定义自己的 codecvt 子类的示例:

The new std::codecvt specializations are a bit harder to use because they have a protected destructor. To get around that you can define a subclass that has a destructor, or you can use the std::use_facet template function to get an existing codecvt instance. Also, an issue with these specializations is you can't use them in Visual Studio 2010 because template specialization doesn't work with typedef'd types and that compiler defines char16_t and char32_t as typedefs. Here's an example of defining your own subclass of codecvt:

template <class internT, class externT, class stateT>
struct codecvt : std::codecvt<internT,externT,stateT>
{ ~codecvt(){} };

std::wstring_convert<codecvt<char16_t,char,std::mbstate_t>,char16_t> convert16;
std::wstring_convert<codecvt<char32_t,char,std::mbstate_t>,char32_t> convert32;

char16_t 特化在 UTF-16 和 UTF-8 之间转换.char32_t 特化，UTF-32 和 UTF-8.

The char16_t specialization converts between UTF-16 and UTF-8. The char32_t specialization, UTF-32 and UTF-8.

请注意，C++11 提供的这些新转换不包括任何直接在 UTF-32 和 UTF-16 之间转换的方法.相反，您只需要组合 std::wstring_convert 的两个实例.

Note that these new conversions provided by C++11 don't include any way to convert directly between UTF-32 and UTF-16. Instead you just have to combine two instances of std::wstring_convert.

***** 我想我会添加一个关于 wchar_t 及其用途的注释，以强调为什么它通常不应该用于 Unicode 或可移植的国际化代码.以下是我的回答的简短版本 https://stackoverflow.com/a/11107667/365496

***** I thought I'd add a note on wchar_t and its purpose, to emphasize why it should not generally be used for Unicode or portable internationalized code. The following is a short version of my answer https://stackoverflow.com/a/11107667/365496

wchar_t 的定义使得任何语言环境的 char 编码都可以转换为 wchar_t，其中每个 wchar_t 只代表一个代码点:

wchar_t is defined such that any locale's char encoding can be converted to wchar_t where every wchar_t represents exactly one codepoint:

类型 wchar_t 是一种独特的类型，其值可以表示支持的语言环境 (22.3.1) 中指定的最大扩展字符集的所有成员的不同代码.-- [basic.fundamental] 3.9.1/5

Type wchar_t is a distinct type whose values can represent distinct codes for all members of the largest extended character set specified among the supported locales (22.3.1). -- [basic.fundamental] 3.9.1/5

这不要求 wchar_t 足够大以同时表示来自所有语言环境的任何字符.也就是说，用于 wchar_t 的编码可能因地区而异.这意味着您不一定使用一种语言环境将字符串转换为 wchar_t，然后使用另一种语言环境将其转换回 char.

This does not require that wchar_t be large enough to represent any character from all locales simultaneously. That is, the encoding used for wchar_t may differ between locales. Which means that you cannot necessarily convert a string to wchar_t using one locale and then convert back to char using another locale.

因为这似乎是 wchar_t 在实践中的主要用途，所以您可能想知道它有什么用处.

Since that seems to be the primary use in practice for wchar_t you might wonder what it's good for if not that.

wchar_t 的最初意图和目的是通过定义它来简化文本处理，使得它需要从字符串的代码单元到文本字符的一对一映射，从而允许使用相同的简单算法与 ascii 字符串一起使用其他语言.

The original intent and purpose of wchar_t was to make text processing simple by defining it such that it requires a one-to-one mapping from a string's code-units to the text's characters, thus allowing the use of same simple algorithms used with ascii strings to work with other languages.

不幸的是，wchar_t 的要求假设字符和代码点之间存在一对一的映射来实现这一点.Unicode 打破了这个假设，因此您也不能安全地将 wchar_t 用于简单的文本算法.

Unfortunately the requirements on wchar_t assume a one-to-one mapping between characters and codepoints to achieve this. Unicode breaks that assumption, so you can't safely use wchar_t for simple text algorithms either.

这意味着便携式软件不能将 wchar_t 用作语言环境之间文本的通用表示，也不能使用简单的文本算法.

This means that portable software cannot use wchar_t either as a common representation for text between locales, or to enable the use of simple text algorithms.

不管怎样，对于可移植代码来说并不多.如果定义了 __STDC_ISO_10646__，则 wchar_t 的值直接表示在所有语言环境中具有相同值的 Unicode 代码点.这使得进行前面提到的区域间转换是安全的.但是，您不能仅仅依靠它来决定您可以以这种方式使用 wchar_t，因为尽管大多数 unix 平台都定义了它，但即使 Windows 在所有语言环境中使用相同的 wchar_t 语言环境，Windows 也不会这样做.

Not much, for portable code anyway. If __STDC_ISO_10646__ is defined then values of wchar_t directly represent Unicode codepoints with the same values in all locales. That makes it safe to do the inter-locale conversions mentioned earlier. However you can't rely only on it to decide that you can use wchar_t this way because, while most unix platforms define it, Windows does not even though Windows uses the same wchar_t locale in all locales.

Windows 没有定义 __STDC_ISO_10646__ 的原因我认为是因为 Windows 使用 UTF-16 作为其 wchar_t 编码，并且因为 UTF-16 使用代理对来表示大于 U+FFFF 的代码点，即表示 UTF-16 不满足 __STDC_ISO_10646__ 的要求.

The reason Windows doesn't define __STDC_ISO_10646__ I think is because Windows use UTF-16 as its wchar_t encoding, and because UTF-16 uses surrogate pairs to represent codepoints greater than U+FFFF, which means that UTF-16 doesn't satisfy the requirements for __STDC_ISO_10646__.

对于平台特定的代码 wchar_t 可能更有用.它本质上在 Windows 上是必需的(例如，某些文件在不使用 wchar_t 文件名的情况下根本无法打开)，尽管据我所知，Windows 是唯一的平台(所以也许我们可以将 wchar_t 视为Windows_char_t").

For platform specific code wchar_t may be more useful. It's essentially required on Windows (e.g., some files simply cannot be opened without using wchar_t filenames), though Windows is the only platform where this is true as far as I know (so maybe we can think of wchar_t as 'Windows_char_t').

事后看来，wchar_t 显然对于简化文本处理或作为独立于语言环境的文本的存储没有用处.可移植代码不应试图将其用于这些目的.

In hindsight wchar_t is clearly not useful for simplifying text handling, or as storage for locale independent text. Portable code should not attempt to use it for these purposes.

这篇关于在字符串、u16string & 之间转换u32字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！