Talk:Unicode

The lead claims that there are currently 149 186 characters in the Standard. That's confusing! Is that actual characters or does it include unprintable code points? I know what a code point is, my point is that the lead shouldn't confuse code points with characters. (I also argue that a "control character" isn't 'really' a character, not a grapheme, but that's a fight for somewhere else.) Writing about Unicode without an early clear explanation of what a code point is, is -I think- awful pedagogy. In fact, I don't think code point - a fundamental aspect of Unicode - is even defined in the article!!!! Wow, just wow.

I also would like someone to verify that Unicode has characters for color. I believe that's wrong/false/misleading. I am aware that certain emoji can be modified by a code point to change some of its color. As far as I know, this is only true with a very small set of code points, and a very very small set of colors (I don't actually know if the colors are well-defined, I'd expect so, but...). These aren't colors, but are color modifiers for those other code points. 174.130.71.156 (talk) 16:00, 13 December 2022 (UTC)Reply

There are no color defining codes in Unicode but there are names of characters that specify a color if displayed on a color device. Searching the word color in the article shows some possibly confusing text about color but nothing outright wrong.

This article leaves a lot to be desired, if you wish to make changes, you should. It's a wiki after all. SchmuckyTheCat (talk) 05:43, 15 December 2022 (UTC)Reply

• There are two Variation Selectors (U+FE0E and U+FE0F) which specify whether an Emoji should be ideally displayed in color or black and white, but other than that, there are no color specifications in Unicode. The term "character" and "code point" are specified in the Unicode Standard, and if you feel that the coverage here is inadequate in conveying the meaning of those terms, I absolutely encourage you to contribute content to better reflect their technical specification. For the record, any code point defined beyond "Not A Character" or "Reserved" is a "character". This means control characters and whitespace are all considered characters in Unicode, just like a letter in an alphabet, a Kanji with On and Kun readings, or a mathematical symbol. VanIsaac, GHTV ^cont_WpWS 06:18, 15 December 2022 (UTC)Reply

The offending sentence is:"The Unicode standard defines three and several other encodings exist, all in practice variable-length encodings." (Sure, you could strain to interpret that to mean "all but UTF-32", but let's keep it clear. It clearly implies all encodings are variable length. Wikipedia's own article on UTF-32 says it is fixed length. (Because it only needs to use 21 of the 32 bits for Unicode code points, it is very inefficient (and rarely used, afaik). But rarely used is not the same as "doesn't exist", and "all are variable" clearly implies it doesn't exist. I'd have to look again, are there really 3 variable Unicode encodings? I can only think of UTF-8 and UTF-16. (and some others that afaik are not "defined" in the Unicode standard (like GB18030), or that are obsolete (like UTF-7).) Replace "all" with "all common encodings" or something similar, and mention UTF-32.174.130.71.156 (talk) 11:43, 15 December 2022 (UTC)Reply

I think the intended meaning of this was that even if code points are fixed-size, modern Unicode is effectively variable-width, as what the user thinks is a "character" sometimes needs multiple code points.Spitzak (talk) 16:40, 15 December 2022 (UTC)Reply

Yes, Unicode includes both combining characters and precomposed characters, e.g., <U+0061 “a” latin small letter a> <U+0308 "¨" combining diaeresis> is equivalent to <U+00E4 "ä" latin small letter A with diaeresis>. Further, some glyphs exist at multiple code points for historical reasons. There is a discussion of cannonical forms in the Unicode standard. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 21:57, 15 December 2022 (UTC)Reply

It seems odd to me to describe code points as "fixed size". They're just an abstract number. It's when you encode (or store) the code points that you get variable lengths, at least for UTF-8, UTF-EBCDIC, and UTF-16 as described in the article. I think combining characters are a red herring for this discussion. DRMcCreedy (talk) 23:10, 15 December 2022 (UTC)Reply

The Unicode standard does restrict the number of code points, so describing them as as fixed length 21-bit or 32-bit data is reasonable. Spitzak is referring to characters, which indeed are variable length, a separate issue from the length of an encoded code point that does deserve mention. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 17:14, 16 December 2022 (UTC)Reply

I object to the reversal by Peter M. Brown, citing WP:ITALICTITLE inappropriately. I'd say that the name, a noun, should not be in italics.

ITALICTITLE referst to the name of a work, ie the work itself (play, periodic, book). However, the Unicode standard is a standard, not a book &tc. not even it's publication. The Standard is abstraction: the set of rules. It is a proper noun full stop. Key is, the article title notes the subject: the standard not the book. DePiep (talk) 17:04, 21 April 2023 (UTC)Reply

@Peter M. Brown: -DePiep (talk) 10:43, 23 April 2023 (UTC)Reply

I don't know if it would be manageable, but Unicode clearly does not have all commonly used symbols. A simple example is the very commonly used 'slash marks' used to count. Most reading this will be familiar with the sequence /, //, ///, ////, and //// with the crossmark (strike-through) diagonal (top left to bottom right) rather than horizontal. (This is typical in the USA, I understand European convention is slightly different). I request the editors to consider the addition of a list of missing (but documented) symbols.40.142.183.146 (talk) 11:49, 9 June 2023 (UTC)Reply

Unicode's non-inclusion of tally marks is covered in Tally marks § Unicode. I don't think it's a good idea to include it also in this article. That would open the door of listing every proposal that has not yet been accepted. Indefatigable (talk) 15:42, 9 June 2023 (UTC)Reply

I also oppose this idea. The set of unencoded symbols is open-ended and may exceed the number of encoded symbols. There would also be no way to determine which unencoded symbols merit mention. DRMcCreedy (talk) 16:01, 9 June 2023 (UTC)Reply

Unicode 16 is set to release in September 2024. I think the following (con)scripts definitely need to be encoded:

• Chữ Việt Trí - an alphabet invented by Tôn Thất Chương in 2012 for Vietnamese language. It's still nicer than Latin-based Quoc Ngu and needs wide recognition as the Shavian and Hangul did.
• Add support for Quikscript.
• Add extra missing runes from Baconsthrope and Sedgeford and Armanen runes
• Possibly add something more.

94.180.80.9 (talk) 07:31, 9 July 2023 (UTC)Reply

Take a look at Unicode's FAQ for Submitting Successful Character and Script Proposals. Wikipedia isn't affiliated with The Unicode Consortium so requests here won't be seen or acted upon by the people who can actually add characters/scripts to the Unicode Standard. DRMcCreedy (talk) 14:39, 9 July 2023 (UTC)Reply

@Spitzak: In the text for example, ḗ (precomposed e with macron and acute above) and ḗ (e followed by the combining macron above and combining acute above) should be rendered identically, the "e" is followed by two distinct combining characters, but they are rendered at a single location. I inserted a space to cause them to display as two separate characters, and Spitzak reverted the change with the comment They are supposed to be combined. In context, I don't understand how it makes sense to combine them, since the text refers to them individually. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 21:40, 16 October 2023 (UTC)Reply

My reading of that sentence is that it's comparing the rendering of the precomposed character with the combining characters so you can see how the two render pretty much side by side. That reading prohibits a space between the combining characters. I suppose you should show the combining characters separated then together if you really want to show the components separately. Something like this, with my additions in green: "For example, ḗ (precomposed e with macron and acute above) and ḗ (e followed by the combining macron above and combining acute above) should be rendered identically, both appearing as an e with a macron (◌̄) and acute accent (◌́), but in practice, their appearance may vary depending upon what rendering engine and fonts are being used to display the characters."'DRMcCreedy (talk) 23:01, 16 October 2023 (UTC)Reply

That reading is correct, but I don't agree with the inference; I would agree that it prohibits rendering a space between the two combining characters, but the does not mean that it prohibits a space in the markup that causes the characters to be rendered adjacent to each other with no intervening space. The text "e&#772;&#769;" renders as ḗ, with the ̄ and ̋ overlaid, while the text "e&#772; &#769;" renders as ē ́ , with no overlay and no intervening space.

Your suggested text should be acceptable. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 13:39, 18 October 2023 (UTC)Reply

Great. I've made the updates. DRMcCreedy (talk) 14:29, 18 October 2023 (UTC)Reply

Sorry, I read it too quickly; that still has the original issue unless you remove the "ḗ" and remove the parentheses from the parenthetical note, i.e.,for example, ḗ (precomposed e with macron and acute above) and e followed by the combining macron above and combining acute above should be rendered identically,. Alternatively,for example, ḗ (precomposed e with macron and acute above) and eōó (e followed by the combining macron above and combining acute above) should be rendered identically,. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 19:25, 18 October 2023 (UTC)Reply

But that wouldn't allow the reader to see if the two equivalent versions (precomposed and combining) render the same on whatever device they're using. I think that's the point of having both the precomposed and combining in the example in the first place. DRMcCreedy (talk) 20:29, 18 October 2023 (UTC)Reply

Welcome, I want the Kurdistan flag on my keyboard 85.94.240.91 (talk) 23:28, 2 November 2023 (UTC)Reply

Unfortunately, the flag of Kurdistan is not presently encoded in Unicode. Remsense聊 23:32, 2 November 2023 (UTC)Reply

Nor will it be added per Unicode's proposal guidelines for flags DRMcCreedy (talk) 00:34, 3 November 2023 (UTC)Reply

@Spitzak, I'm also really not sure what you're talking about exactly—Microsoft seems to have the definition of "Unicode" in line with that of the rest of the world.[1] If they use "Unicode" as a shorthand for "UTF-16" sometimes (the way many people use it as a shorthand for "UTF-8", then the page I just linked seems to do any theoretical disambiguation work, and doesn't really leave us wondering whether they're somehow creating an ambiguity problem for us to solve. Remsense诉 02:28, 8 March 2024 (UTC)Reply

I give up on this, but it is because I was looking at a function called isTextUnicode which returns false for UTF-8. There are a number of other examples where "Unicode" means the 16-bit interface.Spitzak (talk) 06:19, 8 March 2024 (UTC)Reply

There are two separate issues:

Do we have to deal with this?

I believe that we do need to mention the limitations of Unicode support in windows.

Is a hatnote the best way to deal with it?

I believe that the hatnote is inappropriate and that the text should mention the limitation, probably not in the lead.

Perhaps Unicode#Operating systems could say In Microsoft windows, the Unicode support is limited to UTF-16. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 15:47, 8 March 2024 (UTC)Reply

Except it isn't really limited to UTF-16, especially in modern versions. The problem is they use "Unicode" all over the place to mean "16-bit encoding" and do differentiate it from "8-bit encoding". This explicitly excludes every form of Unicode other than UTF-16 and UCS-2 (it also thus includes other 16-bit encodings that are not Unicode, but this is probably not a big deal).Spitzak (talk) 17:30, 8 March 2024 (UTC)Reply

Can you work up text that concisely but accurately describes the m$ nomenclature and support for Unicode and the preferred role of UTF-16? -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 18:38, 8 March 2024 (UTC)Reply

I agree with this. Remsense诉 01:41, 9 March 2024 (UTC)Reply

In the Codespace and code points section, it refers to "the interval ${\displaystyle [0,17\times 2^{16})}$ ". I had to read it several times to figure out what was meant. I originally parsed "0,17" as a European-format decimal number, which made no sense. Eventually I figured what was meant, but it wasn't at all obvious. There is nothing in the referenced Unicode 15 standard which uses that terminology, either. The use of mis-matched bracket and paren is a math construct which makes sense for real intervals, but is less commonly used in integer contexts. It will simply appear wrong to readers without a real analysis background.

May I suggest this might be more understandable replaced with "the range 0 : 1114111"? The origin of the latter number is available later in the sentence (with the hexadecimal number 0x10FFFF). Alternatively, a less obscure notation might be ${\displaystyle [0,(2^{20}+2^{16}-1)]}$ . Tarl N. (discuss) 22:55, 11 May 2024 (UTC)Reply

I think I was the one who originally added this. Please do replace it with something more straightforward. Remsense诉 23:10, 11 May 2024 (UTC)Reply

Done, using prose: in the range from 0 to 1114111,... Tarl N. (discuss) 08:01, 12 May 2024 (UTC)Reply

The only detail for Unicode 1.0.1 is about 20902 CJK Unified Ideographs added, but in total 21204 characters were added and 6 were removed. In total, 308 characters were not mentioned at all. Did I miss something while reading the page? What happened to those characters? Can somebody at least explain to me? Apologies in advance if I wasted your time. Mucksrunt (talk) 13:31, 26 August 2024 (UTC)Reply

The Unicode 1.0.1 changes were messy. They brought Unicode into alignment with ISO 10646 and happened prior to the stability policies in place today. I don't come up with 308 characters but looking through the infoboxes for the various Unicode blocks (which I beleive are accurate), I find these changes with Unicode version 1.0.1:

• Alphabetic Presentation Forms (+1)
• CJK Compatibility Ideographs (+302)
• CJK Symbols and Punctuation (+0)
• CJK Unified Ideographs (+20,902)
• Combining Diacritical Marks (+2)
• Cyrillic (-4)
• Enclosed CJK Letters and Months (-1)
• Greek and Coptic (-9)
• Hebrew (-1)
• Lao (-5)
• Miscellaneous Technical (-2)
• Thai (-5)
  Additionally, the range for Private Use Areas was expanded by 768 code points. DRMcCreedy (talk)

Hey! On a lark, I decided to try a minor redesign of the Unicode block templates while fixing the pressing issue of dark mode support—see Wikipedia:Village pump (technical)#Unicode block template and tell me any thoughts you have, as I think it's probably worthwhile to at least refresh these templates. Remsense ‥ 论 15:44, 11 September 2024 (UTC)Reply

I have two concerns on your proposed redesign: First, the link to the Unicode PDF chart is no longer obvious to the reader as it's now a reference as opposed to being clear in the chart heading. Easy access to the PDF is especially important for not widely supported code ranges. Second, consolidating the notes onto a single line is OK for most of the cases but will be harder to understand for charts with longer notes like Template:Unicode chart Hangul Jamo. DRMcCreedy (talk) 17:41, 11 September 2024 (UTC)Reply

Moving to a reference wasn't my idea directly, as I can see it either way. Per your second point, I would actually handle this by adding additional lines for those extra notes. Remsense ‥ 论 17:53, 11 September 2024 (UTC)Reply

@Drmccreedy I think I've finished iterating on the design for now in response to feedback here and at the Village Pump—I'm still not totally sure how/whether to display the default footnote and the PDF code chart reference, but other than that I think it's just about ready to consider deploying. Any further thoughts? Remsense ‥ 论 05:01, 13 September 2024 (UTC)Reply