Transcript 1,228 new characters:

Unicode 4.0
Mark Davis
President, The Unicode
Consortium
Note: slides differ from proceedings
Overview
 New
Characters
 Conformance
 UAX:
Unicode Standard Annexes
 UCD:
Unicode Character Database
 UTS:
 Not
Unicode Technical Standards
part of the Standard, but can claim
conformance
Properties and Behavior
 Unicode
is not just a list of characters
 Properties and behavior are crucial
 With them, new characters can work
“out of the box”
 Some are part of the standard (BIDI,
Normalization), others are associated
(Collation, Regular Expressions)
New Characters: 1,228

Modern Scripts



Historic Scripts


(additions to) Indic, Khmer, Latin, Greek, Arabic,
Syriac
(minority scripts) Limbu, Tai Le, Osmanya
Linear B, Cypriot, Ugaritic, Shavian, Aegean
Numbers
Symbols


Monograms, digrams, tetragrams, other symbols
modifier & combining characters
New Characters (cont.)
 Special
Characters
 additional
variation selectors (for future
CJK variants), double-diacritics for
dictionary use
 For
a detailed list, see Derived Age in
the UCD 4.0, and the beta Charts.
 Character repertoire corresponds to
ISO/IEC 10646:2003.
Conformance

Substantially improved specification of
conformance requirements



Incorporated UTR #17: Character Encoding
Model, clearly separating encoding forms and
encoding schemes
Tightened definitions of UTF-8, UTF-16, UTF-32
Separate definition of Unicode String

Clarified conformance status of Unicode
Standard Annexes
 Formal definitions of properties & algorithms

Provisional properties
UTF vs. Unicode String

Important Distinction
 UTF



Unique representation for Code Point
All else illegal
C0 80
D800 0061
Unicode String



Sequence of code units
Internal Processing, not interchange
Not necessarily valid UTF
C0 A0
D800 0061
Conformance (cont.)

Formalized policies for stability of the
standard
 Clarification of semantics of important
characters, including BOM
 Revised scope of enclosing combining marks
 Revised semantics of ZWJ for cursive scripts
 Normalization Corrections


U+2F868; U+2F874; U+2F91F; U+2F95F;
U+2F9BF
All corrections subject to strict stability constraints:

For 3.2 repertoire, NFC3.2(X) = NFC4.0(X)
Textual Clarifications

Major changes to Chapters 2, 3, 6, 14 and 15
 Definitive terminology for code points:

graphic, format, control, private-use


surrogate, noncharacter, reserved


= assigned characters
not characters
Substantial improvements to many character
block descriptions, especially Indic
Programming language
identifiers

Now backwards-compatible



Alternate definition for complete stability





Once a Unicode identifier,
Always a Unicode identifier
Fix set of allowed characters
Allow all reserved code points
+ Complete stability
- “Odd” characters
Also see new UTR on Syntax Characters
Case mappings now
normative (but tailorable)
 Clearer
definition of string functions:
 isUpper(),
isLower(), isTitle(), isFold()
 toUpper(), toLower(), toTitle(), toFold()
 Definition
of titlecase uses word
boundaries
 Note
that the Turkic mappings do not
maintain canonical equivalence, without
additional processing.
UAX #9: BIDI

BIDI: Arabic/Hebrew Display


HTML, all modern word processors, OSs,…
New:

canonically equivalence now preserved


shaping is done after reordering


data change, not algorithm
but not across directional boundaries
clarifications of:


ZWJ, ZWNJ
intermediate level processing
UAX #15: Normalization

Unique form for text comparison


W3C Character Model, International Domain
Names, Network File System,…
New:





Description of Stable Code Points.
Notation NFC(x) and isNFC(x), in Notation.
Added pointer to UTN #5 Canonical Equivalences
in Applications
Rewrote Annex 12: Corrigenda for clarity, and to
describe the use of Normalization Corrections.
Added Annex 13: Canonical Equivalence.
UAX #14: Line Breaking

Line-Break (word-wrap) all Unicode text


Customizable for different languages
New:



Negative numbers and dates with hyphens will not
break across lines
Word-Joiner will link any characters (except hard
line breaks)
Behavior of soft hyphen clarified



marks opportunity for breaking, not specific graphic
appearance.
Rules for GL relaxed: SP and ZW override
New Property Values: NL, WJ
UAX #29: Text Boundaries

Default “User Character”, Word, Sentence
boundaries



Customizable for different languages
Word, sentence: tailoring expected
New:


Extracted from 3.0, but significantly revised
Grapheme cluster (“user character”)


Hangul Syllable or other Base
plus (optionally) any number of NSMs
No Sub. Changes
 UAX
#11: East Asian Width
 Guidelines
 UAX
for choosing character width
#24: Script Names
 Default
script assignment
 Used in regular expressions
 Now UAX
Superseded UAXes
 Incorporated
into and thus superseded
by Unicode Version 4.0:
 UAX
#13: Unicode Newline Guidelines
 UAX #19: UTF-32
 UAX #21: Case Mappings
 UAX #27: Unicode 3.1
 UAX #28: Unicode 3.2
Unicode Character Database

Crucial Component of Unicode
 Documentation coalesced into UCD.html.
 New properties and values




UCD fallback props more precisely defined.


Hangul_Syllable_Type, Unicode_Radical_Stroke
CJK numeric values added.
PropertyValueAliases adds block names
for code points not explicitly in data files
New Characters

Appropriate properties assigned
UCD4.0 (cont.)

Modifier letters


Khmer


Two Khmer characters are deprecated; four others
strongly discouraged.
Decimal Digits


The general category of 02B9..02BA, 02C6..02CF
changed to general category Lm.
Numeric_Type=decimal digit now aligned with
General_Category=Nd
Braille

Added script value
UCD4.0 (cont. 2)

Case Mapping


Default Ignorables




Fixed for Turkish, Lithuanian
Hangul Filler characters
Soft-Hyphen, CGJ, ZWS
Arabic End of Ayah and Syriac Abbreviation Mark
no longer DI, shaping classes fixed.
Grapheme_Extend

removes halfwidth katakana marks, most Mc
(except as needed for canonical equivalence)
Unicode Technical Standard
 UTS:
separate standard
 independent
 UTR:
conformance requirements
information and guidelines
 Documents
to UTS
may move from UTR status
UTS #10: Unicode Collation

Significance:



String comparison, matching, searching
Compares all Unicode characters
Handles linguistic features




Accents, Case, Punctuation,…
Contextual weighting,…
Tailor for different languages
Version 4.0.0 due Sept. 2003

From now on, to be sync'ed in repertoire and
version with the Unicode Standard.
UTS #18: Regular Exp.

Significance:


Crucial to many applications: web, XML,…
Unicode adds significant requirements

Level 1: Basic Support




Perl
Level 2: Extended Support
Level 3: Tailored Support
New:


Recently approved as UTS (was UTR)
Adds clearer conformance requirements


Flexible list of features
Partial conformance claims
UTS #6: SCSU
 Simple
Unicode Compression
 Added suitability for XML
 See also Technical Note on BOCU
 Main
difference: preserves binary order
 x < y => BOCU(x) < BOCU(y)
New UTRs
 Draft
UTR #23: Character Properties
 Draft
Character Property Model
 Character
Folding
 Hiragana-Katakana,
 Programming
characters
Case, …
Language IDs, Syntax
Q& A
 Other
talks here:
 Common Locale Data
 interchange
of language-specific data for
sorting, dates, times, currencies
 ICU
 premier
Unicode enablement library
 full-featured, x-platform
 C, C++, Java
Background Slides
Unicode 3.2 (March, 2002)

New Characters: 1,016
 Symbols


Special Characters


Large collection of mathematical symbols,
especially targeted at MathML, recycling symbols,
ornamental brackets.
combining grapheme joiner, word joiner, invisible
operators for math, variation selectors
Modern Scripts

minority scripts of the Philippines
Conformance
 Eliminates
irregular UTF-8
 Defines variation sequences
 Replaces ZWNBSP with Word Joiner
 Clarifies scope of combining marks
(further revised in 4.0)
 Clarifications of conjoining jamo
behavior, hangul syllable structure,
decomposables,
Textual Clarifications
 Combined
vowels in Khmer, characters
discouraged in Khmer
 Use of dingbats
Unicode Standard Annexes
 UAX
#21: Case Mappings (was UTR)
Unicode Character Database

New properties:




IDS_Binary_Operator, IDS_Trinary_Operator,
Radical, Unified_Ideograph,
Default_Ignorable_Code_Point, Deprecated
Soft_Dotted, Logical_Order_Exception
Grapheme_Base,
Grapheme_Extend,Grapheme_Link
DerivedAge
 Normalization Corrections
 Added Property & Property Value Aliases
 Adds StandardizedVariants.html
Related Items

UTS #10: Unicode Collation Algorithm



Ignorable character handling, dual versioning,
more conditions on well-formed weights, separate
weights for CJK and unassigned characters, noncharacters
Note: base version still U3.1
UTR #26: CESU-8
 Unicode Technical Notes
 Updated Character Encoding Stability Policy
 Added Public Review process
 Updated Glossary
Unicode 3.1 (March, 2001)

New Characters: 44,946


Modern scripts


CJK Ideographs (now totaling 71,039)
Historic scripts


First supplementaries encoded!
Old Italic, Gothic, Deseret, Byzantine Musical
Symbols
Symbols

Mathematical Alphanumeric Symbols, (Western)
Musical Symbols
Conformance

Non-shortest-form UTF-8 excluded
 Clarification of the stability of the standard,


code units vs. code points, non-characters,
normative properties, informative properties,
normative references
Revisions of guidelines:

wchar_t, unassigned code points, identifiers

Major revision of Georgian
 Use of ZWNJ and ZWJ for ligatures
 Language tag characters encoded

but discouraged
Unicode Standard Annexes
 UAX
#19: UTF-32
Unicode Character Database

Major revision of PropList properties:




White_Space, Bidi_Control, Join_Control,
Hex_Digit
Alphabetic, Ideographic, Lowercase, Uppercase
ID_Start, ID_Continue, XID_Start, XID_Continue
Noncharacter_Code_Point
Quotation_Mark, Terminal_Punctuation, Math,
Dash, Hyphen, Diacritic, Extender
New properties: Case folding, Scripts
 Added DerivedProperties, NormalizationTest
Related Items

Documented Character Encoding Stability
Policy
 UTS #10: Unicode Collation Algorithm


Merged data files; updated to base version 3.1
UTR #18: Unicode Regular Expression
Guidelines
 UTR #20: Unicode in XML and other Markup
Languages
 UTR #22: Character Mapping Tables
 UTR #24: Script Names
Schedule
 2003, April:
UCD/UAXes
 Final
data files available
 Implementation can proceed
 2003:
September:
 Book
Available