[Unicode]  Unicode Character Database
 

Unicode Data File Format

Revision 3.1.0
Authors Mark Davis and Ken Whistler
Date 2001-02-28
This Version http://www.unicode.org/Public/3.1-Update/UnicodeData-3.1.0.html
Previous Version http://www.unicode.org/Public/3.0-Update1/UnicodeData-3.0.1.html
Latest Version http://www.unicode.org/Public/UNIDATA/UnicodeData.html


Summary

This document describes the format and content of the UnicodeData.txt file in the Unicode Character Database (UCD).

Status

The file and the files described herein are part of the Unicode Character Database  and governed by the UCD Terms of Use given below.

For general information on file formats and table formats, and the implications of normative vs informative properties, see UnicodeCharacterDatabase.html.

Warning: the information in this file does not completely describe the use and interpretation of Unicode character properties and behavior. It must be used in conjunction with the data in the other files in the UCD, and relies on the notation and definitions supplied in The Unicode Standard. All chapter references are to Version 3.1.0 of the standard.

Introduction

This document describes the format of the UnicodeData.txt file, which is one of the files in the Unicode Character Database. The document is divided into the following sections:

Field Formats

Each line represents the data for one encoded character in the Unicode Standard. (For information on the file format, see UCD File Format in UnicodeCharacterDatabase.html).

Every encoded character has a data entry, with the exception of certain special ranges, as detailed below.

The exact ranges represented by start and end characters are:

The following table describes the format and meaning of each field in a data entry in the UnicodeData file.

Field

Name

N/I

Explanation

0 Code value N Code value.
1 Character name N These names match exactly the names published in Chapter 14 of the Unicode Standard, Version 3.0.
2 General Category N This is a useful breakdown into various "character types" which can be used as a default categorization in implementations. See below for a brief explanation.
3 Canonical Combining Classes N The classes used for the Canonical Ordering Algorithm in the Unicode Standard. These classes are also printed in Chapter 4 of the Unicode Standard.
4 Bidirectional Category N See the list below for an explanation of the abbreviations used in this field. These are the categories required by the Bidirectional Behavior Algorithm in the Unicode Standard. These categories are summarized in Chapter 3 of the Unicode Standard.
5 Character Decomposition Mapping N In the Unicode Standard, not all of the mappings are full (maximal) decompositions. Recursive application of look-up for decompositions will, in all cases, lead to a maximal decomposition. The decomposition mappings match exactly the decomposition mappings published with the character names in the Unicode Standard.
6 Decimal digit value N This is a numeric field. If the character has the decimal digit property, as specified in Chapter 4 of the Unicode Standard, the value of that digit is represented with an integer value in this field
7 Digit value N This is a numeric field. If the character represents a digit, not necessarily a decimal digit, the value is here. This covers digits which do not form decimal radix forms, such as the compatibility superscript digits
8 Numeric value N This is a numeric field. If the character has the numeric property, as specified in Chapter 4 of the Unicode Standard, the value of that character is represented with an integer or rational number in this field. This includes fractions as, e.g., "1/5" for U+2155 VULGAR FRACTION ONE FIFTH Also included are numerical values for compatibility characters such as circled numbers.
9 Mirrored N If the character has been identified as a "mirrored" character in bidirectional text, this field has the value "Y"; otherwise "N". The list of mirrored characters is also printed in Chapter 4 of the Unicode Standard.
10 Unicode 1.0 Name I This is the old name as published in Unicode 1.0. This name is only provided when it is significantly different from the current name for the character.
11 10646 comment field I This is the ISO 10646 comment field. It appears in parentheses in the 10646 names list, or contains an asterisk to mark an Annex P note.
12 Uppercase Mapping N Upper case equivalent mapping. If a character is part of an alphabet with case distinctions, and has a simple upper case equivalent, then the upper case equivalent is in this field. See the explanation below on case distinctions. These mappings are always one-to-one, not one-to-many or many-to-one.

For full case mappings, see UTR #21 and SpecialCasing.txt.

13 Lowercase Mapping N Similar to Uppercase mapping
14 Titlecase Mapping N Similar to Uppercase mapping

General Category

The values in this field are abbreviations for the following values. For more information, see the Unicode Standard.

Note: the standard does not assign information to control characters (except for certain cases in the Bidirectional Algorithm). Implementations will generally also assign categories to certain control characters, notably CR and LF, according to platform conventions. See UAX #13: Unicode Newline Guidelines for more information.

Abbr.

Description

Lu Letter, Uppercase
Ll Letter, Lowercase
Lt Letter, Titlecase
Lm Letter, Modifier
Lo Letter, Other
Mn Mark, Non-Spacing
Mc Mark, Spacing Combining
Me Mark, Enclosing
Nd Number, Decimal Digit
Nl Number, Letter
No Number, Other
Pc Punctuation, Connector
Pd Punctuation, Dash
Ps Punctuation, Open
Pe Punctuation, Close
Pi Punctuation, Initial quote (may behave like Ps or Pe depending on usage)
Pf Punctuation, Final quote (may behave like Ps or Pe depending on usage)
Po Punctuation, Other
Sm Symbol, Math
Sc Symbol, Currency
Sk Symbol, Modifier
So Symbol, Other
Zs Separator, Space
Zl Separator, Line
Zp Separator, Paragraph
Cc Other, Control
Cf Other, Format
Cs Other, Surrogate
Co Other, Private Use
Cn Other, Not Assigned (no characters in the file have this property)

Note: The term "L&" is sometimes used to stand for Uppercase, Lowercase or Titlecase letters (Lu, Ll, or Lt).

Bidirectional Category

Please refer to Chapter 3 for an explanation of the algorithm for Bidirectional Behavior and an explanation of the significance of these categories. An up-to-date version can be found on Unicode Standard Annex #9: The Bidirectional Algorithm.

Type

Description

L Left-to-Right
LRE Left-to-Right Embedding
LRO Left-to-Right Override
R Right-to-Left
AL Right-to-Left Arabic
RLE Right-to-Left Embedding
RLO Right-to-Left Override
PDF Pop Directional Format
EN European Number
ES European Number Separator
ET European Number Terminator
AN Arabic Number
CS Common Number Separator
NSM Non-Spacing Mark
BN Boundary Neutral
B Paragraph Separator
S Segment Separator
WS Whitespace
ON Other Neutrals

Character Decomposition Mapping

The tags supplied with certain decomposition mappings generally indicate formatting information. Where no such tag is given, the mapping is designated as canonical. Conversely, the presence of a formatting tag also indicates that the mapping is a compatibility mapping and not a canonical mapping. In the absence of other formatting information in a compatibility mapping, the tag is used to distinguish it from canonical mappings.

In some instances a canonical mapping or a compatibility mapping may consist of a single character. For a canonical mapping, this indicates that the character is a canonical equivalent of another single character. For a compatibility mapping, this indicates that the character is a compatibility equivalent of another single character. The compatibility formatting tags used are:

Tag

Description

<font>   A font variant (e.g. a blackletter form).
<noBreak>   A no-break version of a space or hyphen.
<initial>   An initial presentation form (Arabic).
<medial>   A medial presentation form (Arabic).
<final>   A final presentation form (Arabic).
<isolated>   An isolated presentation form (Arabic).
<circle>   An encircled form.
<super>   A superscript form.
<sub>   A subscript form.
<vertical>   A vertical layout presentation form.
<wide>   A wide (or zenkaku) compatibility character.
<narrow>   A narrow (or hankaku) compatibility character.
<small>   A small variant form (CNS compatibility).
<square>   A CJK squared font variant.
<fraction>   A vulgar fraction form.
<compat>   Otherwise unspecified compatibility character.

Reminder: There is a difference between decomposition and decomposition mapping. The decomposition mappings are defined in the UnicodeData, while the decomposition (also termed "full decomposition") is defined in Chapter 3 to use those mappings recursively.

Canonical Combining Classes

Value

Description

0: Spacing, split, enclosing, reordrant, and Tibetan subjoined
1: Overlays and interior
7: Nuktas
8: Hiragana/Katakana voicing marks
9: Viramas
10: Start of fixed position classes
199: End of fixed position classes
200: Below left attached
202: Below attached
204: Below right attached
208: Left attached (reordrant around single base character)
210: Right attached
212: Above left attached
214: Above attached
216: Above right attached
218: Below left
220: Below
222: Below right
224: Left (reordrant around single base character)
226: Right
228: Above left
230: Above
232: Above right
233: Double below
234: Double above
240: Below (iota subscript)

Note: some of the combining classes in this list do not currently have members but are specified here for completeness.

Decompositions and Normalization

Decomposition is specified in Chapter 3. Unicode Standard Annex #15: Unicode Normalization Forms specifies the interaction between decomposition and normalization. That report specifies how the decompositions defined in UnicodeData.txt are used to derive normalized forms of Unicode text.

Note that as of the 2.1.9 update of the Unicode Character Database, the decompositions in the UnicodeData.txt file can be used to recursively derive the full decomposition in canonical order, without the need to separately apply canonical reordering. However, canonical reordering of combining character sequences must still be applied in decomposition when normalizing source text which contains any combining marks.

Case Mappings

There are a number of complications to case mappings that occur once the repertoire of characters is expanded beyond ASCII. For more information, see UTR #21: Case Mappings.

For compatibility with existing parsers, UnicodeData.txt only contains case mappings for characters where they are one-to-one mappings; it also omits information about context-sensitive case mappings. Information about these special cases can be found in a separate data file, SpecialCasing.txt.

Property Invariants

Values in UnicodeData.txt are subject to correction as errors are found; however, some characteristics of the categories themselves can be considered invariants. Applications may wish to take these invariants into account when choosing how to implement character properties. For more information, see Unicode Policies.

The following is a partial list of known invariants for the Unicode Character Database.

Database Fields

General Category

Combining Classes

Canonical Decomposition

Modification History

This section provides a summary of the changes between update versions of the Unicode Standard.

Unicode 3.1

Modifications made for Version 3.0.1 of UnicodeData.txt include:

Unicode 3.0.1

Modifications made for Version 3.0.1 of UnicodeData.txt include:

Unicode 3.0.0

Modifications made for Version 3.0.0 of UnicodeData.txt include many new characters and a number of property changes. These are summarized in Appendex D of The Unicode Standard, Version 3.0.

Unicode 2.1.9

Modifications made for Version 2.1.9 of UnicodeData.txt include:

Unicode 2.1.8

Modifications made for Version 2.1.8 of UnicodeData.txt include:

Version 2.1.7

This version was for internal change tracking only, and never publicly released.

Version 2.1.6

This version was for internal change tracking only, and never publicly released.

Unicode 2.1.5

Modifications made for Version 2.1.5 of UnicodeData.txt include:

Version 2.1.4

This version was for internal change tracking only, and never publicly released.

Version 2.1.3

This version was for internal change tracking only, and never publicly released.

Unicode 2.1.2

Modifications made in updating UnicodeData.txt to Version 2.1.2 for the Unicode Standard, Version 2.1 (from Version 2.0) include:

Version 2.1.1

This version was for internal change tracking only, and never publicly released.

Unicode 2.0.0

The modifications made in updating UnicodeData.txt for the Unicode Standard, Version 2.0 include:

UCD Terms of Use

Disclaimer

The Unicode Character Database is provided as is by Unicode, Inc. No claims are made as to fitness for any particular purpose. No warranties of any kind are expressed or implied. The recipient agrees to determine applicability of information provided. If this file has been purchased on magnetic or optical media from Unicode, Inc., the sole remedy for any claim will be exchange of defective media within 90 days of receipt.

This disclaimer is applicable for all other data files accompanying the Unicode Character Database, some of which have been compiled by the Unicode Consortium, and some of which have been supplied by other sources.

Limitations on Rights to Redistribute This Data

Recipient is granted the right to make copies in any form for internal distribution and to freely use the information supplied in the creation of products supporting the UnicodeTM Standard. The files in the Unicode Character Database can be redistributed to third parties or other organizations (whether for profit or not) as long as this notice and the disclaimer notice are retained. Information can be extracted from these files and used in documentation or programs, as long as there is an accompanying notice indicating the source.


HomeTerms of UseE-mail