utf8proc

annotate README @ 5:c18366878af9

find changesets by author, revision, files, or words in the commit message
Version 1.0.2

- included a check in Integer#utf8, which raises an exception, if the given code-point is invalid because of being too high (this was missing yet)
- added support for PostgreSQL version 8.2
author jbe
date Tue Dec 26 12:00:00 2006 +0100 (2006-12-26)
parents a49e32490aac
children fcfd8c836c64
rev   line source
jbe@0 1
jbe@0 2 Please read the LICENSE file, which is shipping with this software.
jbe@0 3
jbe@0 4
jbe@0 5 *** QUICK START ***
jbe@0 6
jbe@0 7 For compilation of the C library call "make c-library", for compilation of
jbe@0 8 the ruby library call "make ruby-library" and for compilation of the
jbe@0 9 PostgreSQL extension call "make pgsql-library".
jbe@0 10
jbe@0 11 "make all" can be used to build everything, but both ruby and PostgreSQL
jbe@0 12 installations are required in this case.
jbe@0 13
jbe@4 14 For ruby there is alternatively provided a gem-file "utf8proc-1.0.1.gem".
jbe@4 15
jbe@0 16
jbe@0 17 *** GENERAL INFORMATION ***
jbe@0 18
jbe@0 19 The C library is found in this directory after successful compilation and is
jbe@0 20 named "libutf8proc.a" and "libutf8proc.so". The ruby library consists of the
jbe@0 21 files "utf8proc.rb" and "utf8proc_native.so", which are found in the
jbe@0 22 subdirectory "ruby/". The PostgreSQL extension is named "utf8proc_pgsql.so"
jbe@0 23 and resides in the "pgsql/" directory.
jbe@0 24
jbe@0 25 Both the ruby library and the PostgreSQL extension are built as stand-alone
jbe@0 26 libraries and are therefore not dependent the dynamic version of the
jbe@0 27 C library files, but this behaviour might change in future releases.
jbe@0 28
jbe@2 29 The Unicode version being supported is 5.0.0.
jbe@2 30 Note: Version 4.1.0 of Unicode Standard Annex #29 was used, as version 5.0.0
jbe@2 31 had not been available yet.
jbe@0 32
jbe@0 33 For Unicode normalizations, the following options have to be used:
jbe@0 34 Normalization Form C: STABLE, COMPOSE
jbe@2 35 Normalization Form D: STABLE, DECOMPOSE
jbe@0 36 Normalization Form KC: STABLE, COMPOSE, COMPAT
jbe@2 37 Normalization Form KD: STABLE, DECOMPOSE, COMPAT
jbe@0 38
jbe@0 39
jbe@0 40 *** C LIBRARY ***
jbe@0 41
jbe@0 42 The documentation for the C library is found in the utf8proc.h header file.
jbe@0 43 "utf8proc_map" is most likely function you will be using for mapping UTF-8
jbe@0 44 strings, unless you want to allocate memory yourself.
jbe@0 45
jbe@0 46
jbe@0 47 *** RUBY API ***
jbe@0 48
jbe@0 49 The ruby library adds the methods "utf8map" and "utf8map!" to the String
jbe@0 50 class, and the method "utf8" to the Integer class.
jbe@0 51
jbe@0 52 The String#utf8map method does the same as the "utf8proc_map" C function.
jbe@0 53 Options for the mapping procedure are passed as symbols, i.e:
jbe@2 54 "Hello".utf8map(:casefold) => "hello"
jbe@0 55
jbe@0 56 The descriptions of all options are found in the C header file "utf8proc.h".
jbe@0 57 Please notice that the according symbols in ruby are all lowercase.
jbe@0 58
jbe@0 59 String#utf8map! is the destructive function in the meaning that the string
jbe@0 60 is replaced by the result.
jbe@0 61
jbe@0 62 There are shortcuts for the 4 normalization forms specified by Unicode:
jbe@0 63 String#utf8nfd, String#utf8nfd!,
jbe@0 64 String#utf8nfc, String#utf8nfc!,
jbe@0 65 String#utf8nfkd, String#utf8nfkd!,
jbe@0 66 String#utf8nfkc, String#utf8nfkc!
jbe@0 67
jbe@0 68 The method Integer#utf8 returns a UTF-8 string, which is containing the
jbe@2 69 unicode char given by the code point.
jbe@0 70 0x000A.utf8 => "\n"
jbe@0 71 0x2028.utf8 => "\342\200\250"
jbe@0 72
jbe@0 73
jbe@0 74 *** POSTGRESQL API ***
jbe@0 75
jbe@0 76 For PostgreSQL there is a SQL function supplied named "unifold". This
jbe@0 77 function can be used to prepare index fields in order to be normalized and
jbe@0 78 case-folded, i.e.:
jbe@0 79
jbe@1 80 CREATE TABLE people (
jbe@1 81 id serial8 primary key,
jbe@1 82 name text,
jbe@1 83 CHECK (unifold(name) NOTNULL)
jbe@1 84 );
jbe@0 85 CREATE INDEX name_idx ON people (unifold(name));
jbe@0 86 SELECT * FROM people WHERE unifold(name) = unifold('John Doe');
jbe@0 87
jbe@2 88 NOTICE: The outputs of the function can change between releases, as utf8proc
jbe@2 89 does not follow a versioning stability policy. You have to rebuild
jbe@2 90 your database indicies, if you upgrade to a newer version of utf8proc.
jbe@2 91
jbe@2 92
jbe@0 93 *** TODO ***
jbe@0 94
jbe@0 95 - detect stable code points and process segments independently in order to
jbe@0 96 save memory
jbe@0 97 - do a quick check before normalizing strings to optimize speed
jbe@0 98 - support stream processing
jbe@0 99
jbe@0 100
jbe@0 101 Unicode is a trademark of Unicode, Inc., and may be registered in some
jbe@0 102 jurisdictions.
jbe@0 103
jbe@0 104

Impressum / About Us