Current Path: > > usr > share > perl5 > pod >
Operation : Linux premium131.web-hosting.com 4.18.0-553.44.1.lve.el8.x86_64 #1 SMP Thu Mar 13 14:29:12 UTC 2025 x86_64 Software : Apache Server IP : 162.0.232.56 | Your IP: 216.73.216.111 Domains : 1034 Domain(s) Permission : [ 0755 ]
Name | Type | Size | Last Modified | Actions |
---|---|---|---|---|
perl.pod | File | 16270 bytes | July 28 2025 08:07:09. | |
perl5004delta.pod | File | 56240 bytes | July 28 2025 08:07:09. | |
perl5005delta.pod | File | 34283 bytes | July 28 2025 08:07:09. | |
perl5100delta.pod | File | 55535 bytes | July 28 2025 08:07:09. | |
perl5101delta.pod | File | 43888 bytes | July 28 2025 08:07:09. | |
perl5120delta.pod | File | 89272 bytes | July 28 2025 08:07:09. | |
perl5121delta.pod | File | 10141 bytes | July 28 2025 08:07:09. | |
perl5122delta.pod | File | 9603 bytes | July 28 2025 08:07:09. | |
perl5123delta.pod | File | 4100 bytes | July 28 2025 08:07:09. | |
perl5124delta.pod | File | 3672 bytes | July 28 2025 08:07:09. | |
perl5125delta.pod | File | 7683 bytes | July 28 2025 08:07:09. | |
perl5140delta.pod | File | 144324 bytes | July 28 2025 08:07:09. | |
perl5141delta.pod | File | 7966 bytes | July 28 2025 08:07:09. | |
perl5142delta.pod | File | 6892 bytes | July 28 2025 08:07:09. | |
perl5143delta.pod | File | 7760 bytes | July 28 2025 08:07:09. | |
perl5144delta.pod | File | 6327 bytes | July 28 2025 08:07:09. | |
perl5160delta.pod | File | 133651 bytes | July 28 2025 08:07:09. | |
perl5161delta.pod | File | 6142 bytes | July 28 2025 08:07:09. | |
perl5162delta.pod | File | 3594 bytes | July 28 2025 08:07:09. | |
perl5163delta.pod | File | 4085 bytes | July 28 2025 08:07:09. | |
perl5180delta.pod | File | 119431 bytes | July 28 2025 08:07:09. | |
perl5181delta.pod | File | 6595 bytes | July 28 2025 08:07:09. | |
perl5182delta.pod | File | 5335 bytes | July 28 2025 08:07:09. | |
perl5184delta.pod | File | 4642 bytes | July 28 2025 08:07:09. | |
perl5200delta.pod | File | 115699 bytes | July 28 2025 08:07:09. | |
perl5201delta.pod | File | 10899 bytes | July 28 2025 08:07:09. | |
perl5202delta.pod | File | 12509 bytes | July 28 2025 08:07:09. | |
perl5203delta.pod | File | 9392 bytes | July 28 2025 08:07:09. | |
perl5220delta.pod | File | 130963 bytes | July 28 2025 08:07:09. | |
perl5221delta.pod | File | 10767 bytes | July 28 2025 08:07:09. | |
perl5222delta.pod | File | 12629 bytes | July 28 2025 08:07:09. | |
perl5223delta.pod | File | 8456 bytes | July 28 2025 08:07:09. | |
perl5224delta.pod | File | 4460 bytes | July 28 2025 08:07:09. | |
perl5240delta.pod | File | 64927 bytes | July 28 2025 08:07:09. | |
perl5241delta.pod | File | 8215 bytes | July 28 2025 08:07:09. | |
perl5242delta.pod | File | 4113 bytes | July 28 2025 08:07:09. | |
perl5243delta.pod | File | 11428 bytes | July 28 2025 08:07:09. | |
perl5244delta.pod | File | 4510 bytes | July 28 2025 08:07:09. | |
perl5260delta.pod | File | 101836 bytes | July 28 2025 08:07:09. | |
perl5261delta.pod | File | 7927 bytes | July 28 2025 08:07:09. | |
perl5262delta.pod | File | 7880 bytes | July 28 2025 08:07:09. | |
perl5263delta.pod | File | 7063 bytes | July 28 2025 08:07:09. | |
perl5280delta.pod | File | 72113 bytes | July 28 2025 08:07:09. | |
perl561delta.pod | File | 124713 bytes | July 28 2025 08:07:09. | |
perl56delta.pod | File | 107201 bytes | July 28 2025 08:07:09. | |
perl581delta.pod | File | 38061 bytes | July 28 2025 08:07:09. | |
perl582delta.pod | File | 4470 bytes | July 28 2025 08:07:09. | |
perl583delta.pod | File | 6335 bytes | July 28 2025 08:07:09. | |
perl584delta.pod | File | 7363 bytes | July 28 2025 08:07:09. | |
perl585delta.pod | File | 5889 bytes | July 28 2025 08:07:09. | |
perl586delta.pod | File | 4651 bytes | July 28 2025 08:07:09. | |
perl587delta.pod | File | 8357 bytes | July 28 2025 08:07:09. | |
perl588delta.pod | File | 25272 bytes | July 28 2025 08:07:09. | |
perl589delta.pod | File | 53900 bytes | July 28 2025 08:07:09. | |
perl58delta.pod | File | 115165 bytes | July 28 2025 08:07:09. | |
perlaix.pod | File | 20437 bytes | July 28 2025 08:07:09. | |
perlamiga.pod | File | 5749 bytes | July 28 2025 08:07:09. | |
perlandroid.pod | File | 7871 bytes | July 28 2025 08:07:09. | |
perlapi.pod | File | 443535 bytes | July 28 2025 08:07:09. | |
perlapio.pod | File | 19285 bytes | July 28 2025 08:07:09. | |
perlartistic.pod | File | 7010 bytes | July 28 2025 08:07:09. | |
perlbook.pod | File | 8338 bytes | July 28 2025 08:07:09. | |
perlboot.pod | File | 294 bytes | July 28 2025 08:07:09. | |
perlbot.pod | File | 304 bytes | July 28 2025 08:07:09. | |
perlbs2000.pod | File | 8058 bytes | July 28 2025 08:07:09. | |
perlcall.pod | File | 56706 bytes | July 28 2025 08:07:09. | |
perlce.pod | File | 14602 bytes | July 28 2025 08:07:09. | |
perlcheat.pod | File | 4481 bytes | July 28 2025 08:07:09. | |
perlclib.pod | File | 9619 bytes | July 28 2025 08:07:09. | |
perlcn.pod | File | 4691 bytes | July 28 2025 08:07:09. | |
perlcommunity.pod | File | 7217 bytes | July 28 2025 08:07:09. | |
perlcygwin.pod | File | 27199 bytes | July 28 2025 08:07:09. | |
perldata.pod | File | 46743 bytes | July 28 2025 08:07:09. | |
perldbmfilter.pod | File | 4981 bytes | July 28 2025 08:07:09. | |
perldebguts.pod | File | 38535 bytes | July 28 2025 08:07:09. | |
perldebtut.pod | File | 22152 bytes | July 28 2025 08:07:09. | |
perldebug.pod | File | 39258 bytes | July 28 2025 08:07:09. | |
perldelta.pod | File | 7063 bytes | July 28 2025 08:07:09. | |
perldeprecation.pod | File | 18169 bytes | July 28 2025 08:07:09. | |
perldiag.pod | File | 284572 bytes | July 28 2025 08:07:09. | |
perldos.pod | File | 10522 bytes | July 28 2025 08:07:09. | |
perldsc.pod | File | 25614 bytes | July 28 2025 08:07:09. | |
perldtrace.pod | File | 7958 bytes | July 28 2025 08:07:09. | |
perlebcdic.pod | File | 84233 bytes | July 28 2025 08:07:09. | |
perlembed.pod | File | 37196 bytes | July 28 2025 08:07:09. | |
perlexperiment.pod | File | 7195 bytes | July 28 2025 08:07:09. | |
perlfork.pod | File | 13355 bytes | July 28 2025 08:07:09. | |
perlform.pod | File | 16608 bytes | July 28 2025 08:07:09. | |
perlfreebsd.pod | File | 1610 bytes | July 28 2025 08:07:09. | |
perlfunc.pod | File | 392957 bytes | July 28 2025 08:07:09. | |
perlgit.pod | File | 33509 bytes | July 28 2025 08:07:09. | |
perlgpl.pod | File | 13815 bytes | July 28 2025 08:07:09. | |
perlguts.pod | File | 139329 bytes | July 28 2025 08:07:09. | |
perlhack.pod | File | 40445 bytes | July 28 2025 08:07:09. | |
perlhacktips.pod | File | 55509 bytes | July 28 2025 08:07:09. | |
perlhacktut.pod | File | 6153 bytes | July 28 2025 08:07:09. | |
perlhaiku.pod | File | 1508 bytes | July 28 2025 08:07:09. | |
perlhist.pod | File | 53546 bytes | July 28 2025 08:07:09. | |
perlhpux.pod | File | 30509 bytes | July 28 2025 08:07:09. | |
perlhurd.pod | File | 1993 bytes | July 28 2025 08:07:09. | |
perlintern.pod | File | 54572 bytes | July 28 2025 08:07:09. | |
perlinterp.pod | File | 33687 bytes | July 28 2025 08:07:09. | |
perlintro.pod | File | 22119 bytes | July 28 2025 08:07:09. | |
perliol.pod | File | 34185 bytes | July 28 2025 08:07:09. | |
perlipc.pod | File | 70829 bytes | July 28 2025 08:07:09. | |
perlirix.pod | File | 4395 bytes | July 28 2025 08:07:09. | |
perljp.pod | File | 7521 bytes | July 28 2025 08:07:09. | |
perlko.pod | File | 12259 bytes | July 28 2025 08:07:09. | |
perllexwarn.pod | File | 355 bytes | July 28 2025 08:07:09. | |
perllinux.pod | File | 1488 bytes | July 28 2025 08:07:09. | |
perllocale.pod | File | 68678 bytes | July 28 2025 08:07:09. | |
perllol.pod | File | 9580 bytes | July 28 2025 08:07:09. | |
perlmacos.pod | File | 1001 bytes | July 28 2025 08:07:09. | |
perlmacosx.pod | File | 12060 bytes | July 28 2025 08:07:09. | |
perlmod.pod | File | 26250 bytes | July 28 2025 08:07:09. | |
perlmodinstall.pod | File | 12792 bytes | July 28 2025 08:07:09. | |
perlmodlib.pod | File | 76482 bytes | July 28 2025 08:07:09. | |
perlmodstyle.pod | File | 22575 bytes | July 28 2025 08:07:09. | |
perlmroapi.pod | File | 3212 bytes | July 28 2025 08:07:09. | |
perlnetware.pod | File | 6648 bytes | July 28 2025 08:07:09. | |
perlnewmod.pod | File | 11036 bytes | July 28 2025 08:07:09. | |
perlnumber.pod | File | 8353 bytes | July 28 2025 08:07:09. | |
perlobj.pod | File | 35537 bytes | July 28 2025 08:07:09. | |
perlootut.pod | File | 26783 bytes | July 28 2025 08:07:09. | |
perlop.pod | File | 136252 bytes | July 28 2025 08:07:09. | |
perlopenbsd.pod | File | 1204 bytes | July 28 2025 08:07:09. | |
perlopentut.pod | File | 9455 bytes | July 28 2025 08:07:09. | |
perlos2.pod | File | 93351 bytes | July 28 2025 08:07:09. | |
perlos390.pod | File | 15674 bytes | July 28 2025 08:07:09. | |
perlos400.pod | File | 4768 bytes | July 28 2025 08:07:09. | |
perlpacktut.pod | File | 51282 bytes | July 28 2025 08:07:09. | |
perlperf.pod | File | 49881 bytes | July 28 2025 08:07:09. | |
perlplan9.pod | File | 5125 bytes | July 28 2025 08:07:09. | |
perlpod.pod | File | 22196 bytes | July 28 2025 08:07:09. | |
perlpodspec.pod | File | 68476 bytes | July 28 2025 08:07:09. | |
perlpolicy.pod | File | 25629 bytes | July 28 2025 08:07:09. | |
perlport.pod | File | 87602 bytes | July 28 2025 08:07:09. | |
perlpragma.pod | File | 5176 bytes | July 28 2025 08:07:09. | |
perlqnx.pod | File | 6673 bytes | July 28 2025 08:07:09. | |
perlre.pod | File | 120901 bytes | July 28 2025 08:07:09. | |
perlreapi.pod | File | 30334 bytes | July 28 2025 08:07:09. | |
perlrebackslash.pod | File | 31817 bytes | July 28 2025 08:07:09. | |
perlrecharclass.pod | File | 49029 bytes | July 28 2025 08:07:09. | |
perlref.pod | File | 35304 bytes | July 28 2025 08:07:09. | |
perlreftut.pod | File | 18790 bytes | July 28 2025 08:07:09. | |
perlreguts.pod | File | 38328 bytes | July 28 2025 08:07:09. | |
perlrepository.pod | File | 509 bytes | July 28 2025 08:07:09. | |
perlrequick.pod | File | 18497 bytes | July 28 2025 08:07:09. | |
perlreref.pod | File | 14744 bytes | July 28 2025 08:07:09. | |
perlretut.pod | File | 121257 bytes | July 28 2025 08:07:09. | |
perlriscos.pod | File | 1529 bytes | July 28 2025 08:07:09. | |
perlrun.pod | File | 53550 bytes | July 28 2025 08:07:09. | |
perlsec.pod | File | 26184 bytes | July 28 2025 08:07:09. | |
perlsolaris.pod | File | 29822 bytes | July 28 2025 08:07:09. | |
perlsource.pod | File | 6876 bytes | July 28 2025 08:07:09. | |
perlstyle.pod | File | 8630 bytes | July 28 2025 08:07:09. | |
perlsub.pod | File | 72967 bytes | July 28 2025 08:07:09. | |
perlsymbian.pod | File | 15359 bytes | July 28 2025 08:07:09. | |
perlsyn.pod | File | 44512 bytes | July 28 2025 08:07:09. | |
perlsynology.pod | File | 7778 bytes | July 28 2025 08:07:09. | |
perlthrtut.pod | File | 46459 bytes | July 28 2025 08:07:09. | |
perltie.pod | File | 38607 bytes | July 28 2025 08:07:09. | |
perltoc.pod | File | 694171 bytes | July 28 2025 08:07:09. | |
perltodo.pod | File | 376 bytes | July 28 2025 08:07:09. | |
perltooc.pod | File | 294 bytes | July 28 2025 08:07:09. | |
perltoot.pod | File | 294 bytes | July 28 2025 08:07:09. | |
perltrap.pod | File | 10620 bytes | July 28 2025 08:07:09. | |
perltru64.pod | File | 8492 bytes | July 28 2025 08:07:09. | |
perltw.pod | File | 4477 bytes | July 28 2025 08:07:09. | |
perlunicode.pod | File | 82491 bytes | July 28 2025 08:07:09. | |
perlunicook.pod | File | 25488 bytes | July 28 2025 08:07:09. | |
perlunifaq.pod | File | 13647 bytes | July 28 2025 08:07:09. | |
perluniintro.pod | File | 38340 bytes | July 28 2025 08:07:09. | |
perluniprops.pod | File | 285306 bytes | July 28 2025 08:07:09. | |
perlunitut.pod | File | 7951 bytes | July 28 2025 08:07:09. | |
perlutil.pod | File | 7640 bytes | July 28 2025 08:08:32. | |
perlvar.pod | File | 78364 bytes | July 28 2025 08:07:09. | |
perlvms.pod | File | 50823 bytes | July 28 2025 08:07:09. | |
perlvos.pod | File | 3843 bytes | July 28 2025 08:07:09. | |
perlwin32.pod | File | 39298 bytes | July 28 2025 08:07:09. | |
perlxs.pod | File | 78920 bytes | July 28 2025 08:07:09. | |
perlxstut.pod | File | 50095 bytes | July 28 2025 08:07:09. | |
perlxstypemap.pod | File | 24001 bytes | July 28 2025 08:07:09. |
=encoding utf8 =head1 NAME perlunicook - cookbookish examples of handling Unicode in Perl =head1 DESCRIPTION This manpage contains short recipes demonstrating how to handle common Unicode operations in Perl, plus one complete program at the end. Any undeclared variables in individual recipes are assumed to have a previous appropriate value in them. =head1 EXAMPLES =head2 ℞ 0: Standard preamble Unless otherwise notes, all examples below require this standard preamble to work correctly, with the C<#!> adjusted to work on your system: #!/usr/bin/env perl use utf8; # so literals and identifiers can be in UTF-8 use v5.12; # or later to get "unicode_strings" feature use strict; # quote strings, declare variables use warnings; # on by default use warnings qw(FATAL utf8); # fatalize encoding glitches use open qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8 use charnames qw(:full :short); # unneeded in v5.16 This I<does> make even Unix programmers C<binmode> your binary streams, or open them with C<:raw>, but that's the only way to get at them portably anyway. B<WARNING>: C<use autodie> (pre 2.26) and C<use open> do not get along with each other. =head2 ℞ 1: Generic Unicode-savvy filter Always decompose on the way in, then recompose on the way out. use Unicode::Normalize; while (<>) { $_ = NFD($_); # decompose + reorder canonically ... } continue { print NFC($_); # recompose (where possible) + reorder canonically } =head2 ℞ 2: Fine-tuning Unicode warnings As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings. use v5.14; # subwarnings unavailable any earlier no warnings "nonchar"; # the 66 forbidden non-characters no warnings "surrogate"; # UTF-16/CESU-8 nonsense no warnings "non_unicode"; # for codepoints over 0x10_FFFF =head2 ℞ 3: Declare source in utf8 for identifiers and literals Without the all-critical C<use utf8> declaration, putting UTF‑8 in your literals and identifiers won’t work right. If you used the standard preamble just given above, this already happened. If you did, you can do things like this: use utf8; my $measure = "Ångström"; my @μsoft = qw( cp852 cp1251 cp1252 ); my @ὑπέρμεγας = qw( ὑπέρ μεγας ); my @鯉 = qw( koi8-f koi8-u koi8-r ); my $motto = "👪 💗 🐪"; # FAMILY, GROWING HEART, DROMEDARY CAMEL If you forget C<use utf8>, high bytes will be misunderstood as separate characters, and nothing will work right. =head2 ℞ 4: Characters and their numbers The C<ord> and C<chr> functions work transparently on all codepoints, not just on ASCII alone — nor in fact, not even just on Unicode alone. # ASCII characters ord("A") chr(65) # characters from the Basic Multilingual Plane ord("Σ") chr(0x3A3) # beyond the BMP ord("𝑛") # MATHEMATICAL ITALIC SMALL N chr(0x1D45B) # beyond Unicode! (up to MAXINT) ord("\x{20_0000}") chr(0x20_0000) =head2 ℞ 5: Unicode literals by character number In an interpolated literal, whether a double-quoted string or a regex, you may specify a character by its number using the C<\x{I<HHHHHH>}> escape. String: "\x{3a3}" Regex: /\x{3a3}/ String: "\x{1d45b}" Regex: /\x{1d45b}/ # even non-BMP ranges in regex work fine /[\x{1D434}-\x{1D467}]/ =head2 ℞ 6: Get character name by number use charnames (); my $name = charnames::viacode(0x03A3); =head2 ℞ 7: Get character number by name use charnames (); my $number = charnames::vianame("GREEK CAPITAL LETTER SIGMA"); =head2 ℞ 8: Unicode named characters Use the C<< \N{I<charname>} >> notation to get the character by that name for use in interpolated literals (double-quoted strings and regexes). In v5.16, there is an implicit use charnames qw(:full :short); But prior to v5.16, you must be explicit about which set of charnames you want. The C<:full> names are the official Unicode character name, alias, or sequence, which all share a namespace. use charnames qw(:full :short latin greek); "\N{MATHEMATICAL ITALIC SMALL N}" # :full "\N{GREEK CAPITAL LETTER SIGMA}" # :full Anything else is a Perl-specific convenience abbreviation. Specify one or more scripts by names if you want short names that are script-specific. "\N{Greek:Sigma}" # :short "\N{ae}" # latin "\N{epsilon}" # greek The v5.16 release also supports a C<:loose> import for loose matching of character names, which works just like loose matching of property names: that is, it disregards case, whitespace, and underscores: "\N{euro sign}" # :loose (from v5.16) =head2 ℞ 9: Unicode named sequences These look just like character names but return multiple codepoints. Notice the C<%vx> vector-print functionality in C<printf>. use charnames qw(:full); my $seq = "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"; printf "U+%v04X\n", $seq; U+0100.0300 =head2 ℞ 10: Custom named characters Use C<:alias> to give your own lexically scoped nicknames to existing characters, or even to give unnamed private-use characters useful names. use charnames ":full", ":alias" => { ecute => "LATIN SMALL LETTER E WITH ACUTE", "APPLE LOGO" => 0xF8FF, # private use character }; "\N{ecute}" "\N{APPLE LOGO}" =head2 ℞ 11: Names of CJK codepoints Sinograms like “東京” come back with character names of C<CJK UNIFIED IDEOGRAPH-6771> and C<CJK UNIFIED IDEOGRAPH-4EAC>, because their “names” vary. The CPAN C<Unicode::Unihan> module has a large database for decoding these (and a whole lot more), provided you know how to understand its output. # cpan -i Unicode::Unihan use Unicode::Unihan; my $str = "東京"; my $unhan = Unicode::Unihan->new; for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) { printf "CJK $str in %-12s is ", $lang; say $unhan->$lang($str); } prints: CJK 東京 in Mandarin is DONG1JING1 CJK 東京 in Cantonese is dung1ging1 CJK 東京 in Korean is TONGKYENG CJK 東京 in JapaneseOn is TOUKYOU KEI KIN CJK 東京 in JapaneseKun is HIGASHI AZUMAMIYAKO If you have a specific romanization scheme in mind, use the specific module: # cpan -i Lingua::JA::Romanize::Japanese use Lingua::JA::Romanize::Japanese; my $k2r = Lingua::JA::Romanize::Japanese->new; my $str = "東京"; say "Japanese for $str is ", $k2r->chars($str); prints Japanese for 東京 is toukyou =head2 ℞ 12: Explicit encode/decode On rare occasion, such as a database read, you may be given encoded text you need to decode. use Encode qw(encode decode); my $chars = decode("shiftjis", $bytes, 1); # OR my $bytes = encode("MIME-Header-ISO_2022_JP", $chars, 1); For streams all in the same encoding, don't use encode/decode; instead set the file encoding when you open the file or immediately after with C<binmode> as described later below. =head2 ℞ 13: Decode program arguments as utf8 $ perl -CA ... or $ export PERL_UNICODE=A or use Encode qw(decode); @ARGV = map { decode('UTF-8', $_, 1) } @ARGV; =head2 ℞ 14: Decode program arguments as locale encoding # cpan -i Encode::Locale use Encode qw(locale); use Encode::Locale; # use "locale" as an arg to encode/decode @ARGV = map { decode(locale => $_, 1) } @ARGV; =head2 ℞ 15: Declare STD{IN,OUT,ERR} to be utf8 Use a command-line option, an environment variable, or else call C<binmode> explicitly: $ perl -CS ... or $ export PERL_UNICODE=S or use open qw(:std :encoding(UTF-8)); or binmode(STDIN, ":encoding(UTF-8)"); binmode(STDOUT, ":utf8"); binmode(STDERR, ":utf8"); =head2 ℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding # cpan -i Encode::Locale use Encode; use Encode::Locale; # or as a stream for binmode or open binmode STDIN, ":encoding(console_in)" if -t STDIN; binmode STDOUT, ":encoding(console_out)" if -t STDOUT; binmode STDERR, ":encoding(console_out)" if -t STDERR; =head2 ℞ 17: Make file I/O default to utf8 Files opened without an encoding argument will be in UTF-8: $ perl -CD ... or $ export PERL_UNICODE=D or use open qw(:encoding(UTF-8)); =head2 ℞ 18: Make all I/O and args default to utf8 $ perl -CSDA ... or $ export PERL_UNICODE=SDA or use open qw(:std :encoding(UTF-8)); use Encode qw(decode); @ARGV = map { decode('UTF-8', $_, 1) } @ARGV; =head2 ℞ 19: Open file with specific encoding Specify stream encoding. This is the normal way to deal with encoded text, not by calling low-level functions. # input file open(my $in_file, "< :encoding(UTF-16)", "wintext"); OR open(my $in_file, "<", "wintext"); binmode($in_file, ":encoding(UTF-16)"); THEN my $line = <$in_file>; # output file open($out_file, "> :encoding(cp1252)", "wintext"); OR open(my $out_file, ">", "wintext"); binmode($out_file, ":encoding(cp1252)"); THEN print $out_file "some text\n"; More layers than just the encoding can be specified here. For example, the incantation C<":raw :encoding(UTF-16LE) :crlf"> includes implicit CRLF handling. =head2 ℞ 20: Unicode casing Unicode casing is very different from ASCII casing. uc("henry ⅷ") # "HENRY Ⅷ" uc("tschüß") # "TSCHÜSS" notice ß => SS # both are true: "tschüß" =~ /TSCHÜSS/i # notice ß => SS "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i # notice Σ,σ,ς sameness =head2 ℞ 21: Unicode case-insensitive comparisons Also available in the CPAN L<Unicode::CaseFold> module, the new C<fc> “foldcase” function from v5.16 grants access to the same Unicode casefolding as the C</i> pattern modifier has always used: use feature "fc"; # fc() function is from v5.16 # sort case-insensitively my @sorted = sort { fc($a) cmp fc($b) } @list; # both are true: fc("tschüß") eq fc("TSCHÜSS") fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ") =head2 ℞ 22: Match Unicode linebreak sequence in regex A Unicode linebreak matches the two-character CRLF grapheme or any of seven vertical whitespace characters. Good for dealing with textfiles coming from different operating systems. \R s/\R/\n/g; # normalize all linebreaks to \n =head2 ℞ 23: Get character category Find the general category of a numeric codepoint. use Unicode::UCD qw(charinfo); my $cat = charinfo(0x3A3)->{category}; # "Lu" =head2 ℞ 24: Disabling Unicode-awareness in builtin charclasses Disable C<\w>, C<\b>, C<\s>, C<\d>, and the POSIX classes from working correctly on Unicode either in this scope, or in just one regex. use v5.14; use re "/a"; # OR my($num) = $str =~ /(\d+)/a; Or use specific un-Unicode properties, like C<\p{ahex}> and C<\p{POSIX_Digit>}. Properties still work normally no matter what charset modifiers (C</d /u /l /a /aa>) should be effect. =head2 ℞ 25: Match Unicode properties in regex with \p, \P These all match a single codepoint with the given property. Use C<\P> in place of C<\p> to match one codepoint lacking that property. \pL, \pN, \pS, \pP, \pM, \pZ, \pC \p{Sk}, \p{Ps}, \p{Lt} \p{alpha}, \p{upper}, \p{lower} \p{Latin}, \p{Greek} \p{script_extensions=Latin}, \p{scx=Greek} \p{East_Asian_Width=Wide}, \p{EA=W} \p{Line_Break=Hyphen}, \p{LB=HY} \p{Numeric_Value=4}, \p{NV=4} =head2 ℞ 26: Custom character properties Define at compile-time your own custom character properties for use in regexes. # using private-use characters sub In_Tengwar { "E000\tE07F\n" } if (/\p{In_Tengwar}/) { ... } # blending existing properties sub Is_GraecoRoman_Title {<<'END_OF_SET'} +utf8::IsLatin +utf8::IsGreek &utf8::IsTitle END_OF_SET if (/\p{Is_GraecoRoman_Title}/ { ... } =head2 ℞ 27: Unicode normalization Typically render into NFD on input and NFC on output. Using NFKC or NFKD functions improves recall on searches, assuming you've already done to the same text to be searched. Note that this is about much more than just pre- combined compatibility glyphs; it also reorders marks according to their canonical combining classes and weeds out singletons. use Unicode::Normalize; my $nfd = NFD($orig); my $nfc = NFC($orig); my $nfkd = NFKD($orig); my $nfkc = NFKC($orig); =head2 ℞ 28: Convert non-ASCII Unicode numerics Unless you’ve used C</a> or C</aa>, C<\d> matches more than ASCII digits only, but Perl’s implicit string-to-number conversion does not current recognize these. Here’s how to convert such strings manually. use v5.14; # needed for num() function use Unicode::UCD qw(num); my $str = "got Ⅻ and ४५६७ and ⅞ and here"; my @nums = (); while ($str =~ /(\d+|\N)/g) { # not just ASCII! push @nums, num($1); } say "@nums"; # 12 4567 0.875 use charnames qw(:full); my $nv = num("\N{RUMI DIGIT ONE}\N{RUMI DIGIT TWO}"); =head2 ℞ 29: Match Unicode grapheme cluster in regex Programmer-visible “characters” are codepoints matched by C</./s>, but user-visible “characters” are graphemes matched by C</\X/>. # Find vowel *plus* any combining diacritics,underlining,etc. my $nfd = NFD($orig); $nfd =~ / (?=[aeiou]) \X /xi =head2 ℞ 30: Extract by grapheme instead of by codepoint (regex) # match and grab five first graphemes my($first_five) = $str =~ /^ ( \X{5} ) /x; =head2 ℞ 31: Extract by grapheme instead of by codepoint (substr) # cpan -i Unicode::GCString use Unicode::GCString; my $gcs = Unicode::GCString->new($str); my $first_five = $gcs->substr(0, 5); =head2 ℞ 32: Reverse string by grapheme Reversing by codepoint messes up diacritics, mistakenly converting C<crème brûlée> into C<éel̂urb em̀erc> instead of into C<eélûrb emèrc>; so reverse by grapheme instead. Both these approaches work right no matter what normalization the string is in: $str = join("", reverse $str =~ /\X/g); # OR: cpan -i Unicode::GCString use Unicode::GCString; $str = reverse Unicode::GCString->new($str); =head2 ℞ 33: String length in graphemes The string C<brûlée> has six graphemes but up to eight codepoints. This counts by grapheme, not by codepoint: my $str = "brûlée"; my $count = 0; while ($str =~ /\X/g) { $count++ } # OR: cpan -i Unicode::GCString use Unicode::GCString; my $gcs = Unicode::GCString->new($str); my $count = $gcs->length; =head2 ℞ 34: Unicode column-width for printing Perl’s C<printf>, C<sprintf>, and C<format> think all codepoints take up 1 print column, but many take 0 or 2. Here to show that normalization makes no difference, we print out both forms: use Unicode::GCString; use Unicode::Normalize; my @words = qw/crème brûlée/; @words = map { NFC($_), NFD($_) } @words; for my $str (@words) { my $gcs = Unicode::GCString->new($str); my $cols = $gcs->columns; my $pad = " " x (10 - $cols); say str, $pad, " |"; } generates this to show that it pads correctly no matter the normalization: crème | crème | brûlée | brûlée | =head2 ℞ 35: Unicode collation Text sorted by numeric codepoint follows no reasonable alphabetic order; use the UCA for sorting text. use Unicode::Collate; my $col = Unicode::Collate->new(); my @list = $col->sort(@old_list); See the I<ucsort> program from the L<Unicode::Tussle> CPAN module for a convenient command-line interface to this module. =head2 ℞ 36: Case- I<and> accent-insensitive Unicode sort Specify a collation strength of level 1 to ignore case and diacritics, only looking at the basic character. use Unicode::Collate; my $col = Unicode::Collate->new(level => 1); my @list = $col->sort(@old_list); =head2 ℞ 37: Unicode locale collation Some locales have special sorting rules. # either use v5.12, OR: cpan -i Unicode::Collate::Locale use Unicode::Collate::Locale; my $col = Unicode::Collate::Locale->new(locale => "de__phonebook"); my @list = $col->sort(@old_list); The I<ucsort> program mentioned above accepts a C<--locale> parameter. =head2 ℞ 38: Making C<cmp> work on text instead of codepoints Instead of this: @srecs = sort { $b->{AGE} <=> $a->{AGE} || $a->{NAME} cmp $b->{NAME} } @recs; Use this: my $coll = Unicode::Collate->new(); for my $rec (@recs) { $rec->{NAME_key} = $coll->getSortKey( $rec->{NAME} ); } @srecs = sort { $b->{AGE} <=> $a->{AGE} || $a->{NAME_key} cmp $b->{NAME_key} } @recs; =head2 ℞ 39: Case- I<and> accent-insensitive comparisons Use a collator object to compare Unicode text by character instead of by codepoint. use Unicode::Collate; my $es = Unicode::Collate->new( level => 1, normalization => undef ); # now both are true: $es->eq("García", "GARCIA" ); $es->eq("Márquez", "MARQUEZ"); =head2 ℞ 40: Case- I<and> accent-insensitive locale comparisons Same, but in a specific locale. my $de = Unicode::Collate::Locale->new( locale => "de__phonebook", ); # now this is true: $de->eq("tschüß", "TSCHUESS"); # notice ü => UE, ß => SS =head2 ℞ 41: Unicode linebreaking Break up text into lines according to Unicode rules. # cpan -i Unicode::LineBreak use Unicode::LineBreak; use charnames qw(:full); my $para = "This is a super\N{HYPHEN}long string. " x 20; my $fmt = Unicode::LineBreak->new; print $fmt->break($para), "\n"; =head2 ℞ 42: Unicode text in DBM hashes, the tedious way Using a regular Perl string as a key or value for a DBM hash will trigger a wide character exception if any codepoints won’t fit into a byte. Here’s how to manually manage the translation: use DB_File; use Encode qw(encode decode); tie %dbhash, "DB_File", "pathname"; # STORE # assume $uni_key and $uni_value are abstract Unicode strings my $enc_key = encode("UTF-8", $uni_key, 1); my $enc_value = encode("UTF-8", $uni_value, 1); $dbhash{$enc_key} = $enc_value; # FETCH # assume $uni_key holds a normal Perl string (abstract Unicode) my $enc_key = encode("UTF-8", $uni_key, 1); my $enc_value = $dbhash{$enc_key}; my $uni_value = decode("UTF-8", $enc_value, 1); =head2 ℞ 43: Unicode text in DBM hashes, the easy way Here’s how to implicitly manage the translation; all encoding and decoding is done automatically, just as with streams that have a particular encoding attached to them: use DB_File; use DBM_Filter; my $dbobj = tie %dbhash, "DB_File", "pathname"; $dbobj->Filter_Value("utf8"); # this is the magic bit # STORE # assume $uni_key and $uni_value are abstract Unicode strings $dbhash{$uni_key} = $uni_value; # FETCH # $uni_key holds a normal Perl string (abstract Unicode) my $uni_value = $dbhash{$uni_key}; =head2 ℞ 44: PROGRAM: Demo of Unicode collation and printing Here’s a full program showing how to make use of locale-sensitive sorting, Unicode casing, and managing print widths when some of the characters take up zero or two columns, not just one column each time. When run, the following program produces this nicely aligned output: Crème Brûlée....... €2.00 Éclair............. €1.60 Fideuà............. €4.20 Hamburger.......... €6.00 Jamón Serrano...... €4.45 Linguiça........... €7.00 Pâté............... €4.15 Pears.............. €2.00 Pêches............. €2.25 Smørbrød........... €5.75 Spätzle............ €5.50 Xoriço............. €3.00 Γύρος.............. €6.50 막걸리............. €4.00 おもち............. €2.65 お好み焼き......... €8.00 シュークリーム..... €1.85 寿司............... €9.99 包子............... €7.50 Here's that program; tested on v5.14. #!/usr/bin/env perl # umenu - demo sorting and printing of Unicode food # # (obligatory and increasingly long preamble) # use utf8; use v5.14; # for locale sorting use strict; use warnings; use warnings qw(FATAL utf8); # fatalize encoding faults use open qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8 use charnames qw(:full :short); # unneeded in v5.16 # std modules use Unicode::Normalize; # std perl distro as of v5.8 use List::Util qw(max); # std perl distro as of v5.10 use Unicode::Collate::Locale; # std perl distro as of v5.14 # cpan modules use Unicode::GCString; # from CPAN # forward defs sub pad($$$); sub colwidth(_); sub entitle(_); my %price = ( "γύρος" => 6.50, # gyros "pears" => 2.00, # like um, pears "linguiça" => 7.00, # spicy sausage, Portuguese "xoriço" => 3.00, # chorizo sausage, Catalan "hamburger" => 6.00, # burgermeister meisterburger "éclair" => 1.60, # dessert, French "smørbrød" => 5.75, # sandwiches, Norwegian "spätzle" => 5.50, # Bayerisch noodles, little sparrows "包子" => 7.50, # bao1 zi5, steamed pork buns, Mandarin "jamón serrano" => 4.45, # country ham, Spanish "pêches" => 2.25, # peaches, French "シュークリーム" => 1.85, # cream-filled pastry like eclair "막걸리" => 4.00, # makgeolli, Korean rice wine "寿司" => 9.99, # sushi, Japanese "おもち" => 2.65, # omochi, rice cakes, Japanese "crème brûlée" => 2.00, # crema catalana "fideuà" => 4.20, # more noodles, Valencian # (Catalan=fideuada) "pâté" => 4.15, # gooseliver paste, French "お好み焼き" => 8.00, # okonomiyaki, Japanese ); my $width = 5 + max map { colwidth } keys %price; # So the Asian stuff comes out in an order that someone # who reads those scripts won't freak out over; the # CJK stuff will be in JIS X 0208 order that way. my $coll = Unicode::Collate::Locale->new(locale => "ja"); for my $item ($coll->sort(keys %price)) { print pad(entitle($item), $width, "."); printf " €%.2f\n", $price{$item}; } sub pad($$$) { my($str, $width, $padchar) = @_; return $str . ($padchar x ($width - colwidth($str))); } sub colwidth(_) { my($str) = @_; return Unicode::GCString->new($str)->columns; } sub entitle(_) { my($str) = @_; $str =~ s{ (?=\pL)(\S) (\S*) } { ucfirst($1) . lc($2) }xge; return $str; } =head1 SEE ALSO See these manpages, some of which are CPAN modules: L<perlunicode>, L<perluniprops>, L<perlre>, L<perlrecharclass>, L<perluniintro>, L<perlunitut>, L<perlunifaq>, L<PerlIO>, L<DB_File>, L<DBM_Filter>, L<DBM_Filter::utf8>, L<Encode>, L<Encode::Locale>, L<Unicode::UCD>, L<Unicode::Normalize>, L<Unicode::GCString>, L<Unicode::LineBreak>, L<Unicode::Collate>, L<Unicode::Collate::Locale>, L<Unicode::Unihan>, L<Unicode::CaseFold>, L<Unicode::Tussle>, L<Lingua::JA::Romanize::Japanese>, L<Lingua::ZH::Romanize::Pinyin>, L<Lingua::KO::Romanize::Hangul>. The L<Unicode::Tussle> CPAN module includes many programs to help with working with Unicode, including these programs to fully or partly replace standard utilities: I<tcgrep> instead of I<egrep>, I<uniquote> instead of I<cat -v> or I<hexdump>, I<uniwc> instead of I<wc>, I<unilook> instead of I<look>, I<unifmt> instead of I<fmt>, and I<ucsort> instead of I<sort>. For exploring Unicode character names and character properties, see its I<uniprops>, I<unichars>, and I<uninames> programs. It also supplies these programs, all of which are general filters that do Unicode-y things: I<unititle> and I<unicaps>; I<uniwide> and I<uninarrow>; I<unisupers> and I<unisubs>; I<nfd>, I<nfc>, I<nfkd>, and I<nfkc>; and I<uc>, I<lc>, and I<tc>. Finally, see the published Unicode Standard (page numbers are from version 6.0.0), including these specific annexes and technical reports: =over =item §3.13 Default Case Algorithms, page 113; §4.2 Case, pages 120–122; Case Mappings, page 166–172, especially Caseless Matching starting on page 170. =item UAX #44: Unicode Character Database =item UTS #18: Unicode Regular Expressions =item UAX #15: Unicode Normalization Forms =item UTS #10: Unicode Collation Algorithm =item UAX #29: Unicode Text Segmentation =item UAX #14: Unicode Line Breaking Algorithm =item UAX #11: East Asian Width =back =head1 AUTHOR Tom Christiansen E<lt>tchrist@perl.comE<gt> wrote this, with occasional kibbitzing from Larry Wall and Jeffrey Friedl in the background. =head1 COPYRIGHT AND LICENCE Copyright © 2012 Tom Christiansen. This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself. Most of these examples taken from the current edition of the “Camel Book”; that is, from the 4ᵗʰ Edition of I<Programming Perl>, Copyright © 2012 Tom Christiansen <et al.>, 2012-02-13 by O’Reilly Media. The code itself is freely redistributable, and you are encouraged to transplant, fold, spindle, and mutilate any of the examples in this manpage however you please for inclusion into your own programs without any encumbrance whatsoever. Acknowledgement via code comment is polite but not required. =head1 REVISION HISTORY v1.0.0 – first public release, 2012-02-27
SILENT KILLER Tool