Documentation on iconv
iconv = Convert string to requested character encoding
Performs a character set conversion on the string str from in_charset to out_charset.
in_charset The input charset. out_charset The output charset. If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can't be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, E_NOTICE is generated and the function will return FALSE. str The string to be converted.
Usage, params, and more on iconv
string iconv ( string $in_charset
, string $out_charset
, string $str
)
in_charset
The input charset. out_charset
The output charset. If you append the string //TRANSLIT to out_charset
transliteration is activated. This means that when a character can't be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, E_NOTICE
is generated and the function will return FALSE
. str
The string to be converted.
Returns the converted string or FALSE
on failure.
Notes and warnings on iconv
Basic example of how to use: iconv
Example #1 iconv() example
<?php
$text = "This is the Euro symbol '€'.";
echo 'Original : ', $text, PHP_EOL;
echo 'TRANSLIT : ', iconv("UTF-8", "ISO-8859-1//TRANSLIT", $text), PHP_EOL;
echo 'IGNORE : ', iconv("UTF-8", "ISO-8859-1//IGNORE", $text), PHP_EOL;
echo 'Plain : ', iconv("UTF-8", "ISO-8859-1", $text), PHP_EOL;
?>
The above example will output something similar to:
Original : This is the Euro symbol '€'. TRANSLIT : This is the Euro symbol 'EUR'. IGNORE : This is the Euro symbol ''. Plain : Notice: iconv(): Detected an illegal character in input string in .\iconv-example.php on line 7
Other code examples of iconv being used
Please note that iconv('UTF-8', 'ASCII//TRANSLIT', ...) doesn't work properly when locale category LC_CTYPE is set to C or POSIX. You must choose another locale otherwise all non-ASCII characters will be replaced with question marks. This is at least true with glibc 2.5.
Example:
<?php
setlocale(LC_CTYPE, 'POSIX');
echo iconv('UTF-8', 'ASCII//TRANSLIT', "Žluťoučký kůň\n");
// ?lu?ou?k? k??
setlocale(LC_CTYPE, 'cs_CZ');
echo iconv('UTF-8', 'ASCII//TRANSLIT', "Žluťoučký kůň\n");
// Zlutoucky kun
?>
Interestingly, setting different target locales results in different, yet appropriate, transliterations. For example:
<?php
//some German
$utf8_sentence = 'Weiß, Goldmann, Göbel, Weiss, Göthe, Goethe und Götz';
//UK
setlocale(LC_ALL, 'en_GB');
//transliterate
$trans_sentence = iconv('UTF-8', 'ASCII//TRANSLIT', $utf8_sentence);
//gives [Weiss, Goldmann, Gobel, Weiss, Gothe, Goethe und Gotz]
//which is our original string flattened into 7-bit ASCII as
//an English speaker would do it (ie. simply remove the umlauts)
echo $trans_sentence . PHP_EOL;
//Germany
setlocale(LC_ALL, 'de_DE');
$trans_sentence = iconv('UTF-8', 'ASCII//TRANSLIT', $utf8_sentence);
//gives [Weiss, Goldmann, Goebel, Weiss, Goethe, Goethe und Goetz]
//which is exactly how a German would transliterate those
//umlauted characters if forced to use 7-bit ASCII!
//(because really ä = ae, ö = oe and ü = ue)
echo $trans_sentence . PHP_EOL;
?>
to test different combinations of convertions between charsets (when we don't know the source charset and what is the convenient destination charset) this is an example :
<?php
$tab = array("UTF-8", "ASCII", "Windows-1252", "ISO-8859-15", "ISO-8859-1", "ISO-8859-6", "CP1256");
$chain = "";
foreach ($tab as $i)
{
foreach ($tab as $j)
{
$chain .= " $i$j ".iconv($i, $j, "$my_string");
}
}
echo $chain;
?>
then after displaying, you use the $i$j that shows good displaying.
NB: you can add other charsets to $tab to test other cases.
If you are getting question-marks in your iconv output when transliterating, be sure to 'setlocale' to something your system supports.
Some PHP CMS's will default setlocale to 'C', this can be a problem.
use the "locale" command to find out a list..
$ locale -a
C
en_AU.utf8
POSIX
<?php
setlocale(LC_CTYPE, 'en_AU.utf8');
$str = iconv('UTF-8', 'ASCII//TRANSLIT', "Côte d'Ivoire");
?>
Here is a code to convert ISO 8859-1 to UTF-8 and vice versa without using iconv.
<?php
//Logic from http://twiki.org/cgi-bin/view/Codev/InternationalisationUTF8
$str_iso8859_1 = 'foo in ISO 8859-1';
//ISO 8859-1 to UTF-8
$str_utf8 = preg_replace("/([\x80-\xFF])/e",
"chr(0xC0|ord('\\1')>>6).chr(0x80|ord('\\1')&0x3F)",
$str_iso8859_1);
//UTF-8 to ISO 8859-1
$str_iso8859_1 = preg_replace("/([\xC2\xC3])([\x80-\xBF])/e",
"chr(ord('\\1')<<6&0xC0|ord('\\2')&0x3F)",
$str_utf8);
?>
HTH,
R. Rajesh Jeba Anbiah
In my case, I had to change:
<?php
setlocale(LC_CTYPE, 'cs_CZ');
?>
to
<?php
setlocale(LC_CTYPE, 'cs_CZ.UTF-8');
?>
Otherwise it returns question marks.
When I asked my linux for locale (by locale command) it returns "cs_CZ.UTF-8", so there is maybe correlation between it.
iconv (GNU libc) 2.6.1
glibc 2.3.6
You can use 'CP1252' instead of 'Windows-1252':
<?php
// These two lines are equivalent
$result = iconv('Windows-1252', 'UTF-8', $string);
$result = iconv('CP1252', 'UTF-8', $string);
?>
Note: The following code points are not valid in CP1252 and will cause errors.
129 (0x81)
141 (0x8D)
143 (0x8F)
144 (0x90)
157 (0x9D)
Use the following instead:
<?php
// Remove invalid code points, convert everything else
$result = iconv('CP1252', 'UTF-8//IGNORE', $string);
?>
So, as iconv() does not always work correctly, in most cases, much easier to use htmlentities().
Example: <?php $content=htmlentities(file_get_contents("incoming.txt"), ENT_QUOTES, "Windows-1252"); file_put_contents("outbound.txt", html_entity_decode($content, ENT_QUOTES , "utf-8")); ?>
Here is how to convert UCS-2 numbers to UTF-8 numbers in hex:
<?php
function ucs2toutf8($str)
{
for ($i=0;$i<strlen($str);$i+=4)
{
$substring1 = $str[$i].$str[$i+1];
$substring2 = $str[$i+2].$str[$i+3];
if ($substring1 == "00")
{
$byte1 = "";
$byte2 = $substring2;
}
else
{
$substring = $substring1.$substring2;
$byte1 = dechex(192+(hexdec($substring)/64));
$byte2 = dechex(128+(hexdec($substring)%64));
}
$utf8 .= $byte1.$byte2;
}
return $utf8;
}
echo strtoupper(ucs2toutf8("06450631062D0020"));
?>
Input:
06450631062D
Output:
D985D8B1D8AD
regards,
Ziyad
Like many other people, I have encountered massive problems when using iconv() to convert between encodings (from UTF-8 to ISO-8859-15 in my case), especially on large strings.
The main problem here is that when your string contains illegal UTF-8 characters, there is no really straight forward way to handle those. iconv() simply (and silently!) terminates the string when encountering the problematic characters (also if using //IGNORE), returning a clipped string. The
<?php
$newstring = html_entity_decode(htmlentities($oldstring, ENT_QUOTES, 'UTF-8'), ENT_QUOTES , 'ISO-8859-15');
?>
workaround suggested here and elsewhere will also break when encountering illegal characters, at least dropping a useful note ("htmlentities(): Invalid multibyte sequence in argument in...")
I have found a lot of hints, suggestions and alternative methods (it's scary and in my opinion no good sign how many ways PHP natively provides to convert the encoding of strings), but none of them really worked, except for this one:
<?php
$newstring = mb_convert_encoding($oldstring, 'ISO-8859-15', 'UTF-8');
?>
Here is how to convert UTF-8 numbers to UCS-2 numbers in hex:
<?php
function utf8toucs2($str)
{
for ($i=0;$i<strlen($str);$i+=2)
{
$substring1 = $str[$i].$str[$i+1];
$substring2 = $str[$i+2].$str[$i+3];
if (hexdec($substring1) < 127)
$results = "00".$str[$i].$str[$i+1];
else
{
$results = dechex((hexdec($substring1)-192)*64 + (hexdec($substring2)-128));
if ($results < 1000) $results = "0".$results;
$i+=2;
}
$ucs2 .= $results;
}
return $ucs2;
}
echo strtoupper(utf8toucs2("D985D8B1D8AD"))."\n";
echo strtoupper(utf8toucs2("456725"))."\n";
?>
Input:
D985D8B1D8AD
Output:
06450631062D
Input:
456725
Output:
004500670025
On some systems there may be no such function as iconv(); this is due to the following reason: a constant is defined named `iconv` with the value `libiconv`. So, the string PHP_FUNCTION(iconv) transforms to PHP_FUNCTION(libiconv), and you have to call libiconv() function instead of iconv().
I had seen this on FreeBSD, but I am sure that was a rather special build.
If you'd want not to be dependent on this behaviour, add the following to your script:
<?php
if (!function_exists('iconv') && function_exists('libiconv')) {
function iconv($input_encoding, $output_encoding, $string) {
return libiconv($input_encoding, $output_encoding, $string);
}
}
?>
Thanks to tony2001 at phpclub.net for explaining this behaviour.
mirek code, dated 16-May-2008 10:17, added the characters `^~'" to the output.
This function will strip out these extra characters:
<?php
setlocale(LC_ALL, 'en_US.UTF8');
function clearUTF($s)
{
$r = '';
$s1 = @iconv('UTF-8', 'ASCII//TRANSLIT', $s);
$j = 0;
for ($i = 0; $i < strlen($s1); $i++) {
$ch1 = $s1[$i];
$ch2 = @mb_substr($s, $j++, 1, 'UTF-8');
if (strstr('`^~\'"', $ch1) !== false) {
if ($ch1 <> $ch2) {
--$j;
continue;
}
}
$r .= ($ch1=='?') ? $ch2 : $ch1;
}
return $r;
}
?>
Many mail servers don't handle utf-8 correctly as they assume iso-8859-x encodings, so you would want to convert the headers, subject and body of an email prior to sending it out.
If iconv() and mb_convert_encoding() are missing the following function can be used to convert UTF8 to iso-8859-7 encoding. It discards all characters that are not 2-byte greek characters or single-byte (ascii).
<?php
function conv_utf8_iso8859_7($s) {
$len = strlen($s);
$out = "";
$curr_char = "";
for($i=0; $i < $len; $i++) {
$curr_char .= $s[$i];
if( ( ord($s[$i]) & (128+64) ) == 128) {
//character end found
if ( strlen($curr_char) == 2) {
// 2-byte character check for it is greek one and convert
if (ord($curr_char[0])==205) $out .= chr( ord($curr_char[1])+16 );
else if (ord($curr_char[0])==206) $out .= chr( ord($curr_char[1])+48 );
else if (ord($curr_char[0])==207) $out .= chr( ord($curr_char[1])+112 );
else ; // non greek 2-byte character, discard character
} else ;// n-byte character, n>2, discard character
$curr_char = "";
} else if (ord($s[$i]) < 128) {
// character is one byte (ascii)
$out .= $curr_char;
$curr_char = "";
}
}
return $out;
}
?>
<?php
//script from http://zizi.kxup.com/
//javascript unesape
function unescape($str) {
$str = rawurldecode($str);
preg_match_all("/(?:%u.{4})|&#x.{4};|&#\d+;|.+/U",$str,$r);
$ar = $r[0];
print_r($ar);
foreach($ar as $k=>$v) {
if(substr($v,0,2) == "%u")
$ar[$k] = iconv("UCS-2","UTF-8",pack("H4",substr($v,-4)));
elseif(substr($v,0,3) == "&#x")
$ar[$k] = iconv("UCS-2","UTF-8",pack("H4",substr($v,3,-1)));
elseif(substr($v,0,2) == "&#") {
echo substr($v,2,-1)."<br>";
$ar[$k] = iconv("UCS-2","UTF-8",pack("n",substr($v,2,-1)));
}
}
return join("",$ar);
}
?>
Here is an example how to convert windows-1251 (windows) or cp1251(Linux/Unix) encoded string to UTF-8 encoding.
<?php
function cp1251_utf8( $sInput )
{
$sOutput = "";
for ( $i = 0; $i < strlen( $sInput ); $i++ )
{
$iAscii = ord( $sInput[$i] );
if ( $iAscii >= 192 && $iAscii <= 255 )
$sOutput .= "&#".( 1040 + ( $iAscii - 192 ) ).";";
else if ( $iAscii == 168 )
$sOutput .= "&#".( 1025 ).";";
else if ( $iAscii == 184 )
$sOutput .= "&#".( 1105 ).";";
else
$sOutput .= $sInput[$i];
}
return $sOutput;
}
?>
On my system, according to tests, and also as reported by other people elsewhere, you can combine TRANSLIT and IGNORE only by appending
//IGNORE//TRANSLIT
strictly in that order, but NOT by appending //TRANSLIT//IGNORE, which would lead to //IGNORE being ignored ( :) ).
Anyway, it's hard to understand how one could devise a system of passing options that does not allow to couple both options in a neat manner, and also to understand why the default behaviour should be the less useful and most dangerous one (throwing away most of your data at the first unexpected character). Software design FAIL :-/
If you want to normalize a filename on Mac OS X, because it is in UTF-8 NFD and you need UTF-8 NFC
(See: http://en.wikipedia.org/wiki/Unicode_equivalence#Combining_and_precomposed_characters)
you may use:
<?php
$filename_nfc = iconv("UTF-8-MAC", "UTF-8", $filename_nfd);
?>
For transcoding values in an Excel generated CSV the following seems to work:
<?php
$value = iconv('Windows-1252', 'UTF-8//TRANSLIT', $value);
?>
Turkish characters is error with the used set_locale and strftime commands.
My key example:
<?php iconv('','UTF-8',strftime('%d %B %Y %A')); ?>
To strip bogus characters from your input (such as data from an unsanitized or other source which you can't trust to necessarily give you strings encoded according to their advertised encoding set), use the same character set as both the input and the output, with //IGNORE on the output charcter set.
<?php
// assuming '†' is actually UTF8, htmlentities will assume it's iso-8859
// since we did not specify in the 3rd argument of htmlentities.
// This generates "â[bad utf-8 character]"
// If passed to any libxml, it will generate a fatal error.
$badUTF8 = htmlentities('†');
// iconv() can ignore characters which cannot be encoded in the target character set
$goodUTF8 = iconv("utf-8", "utf-8//IGNORE", $badUTF8);
?>
The result of the example does not give you back the dagger character which was the original input (it got lost when htmlentities was misused to encode it incorrectly, though this is common from people not accustomed to dealing with extended character sets), but it does at least give you data which is sane in your target character set.