htmlentities and utf8

Ever wonder why your utf8 charactors doesn't work with htmlentities?
try htmlspecialchars
Example:
$sResult = htmlspecialchars($sText, ENT_COMPAT, "UTF-8");

MySQL data contains chinese charactor become äöüßæåéé

It took me hours figuring this.

All my mysql data became symbols.

Then finally, i found the culprit.

1st, i converted my field and default charset of my table to unicode.

But i did not update the existing data to unicode. So it was my mistake.

2nd, in php, i changed my mysql_set_charset("UTF-8");

So now all the data from the table which contains original latin will be forced to utf-8.

So the symbol appear in my html output, so does phpmyadmin.

To fix this, i wrote a program to retrieve the data from latin charset, and

then output it as a code in array to replace the data later back to the database as utf-8.

Example:

mysql_set_charset("latin1", $conndb);

mysql_query("select ...");

$aData = mysql_fetch_assoc($query);

mysql_set_charset("utf-8", $conndb);

$aData["fieldname"] = utf8_encode($aData["fieldname"]);

//then update those record with newly encoded utf8

mysql_query("update table ... ");

so, the next time we query the database, ensure to use

mysql_set_charset("utf-8", $conndb);

utf encoding and json

If someone tell you that json doesnt work with utf8, they are wrong.

It does support utf8.

The main problem is the database side.

Ensure to set encoding of mysql connection charset to utf8 to have it working properly.

mysql_set_charset("utf8", $conn);//support for unicode

MODx Revolution and UTF8

If you ever need to insert in latin symbol or unicode into modx title or content...

Modify modx_site_content fields:

+ pagetitle, longtitle, description, introtext, content

to unicode_general_ci encoding.

(Note: This is a fulltext index, so you will have to execute 1 statement to alter all 5 fields)

And also modify template variable content:
ALTER TABLE  `modx_site_tmplvar_contentvalues` CHANGE  `value`  `value` TEXT CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL

Next, modify core/config/config.inc.php and change the line:

+ $database_connection_charset = 'utf8';

+ $database_dsn = 'mysql:host=...;dbname=...;charset=utf8';

And you may start testing it it works :)

In my setup, its somehow in njis charset... (im not sure why)