Quantcast
Channel: MySQL Forums - Character Sets, Collation, Unicode
Viewing all 294 articles
Browse latest View live

can I use latin1 and utf8 at the same time? (no replies)

$
0
0
hi
Currenty I've got latin1, but I'd like to set utf8 as default charset for all levels (server,db,table,session,client).
Please, let me know if having set this in my.ini in client and server sections (of course it applies only for new databases and tables) I can still use my old databases and tables in latin1 fashion.
Or, will it collide when quering the data. Then should I unify?
As for backuping, I'm performing offline backups (not mysqldump), so restoring shouldn't be an issue.
I wouldn't like to do it as I have a TB db and the convertion will be troublesome.

it seems like each schema is independent, having own db.opt setting's file.

so now the only thing I try to figure out id what the application/client should be aware of.

Can you see here any risks to jumble the database?

Use of [] square-brackets to enclose table/column names (1 reply)

$
0
0
I am trying to connect into a MySQL schema via an existing application. That application allows you to specify a connection string which I was able to implment ... the connection now works.

Unfortunately, the application includes the use of [] (square brackets) to enclose table and column names when interacting with the MySQL tables ... it's hard-coded into the application so, I can't change that.

As you know, MySQL can be configured to use `, " or '.

Does anyone know of a MySQL configuration parameter that would allow me to specify that this MySQL schema should use [] instead? In all the reading I've done so far, I fear the answer is no.

Thank you for anything you can offer.

Doug

Special Characters (äöüß), switching to UTF8, collation-problems (1 reply)

$
0
0
Hi all,

we have a larger DBMS (not an oracle-product) that is able to export data into a mySQL system. Or more precisely we are currently at a stage where the mySQL export is about to turn productive.
With more test-data available, we are encountering more problems. One of the larger problems we've run into recently was the handling of special characters, namely german "umlauts", which are äöü and ß. Others might be of importance and not yet discovered.
It turned out that somewhere between out application and mySQL server, the statement was converted into UTF8 (supposedly dbx-driver-component), and we haven't been able to change this despite trying a myriad of different settings as well programmatically (e.g. Connection.SetParameter("CharSet", "..."); )
as in mySQL table-wise or DB-wise (alter table/db, set charset= collate=).

So we finally resorted to converting all db-tables to UTF8. The transfer of special-characters now works. However all collations for UTF8 are not case-sensitive except for the bin-collation. The bin-collation however is not usable, since we need to sort data using "ORDER BY" in our selects instead of internal sorting algorithms local to our applications.
However, our source-DBMS is case-sensitive and we do have unique indices on many of our varchar-columns, which do not translate into this mySQL-configuration.

Finally, the question is: how can we handle case-sensitive mySQL with special characters as äöüß ?

Thanks a lot for any input!
Regards,
P.

accented letters, display and search (no replies)

$
0
0
Hi,
as the title, I have trouble with these blessed accented letters.
I know that the 'subject was covered billions of times, but just can not seem to solve.
Mysql version 5.5.29 64bit.
I have the following problem:
I create a database starting with character encoding to utf8 with the command

"Crate the official database;"

I create a table (up to some versions ago), the column names I could put an accent not anymore (first issue), but this matters little, then run

"Create table customer (id int not null auto_increment, City varchar (255) not null, primary key (id));"

I'm going to create my table insert (from ms-dos prompt), created in the column, the character "ò" through

"Insert into customers (City) VALUES ('o');"

and here comes the first problem, the terminal answers me

"Mysql> insert into customers (City) values ​​(" ò ");
ERROR 1366 (HY000): Incorrect string value: '\ x95' for column 'City' at row 1
mysql> "

then I force the connection to the database with the command

"Set names latin1"

Rieseguo the 'insert, is magically takes me an accented letter.
Now, to business needs, the database will be 'backupped' (pass the word) 1 time per day.
Then running back, I like to check the correct data import, what 'now there is a need, then I get disconnected mysql, and run the command

"Mysqldump-u user-p official> backup.sql"

OnRPG create a database to test the 'import

"I try create database;"

at this point the backup through reimporto

"Mysqldump-u user-p I try <backup.sql"

Result I find a table with columns 'id and Citta' but completely empty.

I tried one of the many attempts, forcing the 'exports in latin1, in that case I inserted the data into the database, but when executing an

"Select * from clients where City LIKE '%ò%';"

Comes back to me empty, but if I try '%o%', I found the field. But ò stressed, had a strange character, but by the carattare phpmyadmin I saw him right!
Does anyone know how to solve this problem permanently boring?

Thanks for any replies

Alessandro

Is STR_TO_DATE function doesn't work with UCS2 charset? (no replies)

$
0
0
Dear All,

Pleaes help me with this quick query.

[Server version: 5.5.29-log MySQL Community Server (GPL)]
I am working with a table which is having default charset ucs2. One of the coloumn is varchar(64) contains datetime information.

I need to convert that varchar column to datetime in order to do some business logic. I used STR_TO_DATE function, It didn't work.
I got a NULL, with a warning code 1411 and message - Incorrect datetime value: '' for function str_to_date.

I did a quick experiment, changed the column charset to latin1. It's working.

My query is, does the function STR_TO_DATE is incompatible with ucs2 charset?
Is there any documentation available which mention about this incompatibility with certian set of charsets?(I serched this entire week and didn't find one, help me if you find one)

For some function like char_length() are mentioned as multi-byte safe.
Therefore, do I need to consider, the functions which are not mentioned the same way are incompatible with multi-byte?

Kindly help.

Characters not getting stored (3 replies)

$
0
0
Hello, I have yet another problem with character sets. I have read lots of threads here and the very informative page at http://mysql.rjweb.org/doc.php/charcoll, but I found no answer.

Here is the issue: I'm designing a new database from scratch on MySQL version 5.5.27-log and I would like to use utf8 with collation utf8_unicode_ci throughout, because it will have to store names in German, Italian and English.

I defined the database to use DEFAULT CHARACTER SET = utf8 COLLATE utf8_unicode_ci, I declared each table with DEFAULT CHARACTER SET = utf8 and I even repeated the same thing in most columns of the tables.

The only two tables I double checked with SHOW CREATE TABLE tbl show everything as utf8 with collate utf8_unicode_ci. The others don't matter because I have the problem already with one of these.

I tried different things and basically tried to store 'Forlì' and (just to mess with it) 'Förlìßen'.

With the first one I get a warning "Incorrect string value '\x8D' for column 'birthplace' at row 1" and when I select it, it shows just 'Forl' without the final 'ì'. The second one too gets truncated at the first "invalid" character that it encounters: ö, with an "Incorrect string value" error message. That means it stores only 'F'.

The test with 'birthplace', hex(brithplace), length(birthplace) and char_length(birthplace) shows the following:

for 'Forlì': 'Forl', 466F726C, 4, 4.
for Förlìßen: 'F', 46, 1, 1.

All of this happens when I set the "character_set_*" variables to utf8 (other than the filesystem, which I didn't touch).

With other combinations of "character_set_*" I do get different symbols on the screen and different hex numbers in the table, but I have not got the right one yet: if I set names to latin1 or cp850 (which appears to be the default in my installation, the initial set-up I get when starting the console is that) and leave character_set_database at utf8, then it does not truncate the strings at the first non-english character, but the results are only shown correct if I enter the data from console.

I tried importing the data from a file, saved both as latin1 and as utf8, and it does store something, but not the right thing, regardles of the variables setting.

I dropped the database and recreated it several times with different settings of the variables at the beginning, but I don't get it right.

In all these situations I'm entering the letters from a german keyboard under Windows 7, if that can make any difference.

I'm completely confused: it seems it only works if I enter the data from the keyboard, and this again if I don't set names to utf8 (which I thought would actually be the right thing to do). Also I would like to import them from a file because sometimes I don't have access to the server and it would be more convenient for me to prepare a file and then import it all at once.

Any help would be appreciated.

Thank you.

mysql_query and embarcadero c++ (1 reply)

$
0
0
Hello,
I use MySQL C API with Embacadero c++ without any problems.
But when I try to send any wide character (UNICODE) like Arabic symbols, appears as not readable characters in MySQL database. because the SQL string that I send it by mysql_query function must be (const char *) and not (const wchar_t *).

thanks for any helps.

mysql can not to insert unicode16 uncommon word (5 replies)

$
0
0
connect with ODBC “DRIVER={MySQL ODBC 5.2w Driver};SERVER=%s;PORT=%d;DATABASE=%s;UID=%s;PASSWORD=%s;OPTION=3;”
szSQL.Format(_T("INSERT INTO test")
_T("(Name)")
_T("VALUES('%s')"), _T("𢈢"));
m_Conn.pConn->Execute((_bstr_t)szSQL, &RecordsAffected, adCmdText);


The program does not return,This problem has troubled me for a long time,Who can help me answer this question thanks

utf8 column is storing only half of the maximum number of characters (1 reply)

$
0
0
Hello,

I am using MySQL 5.0.18-max. I have a column of type varchar(4000) and character encoding utf8. The JDBC URL includes useUnicode=true and characterEncoding=UTF-8. I discovered that only the first 2000 characters (i.e. one-half the character limit) are being stored. All of the characters are ASCII (in the row where the problem was discovered), so I'm guessing that MySQL allocated 2 bytes/character when updating or inserting the text but assumed 1 byte/character when looking for the cutoff point.

I don't want to change the character encoding to latin1 because other rows in this column might contain non-ASCII characters (such as left- and right-quotes) that were copied and pasted from MS Word.

I suppose I could simply double the maximum number of characters, but I would like to know what is causing this problem.

This problem is hurting productivity for an entire department within my company, so a prompt reply would be greatly appreciated:)

Regards,
Michael Rivard

Can't identify character set and collation used in string comparisons (no replies)

$
0
0
Hi,

I have a table with a string column and try to determine what character set and collation is being used for the comparisons when searching for values.

Charset for current schema, table and column: latin1
Collation for the specified column: latin1_swedish_ci
Charset for connection and rest: utf-8

Suppose you have done the following:
CREATE TABLE `collation_test` (
`Name` varchar(1) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

INSERT INTO `collation_test` VALUES ('u'),('ü'),('e'),('é'),('ö'),('ø');

Now...

SELECT `Name`
FROM collation_test
WHERE `Name` = 'e';
=> 2 rows (e and é)

SELECT `Name`
FROM collation_test
WHERE `Name` = 'u';
=> 1 row (u)

SELECT `Name`
FROM collation_test
WHERE `Name` = 'ø';
=> 1 row (ø)

SELECT STRCMP('é', 'e');
=> 0 (they are considered equal when using the default charset and collation)

SELECT STRCMP(_latin1 'é' COLLATE latin1_swedish_ci, _latin1 'e' COLLATE latin1_swedish_ci);
=> -1 (they're NOT equal when specifying latin1)

It first seemed to me that utf-8 was used with some unknown collation, but after checking this site:
http://mysql.rjweb.org/doc.php/charcoll
then I can't find anything matching these tests.

What am I doing wrong here? Any help appreciated.

David

data type and Character set for DUTCH language (3 replies)

$
0
0
Dear All,

I need to store characters "Pré intake" into database. Please let me know what data type and collate should I use.

I would appreciate early help.

solution for insert certain utf-8 data into Mysql (1 reply)

$
0
0
any one want inser utf8 data in mysql teble and also want to see data in your formet u just do one thing

change charset & collate by below formet

CHARSET=utf8
COLLATE=utf8_unicode_ci

after this u will show your data in utf encoding in mysql database tables and your browser also

UTF8, latin1 and performance impact (1 reply)

$
0
0
Hi MySQL people,

Here's the story. I have 2 tables, in latin1, and a copy of these in UTF8 (since data itself is 100% English language strings for now).
Let's name the tables table_a_latin1, table_b_latin1, table_a_utf8, table_b_utf8 (utf8_unicode_ci).
These tables use MyISAM, they have about 6M row each, rows are rather long (as in 30 attributes, all strings for table_a, and 20 for table_b). Both have their PRIMARY INDEX on a int 'id', and a UNIQUE index on the first column after that: 'name' VARCHAR(10).
I have also checked table sizes:
table_a: 857M
table_a indexes: 148M for latin1 and 115M for UTF8 (how is this possible?)
table_b: 1.4G
table_b indexes: 356M for latin1 and 304M for UTF8 (...?)

I am comparing time spent on INNER JOIN.
Everything looking equal to me, I want to compare time spent on a query like:
SELECT * FROM 'table_a' tablea INNER JOIN 'table_b' tableb ON tablea.'name' = tableb.'name';

When joining the 2 latin1 tables, query is like 3 times faster than the join on their utf8 equivalent.

For instance, a SELECT COUNT(*) on the latin1 tables takes 35s, and 1min50s on the UTF8 tables.

When using EXPLAIN, the index is correctly identified, difference being that key_len is 12 for latin1 and 32 for UTF8.

I'd like to stress that the data is from English language only (at least for now). If I got it right, UTF8 doesn't use more than the one byte to encode a latin1 character. So a comparison between 2 utf8 strings should not take much longer than the one on latin1, or should it?

Considering the impact on performance (it's a somewhat frequent join), it's really important to me to understand what is happening, and if I'm doing something wrong.
Let me know if some info is missing.

Clean up Character Set mess in the same field of one table (1 reply)

$
0
0
Dear MySQL experts,

I have a problem with a table that contains strings using different character sets (latin1 and utf8) in the same field.

The cause was probably a switch of the field's collation at some point and/or a change in the application that saved the strings into the field.

I would like to change the strings in to one common character set.

At the moment I can get the correct values for the two types of entries as follows:


For "group 1":

SET NAMES utf8;

SELECT id, groupname, stringfield FROM tablename;

=> 1, "group 1", "text with correct ä,ö,ü, ..."
=> 2, "group 2", "text with incorrect ä, ö ..."

------------------

For "group 1":

SET NAMES latin1;

SELECT id, groupname, stringfield FROM tablename;

=> 1, "group 1", "text with correct ä,ö,ü,..."
=> 2, "group 2", "text with incorrect � � �"


I have tried different things, like using CONVERT(stringfield USING utf8/latin1), CAST(stringfield AS CHAR(2000) CHARACTER SET utf8/latin1 but to no avail.


I would like to convert either the utf8 strings into latin1 or vice versa and save the compatible strings in a new field.

Any ideas on how to do that?

Thank you very much in advance for any help!

Tobias

LOWER() influences utf8_bin collation? (3 replies)

$
0
0
I've encountered a strange behaviour of LOWER() function, which in some way influences sorting behaviour. The database and table are all UTF8, with collation set to utf8_bin. Mysql 5.6.14 on FreeBSD.

Example code:

SET NAMES utf8 COLLATE utf8_bin;
CREATE TABLE t (a varchar(4)) DEFAULT CHARACTER SET 'UTF8' COLLATE 'utf8_bin';
INSERT INTO t VALUES ('zzz');
INSERT INTO t VALUES ('tt');
INSERT INTO t VALUES ('m');
INSERT INTO t VALUES ('kkk');
INSERT INTO t VALUES ('bb');
INSERT INTO t VALUES ('a');

mysql> SELECT a FROM t ORDER BY a;
+------+
| a    |
+------+
| a    |
| bb   |
| kkk  |
| m    |
| tt   |
| zzz  |
+------+

mysql> SELECT a FROM t ORDER BY LOWER(a);
+------+
| a    |
+------+
| m    |
| a    |
| bb   |
| kkk  |
| tt   |
| zzz  |
+------+
6 rows in set (0.00 sec)

While both SELECTs should return exactly same values, they do not. LOWERed results have single-letter values ordered first, I have no idea why. Can somebody shed some light?

PS. I know I can use utf8_general_ci for case-insensitive ordering. I'm not asking for a walkaround solution ;)

Insert Chinese Special Character gibberish ??? (2 replies)

$
0
0
Hello, Everybody. I Want to ask you a question? thanks.

We use Windows clicent , character set is GBK.
Server is Linux, MySQL 5.6.17 community edition, database character set is utf8.

SHOW VARIABLES LIKE 'character%'

Variable_name Value
------------------------------------------
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_filesystem binary
character_set_results utf8
character_set_server utf8
character_set_system utf8
--------------------------------------------

When insert Standard Chinese character, data show normal.
e.g.
create table testgbk(nm varchar(20) charset utf8);
insert into testgbk values('测试'),('我们’);
select * from testgbk;

result:
测试
我们

When insert Special Chinese character:  , ,  ,, data show gibberish.
insert into testgbk values(''),('');

result:
?
?

Italian locale on format() (1 reply)

$
0
0
Hi folks!
Is possible that mysql have some racist behavior against Italians, or a positive one for Deutschland?

Whereas format(float(9,2), 2, 'de_DE') return the correct number where ',' is the decimal separator and '.' is group separator, format(float(9,2), 2, 'it_IT') return the number without grouping the thousands.

- format(123456.56, 2, 'de_DE') -> 123.456,56
- format(123456.56, 2, 'it_IT') -> 123456,56

Is that a problem of mysql or my system locale? I have no clue.

mysql Ver 14.14 Distrib 5.5.38, for debian-linux-gnu (x86_64)

incorrect string error when i run mysql for a particular column please help me (1 reply)

$
0
0
whenever i load in a particular table it loads the data but some records are not been loaded

instead it shows an error called Incorrect String value

for a particular column[Original_description]

please help me what i can i do

Can someone please explain this misbehavior? (3 replies)

$
0
0
SET `collation_connection` = 'utf8_bin';
SET @wrong = CONVERT(CAST(0xD187D0B5D180D0BDD18BD0B9 AS CHAR) USING 'cp1251') COLLATE 'cp1251_general_ci';
SET @right = CONVERT(CAST(0xD187D191D180D0BDD18BD0B9 AS CHAR) USING 'cp1251') COLLATE 'cp1251_general_ci';
SELECT @wrong AS `wrong`, @right AS `right`,
IF(@wrong = @right, 'Bogus comparison', 'Right is right') AS `equality`;


versus

SET `collation_connection` = 'utf8_bin';
SET @wrong = CONVERT(CAST(0xD187D0B5D180D0BDD18BD0B9 AS CHAR) USING 'utf8') COLLATE 'utf8_general_ci';
SET @right = CONVERT(CAST(0xD187D191D180D0BDD18BD0B9 AS CHAR) USING 'utf8') COLLATE 'utf8_general_ci';
SELECT @wrong AS `wrong`, @right AS `right`,
IF(@wrong = @right, 'Bogus comparison', 'Right is right') AS `equality`,
@@collation_connection AS `collation`;

SET `collation_connection` = 'utf8_bin';
SET @wrong = CONVERT(CAST(0xD187D0B5D180D0BDD18BD0B9 AS CHAR) USING 'utf8') COLLATE 'utf8_unicode_ci';
SET @right = CONVERT(CAST(0xD187D191D180D0BDD18BD0B9 AS CHAR) USING 'utf8') COLLATE 'utf8_unicode_ci';
SELECT @wrong AS `wrong`, @right AS `right`,
IF(@wrong = @right, 'Bogus comparison', 'Right is right') AS `equality`,
@@collation_connection AS `collation`;

Why is latin1 used? (2 replies)

$
0
0
Hello,

i've got some Problems with an update to mysql 5.6 from 5.5.
All server variables are set to utf8. All columns are collation of utf8_german2_ci or utf8_unicode_ci.

If i make an select statement like "select COLUMN from TABLE where Column > 0" it works. When i add collate utf8_unicode_ci at the end of the statement it produces this error:
#1253 - COLLATION 'utf8_unicode_ci' is not valid for CHARACTER SET 'latin1'
Selected Column is an Integer used for a foreign key.

i have to use COLLATE in it, because in an UNION of two Select statements the Database shows this Error: #1271 - Illegal mix of collations for operation 'UNION'


But all is utf8 (utf8_unicode_ci & utf8_german2_ci), where does latin1 come from?

Much thanks and best regards

Fabian
Viewing all 294 articles
Browse latest View live


Latest Images

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>