I’ve recently had to move a few databases from MySQL 4.x to MySQL 5.x. One of the most important differences is that the 5.x family understands character encodings. Not exactly fresh news, version 5.0 was issued in 2003, but there is still a lot of 4.x installations around.

MySQL 5.0 no longer happily accepts any byte string into a VARCHAR or TEXT field. It stores encoding names as part of the table structure, and converts between encodings when necessary. MediaWiki or WordPress, when run on MySQL 4.x, store data in UTF-8, but the database itself doesn’t “know” about it. Everything seems fine, until you dump your database to a file and load it into MySQL 5.0 (or above). What happens, is that your text is considered to be latin1 (a.k.a. ISO-8859-1). If you happen to have any non-English characters as, say, article names in MediaWiki, you’re going to end up with an error message such as:

A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:

(SQL query hidden)

from within function “Article::pageData”. MySQL returned error “1267: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation ‘=’ (localhost)”.

To fix the problem, you need to tell MySQL that your text data is really UTF-8, not latin1. You need to find all the columns of type VARCHAR or TEXT, and modify them to have UTF-8 character set. For example, if your column is VARCHAR(255), you can execute this statement:

ALTER TABLE your_table
MODIFY COLUMN your_column VARCHAR(255)
CHARACTER SET utf8
COLLATE utf8_bin;

(The utf8_bin collation is needed to keep your sorting case-sensitive.)

However, MySQL will convert your text from latin1 to UTF-8, and your text will still appear “wrong”. You can fix it in one more step. The problem is that you had UTF-8 taken to be latin1 and then stored as UTF-8. To fix this, you need to “convert” your text from UTF-8 back to latin1, and then make MySQL take it as UTF-8, but, importantly, without converting it. This can be achieved by temporarily casting your data to binary — this operation doesn’t trigger encoding changes. You can then cast your data into any encoding you want. In a nutshell, you need to go: UTF-8 →(conversion)→ latin1 → binary → UTF-8.

UPDATE your_table
SET your_column = CONVERT(
CONVERT(
CONVERT(
your_column
USING latin1
)
USING binary
)
USING utf8
);

It may take you a while to understand it. If you want to get a better feel of what’s going on, consider the following, equivalent example in shell. Let’s assume you’re using Polish diacritics: ąćęłńóśżź. Your Polish page title “Café” might in a garbled form look something like this:

Café

Assuming your system is natively UTF-8 (most of modern Linux distributions are), an easy way to simulate text garbling is the following shell expression.

echo Café \
| iconv -f utf-8 -t utf-8 \
| iconv -f latin1 -t utf-8
Café

Converting from UTF-8 to UTF-8 seems stupid, but I wanted it to be very clear: what we have here, is an UTF-8 string, as output by the second line, taken to be latin1 in the third line. This is how your text can become garbled. An obvious way to fix it, is to run the process backwards:

echo Café \
| iconv -f utf-8 -t latin1 \
| iconv -f utf-8 -t utf-8
Café

Again, the UTF-8 to UTF-8 conversion is preserved to make this crucial point explicit. Your string was converted to latin1, and then taken to be UTF-8.

Back to our problem. We know how to convert the data, but we need to find all the tables and columns taht need converting. Conveniently, MySQL offers an “information_schema” database, which allows us to read information about MySQL tables. It’s enough to run this query to find all the tables of interest:

SELECT
table_name,
column_name,
column_type,
character_set_name
FROM
columns
WHERE
table_schema = ‘your_table’
AND
(
data_type = ‘varchar’
OR
data_type = ‘text’
)
AND
character_set_name != ‘utf8′
;

If you’re really lazy, as I am, your data has exactly this problem (UTF-8 taken to be latni1 and looking garbled), you can use a Python script I wrote. But please be careful! You’re using it at your own risk! Backup your database first! If this script damages your database and you lose all your data, it’s your problem, not mine. You have been warned. Here’s the script.

Note: the script will only print SQL statements to screen. You have to execute them youself.

People get usually famous for the things they’ve done. Well, that’s not entirely true. They usually get famous for the things they’ve done, when they were successful. You don’t get famous for attempting and being unsuccessful, now do you?

It works the same way for the scientific publications. All scientists work hard trying various things, and when they finally succeed, they publish a paper. But what happens with all those hours spend on unsuccessful attempts? Nobody seems to be proud of blowing a whole laboratory up. Or whatever didn’t work for them. This means that other people can never learn that something was unsuccessful and they’re likely to get the same, unfeasible, idea and repeat the same research. Needless to say, unsuccessfully.

Not that I’m proud of what I’ve done here, but I will at least allow other people to find this post on Google, when searching for genetic data and relational database. I’ll describe what I did, so they at least don’t do it the way I did.

There is a project called Hapmap, which is essentially a publicly available genetic data set. It is now being used for various genetic studies, usually for finding associations between specific genes and diseases. That’s what the genetic epidemiology is all about, I think. To find the guilty gene. I wonder when will they start fixing people with broken genes. It’s a only matter of time. Anyway, Hapmap allows you to download a whole bulk of genetic data and torture them in any way you want, just listening to the bits squeaking in pain. That’s a really great thing.

For my project, there was a problem of reformatting the data, so they could be fed to the program that analyzes them. Sounds easy, but writing a C++ code to do that is neither quick nor nice. And it works quite slow. In fact, I’m just watching one of my Core 2 Duo cores working at 100% for… how long?… 103 minutes so far, working with one chromosome. So I got this idea to put all the Hapmap data into an SQL database so I could throw declarative statements at it, getting whatever I asked for.

Genetic data as in Hapmap phase II, is generally a matrix with 210 rows (individuals) and 3.3 million columns (loci). Relational database is all about tables, but you can’t create a table with 3.3 million columns. PostgreSQL has a limitation to 65 thousands or so. I decided to try a sparse matrix implementation with one table for individuals, one for loci and one connecting table with two foreign keys and two fields for allele, defining a genotype. I was expecting it to be somewhat demanding for a machine, but having a Core 2 Duo with 2GB RAM and 70GB hard disk, how hard can it be? One row of the connecting table consists of 4 + 8 + 1 + 1 bytes, giving 14 bytes in total. 3.3 million times 210 would give about 700 million rows. Not every locus is polymorphic in every population, so in practice there were about 500 million rows in that table. 500 million times 14 bytes gives about 7GB. Piece of cake.

CREATE TABLE individual (indiv_no SERIAL, indiv_id VARCHAR);

CREATE TABLE locus (locus_no BIGSERIAL, snp_id VARCHAR, position INT8);

CREATE TABLE observed_genotype(indiv_no INTEGER, locus_no INT8, a1 CHAR, a2 CHAR);

Those SQL statements are simplified for better readability, in reality they had more stuff in them, like primary and foreign key declarations.

I sat down and hacked few Python scripts to prepare the data and load them into PostgreSQL. At first, my computer almost melted, then I found some more efficient ways to handle the data and finally got them all loaded into the database. The database had 38GB.

That is pretty heavy. It’s about 5 times more than the actual data takes. Not mentioning that using continuous bit arrays would allow to store the whole thing in about 250MB. We’re in relational world, you know. Things just need to take space.

There was only 4GB disk space left. The data was loaded, but the connecting table didn’t have the primary key. It had to be dropped, otherwise it would’ve taken too long to load the data. I happily launched phppgadmin and made few clicks to create a primary key. I’m getting old, you know. I know I could’ve typed it. But regardless to whether I’d type or click it in, the PostgreSQL took the remaining 4GB to create the index, but it wasn’t enough so it rolled back and returned the space to the file system.

That was basically it. I’ve had enough, dropped the whole database and started hacking with text files and coreutils. If you had an idea to put whole Hapmap data set into PostgreSQL, think if you have enough resources. Maybe, if you’re in 2012, sitting in front of a 16GHz eight-core machine with 32GB RAM and 1TB hard disk you do. But I’m in 2007 and I don’t. So, here is my opposite to “success story”. I hope you appreciate that. And no, I’m not very proud of it; but I value other people’s time and I think I might help someone before they spend a week doing the same thing.

Nevertheless, it could be feasible to load Hapmap genetic data into some specifically optimized relational database. Perhaps Oracle guys have something that could handle it. It’s still very tempting to be able to manipulate genetic data in a high level, declarative manner.

I have published my MSc thesis on-line. It’s available for (free) download in PDF format. It contains:

1. An example of complicated data reshaped to a form which allows statistical analysis
2. A method of comparing hospitals fairly

Read more and download the thesis.

Tetralogy of Fallot is a significant and complex congenital heart disease. It consists of four heart malformations. So if a patient is described to have TOF, it means that she/he has all those four malformations together.

However, the separate malformations are already present on the diagnoses list, as separate entities. From a data-modeling perspective, it’s a redundancy on the factors (malformations, diseases) list. This leads to problems with interpretation. As the VSD is one of TOF’s components, is already present on the diseases list, and users are allowed to enter both VSD and TOF diagnoses, there are patients with all four combinations in the database.

People might write it like this:

• TOF
• VSD
• TOF and VSD
• None

But I would prefer the tabular notation:

+-----+-----+
| TOF | VSD |
+-----+-----+
|  -  |  -  |
|  X  |  -  |
|  -  |  X  |
|  X  |  X  |
+-----+-----+

It should be clear now, why there are exactly four combinations. And all four are present in the database. The TOF and VSD combination is invalid, because VSD is already present in TOF. Well, it’s doctors who know that, because the data structure doesn’t reflect this knowledge.

Now, how to fix it? Should the data entry software prohibit entering both TOF and VSD? No. It would mean implementing an exceptional behaviour of the application, and leave the weak data structure. Instead, I would make an alternative data structure:

+-----+--------------------+------------------+----------------------------+
| VSD | Pulmonary stenosis | Overriding aorta | Right ventric. hypertrophy |
+-----+--------------------+------------------+----------------------------+
|  -  | -                  | -                | -                          |
|  X  | -                  | -                | -                          |
|  X  | X                  | X                | X                          |
|  X  | X                  | X                | X                          |
+-----+--------------------+------------------+----------------------------+

It doesn’t mean that the data entry software users would have to enter every component separatery. For such complex diseases a presets would be helpful: click the TOF button, the four factors jump in. But if you have an exceptional patient, you can tweak every single factor independently.

The reports

When viewing information about VSD and related diseases, surgeon doesn’t want to see, what percent of patients with VSD also have TOF. Also, when viewing TOF-related diseases, the surgeon doesn’t want to see VSD, because all patients with TOF have VSD by definition.

New relation

So I have to add new data structures. There will be new relation between the diseases:

• Is part of

The results in the reports will be:

• When viewing the superior disease (here: TOF), patients who have the inferior disease (here: VSD) will be shown, but the information about the inferior disease will be discarded, so they will appear as they wouldn’t have it.
• When viewing the inferior disease (here: VSD), the patients with the superior disease (here: TOF) as well will not be included and/or shown.

The puzzle

Malcolm has asked, how to find the classes that were attended by all of the students from a given list. Then, he proposed a solution with a HAVING clause. I’ll call it the one-join solution. I’d like to suggest another one, which I’ll call multi-join.

I’ve made a benchmark to evaluate the execution time. A statistical tool was used to create a mathematical model of the execution time.

The database structure

CREATE TABLE Class (
    id  integer NOT NULL PRIMARY KEY);
CREATE TABLE Student (
    id integer NOT NULL PRIMARY KEY);

CREATE TABLE Reln_Class_Student (
    class_id integer NOT NULL REFERENCES Class(id),
    student_id integer NOT NULL REFERENCES Student(id),
    PRIMARY KEY (class_id, student_id));

My solution

Like in Malcolm’s example, following example finds the classes that are attended by both 253 and 289 students.

SELECT id
FROM
    Class AS C
    INNER JOIN Reln_Class_Student AS s253
        ON (C.id = s253.class_id)
    INNER JOIN Reln_Class_Student AS s289
        ON (C.id = s289.class_id)
WHERE
    s253.student_id = 253
    AND
    s289.student_id = 289
;

In my example, the number of students on the list (here: two) is equal to number of joins that are to be performed. Each join must have different alias assigned, hence “s253″ and “s289″ aliases for joined tables.

Without indexes on the connecting table

Malcolm’s solution found the class with list of 10 students in 214ms, while my query did the same job in 5.8ms, 36 times faster. With a short (2-element) student list, Malcolm’s query takes 130ms, while my query completes in 4.9ms, 26 times faster.

I analyzed the planner (EXPLAIN ANALYZE) output. Despite my query uses multiple joins, it always uses index scans, while with Malcolm’s query, planner uses sequential scans (over the long Reln_Class_Student table), which effectively kill the performance. It’s faster to perform 10 index scans than just one sequential one.

The benchmark

The environment

The test was done on a Celeron M 1.5GHz with 768MB of RAM, on PostgreSQL 8.1, under Ubuntu Linux 6.06 distribution.

The data

Let’s change the nomenclature to a more generic example: documents and tags.

• 2000 documents
• 100 tags

Each tag had a number and assigned frequency. tag 1 had 0% frequency, tag 2 had 1% frequency, and so on. Tags were distributed independently from each other.

The tests

Queries with two keywords were tested. All possible combinations of the tags were tested. In total, 20 thousands observations were recorded (100 × 100 × 2). Each observation had 4 properties:

• Query type (one-join or multi-join)
• Tag 1 id (~frequency, from 0 to 99)
• Tag 2 id (~frequency, from 0 to 99)
• Execution time (in ms)

The analysis

All the observations were imported into the R-project statistical package. A linear regression were used to create a mathematical model for the execution time. The following model was found:

Call:
lm(formula = time ~ tag1 * tag2 * qrytype, data = B)Residuals:

Min       1Q   Median       3Q      Max
-36.9016  -5.8601  -2.8720   0.8972 794.9966
Coefficients:
                             Estimate Std. Error t value Pr(>|t|)

(Intercept)               27.0779277  0.8396820  32.248  < 2e-16 ***
tag1                      -0.0626129  0.0144355  -4.337 1.45e-05 ***
tag2                       0.2447616  0.0144355  16.956  < 2e-16 ***
qrytypeone-join           -2.0632741  1.1874897  -1.738   0.0823 .
tag1:tag2                  0.0013108  0.0002482   5.282 1.29e-07 ***
tag1:qrytypeone-join       0.1914354  0.0204149   9.377  < 2e-16 ***
tag2:qrytypeone-join       0.1304854  0.0204149   6.392 1.68e-10 ***
tag1:tag2:qrytypeone-join -0.0015868  0.0003510  -4.521 6.18e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 20.68 on 19992 degrees of freedom
Multiple R-Squared: 0.2284,     Adjusted R-squared: 0.2281
F-statistic: 845.5 on 7 and 19992 DF,  p-value: < 2.2e-16

The three greatest coefficients are:

• tag2
  the more popular second tag, the longer execution time
• tag1:qrytypeone-join
  when one-join query is used, additional time on tag1 is used
• tag2:qrytypeone-join
  when one-join query is used, additional time on tag2 is used

The qrytypeone-join doesn’t show statistical significance. It’s probably the effect of some queries with one-join query that executed very fast. They are visible on the plot.

The plot

Click the thumbnail to see the large version.

• One-join queries: Red
• Multi-join queries: Blue

The plot shows timings of the queries. The lower, the better. As you can see, the tag-frequency to time ratio relation is linear, but the slope is steeper for multi-join queries. You can see the same thing in the model as the tagX:qrytypeone-join coefficients.

Conclusion

Multi-join query is always at least as fast as one-join. Both queries perform essentially similar if both tags are of low frequency. The more frequent tags, the bigger difference between queries, in favor for multi-join query.