Troubleshooting TokuDB Corruption

I recently ran across some TokuDB table corruption, which was not easily identifiable at first, and the error log entry was not too verbose either, so I wanted to share that experience here.

Basically, TokuDB crashed, and then mysqld had problems restarting afterward. Just for reference, the error log had the following in the stack trace:

/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c48)[0x7fb25be75c48]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71cbd)[0x7fb25be75cbd]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z29toku_deserialize_bp_from_disk...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z23toku_ftnode_pf_callback...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z30toku_cachetable_pf_pinned_pair..
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z24toku_ft_flush_some_child...
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z28toku_ftnode_cleaner_callback...
/usr/lib64/mysql/plugin/ha_tokudb.so(_ZN7cleaner11run_cleaner...
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xcfeee)[0x7fb25bed3eee]
/lib64/libpthread.so.0(+0x79d1)[0x7fb39310b9d1]
/lib64/libc.so.6(clone+0x6d)[0x7fb391828b5d]

The crash was not really the problem here; that is another story.

But rather mysqld not starting back up was an issue.

Given this crash, and mysqld not starting back up, we needed to check every fractal tree file in the data directory.

I’m not an expert with this, but what I did, and you may find helpful too, is to iteratively check each fractal tree file with “tokuftdump –nodata tokutablename”.

If it does not return anything, then there is a problem with the table – not too scientific 😉 – but was effective in this case.

If you have just a couple files to check, this is easy enough to run per file. If there are a lot to check, then a simple shell loop with the check (and output the name at the time) can help.

Any corrupt keys should be then dropped and recreated.

That should get you back going.

On a side note, when looking more into “tokuftdump –nodata”, I did run across these 2 commands (one-liners), which allow you to check the compression format of the tables also using tokuftdump –nodata. Thus, it illustrates the use of this command for another feature, and it’s easily modifiable if you’re interested:

Pre-MariaDB 5.5.37:

mysql -sNe 'SELECT dictionary_name, internal_file_name FROM 
information_schema.tokudb_file_map WHERE dictionary = "main" 
OR dictionary LIKE "key-%"' 
|perl -F'\t' -ane '@out=qx(tokuftdump --nodata --rootnode /var/lib/mysql/$F[1]); 
foreach $ftdump (@out) { if($ftdump=~/^ compression_method=([0-9]+)$/) 
{ print $F[0]."\t".$1."\n"; last } }'

MariaDB 5.5.37+ (b/c “dictionary” column changed to “table_dictionary_name”):

mysql -sNe 'SELECT dictionary_name, internal_file_name FROM 
information_schema.tokudb_file_map WHERE table_dictionary_name = "main" 
OR table_dictionary_name LIKE "key-%"' 
|perl -F'\t' -ane '@out=qx(tokuftdump --nodata --rootnode /var/lib/mysql/$F[1]); 
foreach $ftdump (@out) { if($ftdump=~/^ compression_method=([0-9]+)$/) 
{ print $F[0]."\t".$1."\n"; last } }'

For reference, these are documented here.

Hope this helps.

 

Troubleshooting TokuDB ERROR 1126 – API Version Mismatch or bitmap_free

When setting up TokuDB, you may encounter error 1126.

I’ve seen 2 recent invocations of it, so I wanted to share them both here in case you run into this issue:

MariaDB [(none)]> install soname 'ha_tokudb';
ERROR 1126 (HY000): Can't open shared library 
'/usr/lib/mysql/plugin/ha_tokudb.so' (errno: 2, undefined symbol: bitmap_free)
MariaDB [(none)]> install soname 'ha_tokudb';
ERROR 1126 (HY000): Can't open shared library 'ha_tokudb.so' 
(errno: 8, API version for STORAGE ENGINE plugin TokuDB not 
supported by this version of the server)

The latter is a bit more descriptive, but the former is fairly cryptic.

Given the latter, as you may have guessed it, if you run into either of these, you have the wrong version of ha_tokudb.so in your plugin directory – that is, it is not the correct version for your MariaDB/MySQL.

In both cases, 10.0.11 was used. In the first case, the ha_tokudb.so was from 5.5.37. In the second case, the ha_tokudb.so was from 10.0.12.

Hope this helps.

 

Disabling Transparent Hugepages for TokuDB

If you want to use TokuDB with MariaDB, MySQL, or Percona Server, you will need to disable support for transparent hugepages in Linux.

Fortunately, this is very easy to check, and to change.

An easy way to check is with:

cat /sys/kernel/mm/transparent_hugepage/enabled

This will return something like:

[always] madvise never

Note the word surrounded by “[]” is what this option is set to. So the above is set to “always”. To disable it, we want it set to “never”.

I’ve found the easiest way to change/set this is to add the below to your /etc/rc.local file (and then reboot your system):

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

There is another method listed here if you use grub to boot your kernel.

If you try to enable TokuDB without first changing this, then from the command line, you would see the following error:

MariaDB [(none)]> install soname 'ha_tokudb';
ERROR 1123 (HY000): Can't initialize function 'TokuDB'; 
Plugin initialization function failed.

Or, in the error log, you would see this *slightly* more clear message:

TokuDB: Transparent huge pages are enabled, according to 
/sys/kernel/mm/transparent_hugepage/enabled.  TokuDB will be disabled. 
To use TokuDB disable huge pages in your kernel or, for testing, set 
the environment variable TOKU_HUGE_PAGES_OK to 1
140714 17:25:57 [ERROR] TokuDB:  Huge pages are enabled, disable them before continuing
140714 17:25:57 [ERROR] Plugin 'TokuDB' init function returned error.
140714 17:25:57 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.

If interested in how to enable TokuDB in MariaDB, please visit here or here.

Or, if you’re interested in why TokuDB requires hugepages to be disabled, PeterZ wrote a nice post about that here.

Hope this helps.

 

Enabling TokuDB in MariaDB is a Breeze

TokuDB is gaining more and more popularity and many people are finding it very helpful for certain cases.

Using/enabling it in some distributions can be quite a pain, but enabling it in MariaDB is easy as 1-2-…, well, only 1-2, since that’s all there is to it!

1. Ensure you have the correct MariaDB version:

  • MariaDB 5.5.36+
  • MariaDB 10.0.9+
  • Note: Linux 64-bit systems only – specific packages include: Ubuntu, Debian, Fedora, CentOS, Red Hat
  • Note: If using the Linux tarball – it must be the version built with glibc 2.14+

2. Run this command:

INSTALL SONAME 'ha_tokudb';

or update my.cnf file with:

[mysqld]
plugin-load=ha_tokudb

There is one requirement from TokuDB, which is to have transparent hugepages disabled on Linux, so some folks may indeed have a 3rd step, if you have this enabled. Luckily, this is very easy to check and change as well.

When you have it enabled correctly, you will see it listed when invoking SHOW ENGINES:

MariaDB [(none)]> show engines;
+--------+---------+------------------------------------------------...
| Engine | Support | Comment                                        ...
+--------+---------+------------------------------------------------...
| ...
| TokuDB | YES     | Tokutek TokuDB Storage Engine with Fractal Tree...
| ...

If interested, there are more details and specifics on enabling TokuDB in MariaDB here. (For instance, if you’re running 5.5.33-5.5.35 or 10.0.5-10.0.8, you may also be able to enable TokuDB, but in a slightly different manner. But there are a few more other details as well.)

Hope this helps.

 

MariaDB 10.0.11 Overview and Highlights

MariaDB 10.0.11 was recently released, and is available for download here:

https://downloads.mariadb.org/mariadb/10.0.11/

This is the second GA release of MariaDB 10.0, and 12th overall release of MariaDB 10.0.

This is primarily a bug-fix release.

Here are the main items of note:

  1. Updated TokuDB engine to version 7.1.6
  2. Updated Spider storage engine to version 3.2 (now Gamma)
  3. Updated XtraDB storage engine to version 5.6.17-65.0
  4. Updated InnoDB storage engine to version 5.6.17
  5. Updated performance_schema to version 5.6.17
  6. Updated Connect, and OQGraph engines.
  7. Online ALTER TABLE works for partitioned tables
  8. New system variable default_regex_flags. To make MariaDB RLIKE operator behave in a non-standard but backward compatible way use
    SET @@default_regex_flags='DOTALL';
  9. As per the MariaDB Deprecation Policy, this will be the last release of MariaDB 10.0 for both Ubuntu 12.10 “Quantal” and Mint 14 “Nadia”.

You can read more about the 10.0.11 release here:

https://mariadb.com/kb/en/mariadb-10011-release-notes/

And if interested, you can review the full list of changes in 10.0.11 (changelogs) here:

https://mariadb.com/kb/en/mariadb-10011-changelog/

Hope this helps.

 

MariaDB 10.0.10 Overview and Highlights

MariaDB 10.0.10 was recently released, and is available for download here:

https://downloads.mariadb.org/mariadb/10.0.10/

This is the first GA (“Generally Availability“, aka “recommended for production systems”) release of MariaDB 10.0, and 11th overall release of MariaDB 10.0.

Since this is the initial 10.0 GA release, this is primarily a bug-fix and polishing release.

Here are the main items of note:

  1. The Audit Plugin is now included in MariaDB (MDEV-5584)
  2. Improved XtraDB performance by fixing incorrect calculation of flushed pages (MDEV-5949)
  3. Fix for GTID duplicate key multi-master corruption bug (MDEV-5804)
  4. Default TokuDB compression is now TOKUDB_ZLIB (instead of TOKUDB_UNCOMPRESSED)
  5. Various algorithm improvements for engine-independent table statistics (EITS) (MDEV-5901, MDEV-5917, MDEV-5950, MDEV-5962, MDEV-5926)

You can read more about the 10.0.10 release here:

https://mariadb.com/kb/en/mariadb-10010-release-notes/

And if interested, you can review the full list of changes in 10.0.10 (changelogs) here:

https://mariadb.com/kb/en/mariadb-10010-changelog/

Hope this helps.