Posts Tagged ‘assertion failure’

MySQL 5.5.39 Overview and Highlights

Friday, August 1st, 2014

MySQL 5.5.39 was recently released (it is the latest MySQL 5.5, is GA), and is available for download here:

http://downloads.skysql.com/archive/index/p/mysql/v/5.5.39

This release, similar to the last 5.5 release, is mostly uneventful.

There were two “Functionality Added or Changed” and 24 bugs fixed.

The “Functionality Added or Changed” changes are:

  • CMake support was updated to handle CMake version 3.
  • The timed_mutexes system variable has no effect and is deprecated.

Out of the 24 bugs, most seemed rather minor or obscure, but here are the ones I think are worth noting (crashing, security, wrong results, deadlock):

  • InnoDB: Opening a parent table that has thousands of child tables could result in a long semaphore wait condition.
  • Partitioning: Selecting from a table having multiple columns in its primary key and partitioned by LIST COLUMNS(R), where R was the last (rightmost) column listed in the primary key definition, returned an incorrect result. (Bug #71095)
  • Replication: When using row-based replication, updating or deleting a row on the master that did not exist on the slave led to failure of the slave when it tried to process the change. This problem occurred with InnoDB tables lacking a primary key. (Bug #72085)
  • Replication: A group of threads involved in acquiring locks could deadlock in a certain scenario. (Bug #69954)
  • ALTER TABLE on a partitioned table could result in the wrong storage engine being written into the table’s .frm file and displayed in SHOW CREATE TABLE.
  • MyISAM temporary files could be used to mount a code-execution attack.
  • An assertion could be raised when creating a index on a prefix of a TINYBLOB or GEOMETRY column in an InnoDB column.
  • Deadlock could occur if three threads simultaneously performed INSTALL PLUGIN, SHOW VARIABLES, and mysql_change_user(). (Bug #71236, Bug #72870)
  • MySQL did not compile with Bison 3. A workaround is to downgrade to Bison 2. (Bug #71250)

For reference, the full 5.5.39 changelog can be viewed here:

http://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-39.html

Hope this helps.

 

Dealing with Assertion failure in file fut0lst.ic : addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA

Friday, December 9th, 2011

I recently wrote an article on dealing with an assertion failure in log/log0recv.c, specifically !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table).

I mention it because this occurred after a system outage, and I just encountered another system outage (either HDD power outage or some other serious HDD event), with a completely different assertion failure and error message. Similar to the one above, it’s also kind of obscure, so I wanted to post this so people searching for it will find this.

For reference, the first outage assertion failure was this:

InnoDB: Assertion failure in thread 139838283589376 in file
log/log0recv.c line 1094
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) ==
dict_table_is_comp(index->table)

Here is the new assertion failure:

111201 16:45:00 InnoDB: Assertion failure in thread 4500 in
file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >=
FIL_PAGE_DATA

As you can see, they are completely different, yet both due to similar, catastrophic events:

In fact, there are a number of relevant/useful snippets from this error log (in case you see one or more of these):

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 2248, page 22 in doublewrite buf.
InnoDB: Warning: a page in the doublewrite buffer is not within space
InnoDB: bounds; space id 0 page number 1775, page 31 in doublewrite buf.
...
111201 16:45:00 InnoDB: ERROR: We were only able to scan the log up to
InnoDB: 670800896, but a checkpoint was at 670801309.
InnoDB: It is possible that the database is now corrupt!
111201 16:45:00 InnoDB: Error: page 7 log sequence number 924466223
InnoDB: is in the future! Current system log sequence number 670801309.
...
111201 16:45:00 InnoDB: Assertion failure in thread 4500 in file
fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset
>= FIL_PAGE_DATA
...
111201 16:45:00 - mysqld got exception 0xc0000005 ;
...
000000013F6A4C6F mysqld.exe!my_osmaperr()
000000013F69B9A0 mysqld.exe!my_osmaperr()
000000013F4727D6 mysqld.exe!?ha_initialize_handlerton@@
YAHPEAUst_plugin_int@@@Z()
000000013F46C172 mysqld.exe!?plugin_lock_by_name@@
YAPEAUst_plugin_int@@PEAVTHD@@PEBUst_mysql_lex_string@@H@Z()
000000013F4713E9 mysqld.exe!?plugin_init@@YAHPEAHPEAPEADH@Z()
000000013F45BDA7 mysqld.exe!handle_shutdown()
000000013F45C91A mysqld.exe!?win_main@@YAHHPEAPEAD@Z()
000000013F45CD90 mysqld.exe!?mysql_service@@YAHPEAX@Z()
000000013F45D0A3 mysqld.exe!?mysqld_main@@YAHHPEAPEAD@Z()
000000013F7FBB27 mysqld.exe!my_mb_ctype_mb()
00000000777F652D kernel32.dll!BaseThreadInitThunk()
0000000077A2C521 ntdll.dll!RtlUserThreadStart()

As you can see, the corruption was severe … so severe that InnoDB wouldn’t even start with innodb_force_recovery set to 6!

As discussed in the previous post, setting innodb_doublewrite=1, innodb_flush_log_at_trx_commit=1, sync_binlog=1, and having a battery backed cache can be the best bets against such issues.

However, I will say that in this case, innodb_flush_log_at_trx_commit=1 and innodb_doublewrite=1 were both set, and binary logging had been disabled, leaving only the battery backed cache to be desired.

Luckily, this was a test environment (and recovery from a backup was possible). Otherwise, using the InnoDB Recovery Tool would be necessary, which can be an undertaking.

Morals of the story: take regular backups (as always) and invest in a battery backed cache. :)

 
 

Added a Table of Contents

Wednesday, November 9th, 2011

Not a big deal, but I just added a “Table of Contents” page to my blog to make finding older articles much easier.

I noticed most of my posts are quite lengthy, and it can take a bit of searching/clicking to find an older entry. So unless you happen to recall the ‘month/year’ it was published, which I don’t even remember that, then hopefully this will help.

Really simple, and looks just like this:

My hopes are that this will aid in making some posts easier to find (such as ones about InnoDB Recovery, Recovery with an Individual .ibd, Proxy-related articles, Error-related articles, How-to posts, and so forth).

You can see the full “table of contents” here:

http://www.chriscalender.com/?page_id=399

Happy reading :)

 
 
 

Dealing with Assertion failure in log/log0recv.c – !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)

Sunday, November 6th, 2011

This is a somewhat uncommon error, so I wanted to take a moment and post the error and an explanation in order to make it easier for those who run into this in the future.

The main error is this (full pasted below):

InnoDB: Assertion failure in thread 139838283589376 in file log/log0recv.c line 1094
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)

Basically, this is what it can look like after a power outage. In fact, this is the only time I have seen it, but I won’t say that it is the only way you could encounter it.

How do I deal with this, you ask?

Well, the above error shows that the internal InnoDB dictionary is corrupt.

Unfortunately, in this situation, you must recreate the tablespace. And if you don’t have a current backup + binlogs, then that means you will also need to force InnoDB recovery (i.e., set innodb_force_recovery) in order to dump all of your data.

Once you have the dump, you will need to start from scratch with InnoDB, so to speak. This means deleting (or copy to safe location) all ib_logfile*, ibdata*, *.ibd, and *.frm of all InnoDB tables. Then, remove the innodb_force_recovery option and restart mysqld (allowing it to create new ibdata file(s) and ib_logfile files). And finally, reimport the data you obtained from the mysqldump.

Now, I should reiterate that this was due to a power failure. It is possible for InnoDB to handle this in a more graceful fashion (perhaps less/no corruption), but it invloves setting the following two variables (warning: both will have an impact on performance, so best to test thoroughly before implementing in production):

sync_binlog = 1
innodb_flush_log_at_trx_commit = 1

For sync_binlog:

“A value of 1 is the safest choice because in the event of a crash you lose at most one statement or transaction from the binary log. However, it is also the slowest choice (unless the disk has a battery-backed cache, which makes synchronization very fast).”

http://dev.mysql.com/doc/refman/5.5/en/replication-options-binary-log.html#sysvar_sync_binlog

For innodb_flush_log_at_trx_commit:

“When the value is 1 (the default), the log buffer is written out to the log file at each transaction commit and the flush to disk operation is performed on the log file. … The default value of 1 is required for full ACID compliance. You can achieve better performance by setting the value different from 1, but then you can lose up to one second worth of transactions in a crash. With a value of 0, any mysqld process crash can erase the last second of transactions. With a value of 2, only an operating system crash or a power outage can erase the last second of transactions. InnoDB’s crash recovery works regardless of the value. … For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, use innodb_flush_log_at_trx_commit=1 and sync_binlog=1 in your master server my.cnf file.”

However, even with both of these set, do note this caution stated in the manual:

“Caution: Many operating systems and some disk hardware fool the flush-to-disk operation. They may tell mysqld that the flush has taken place, even though it has not. Then the durability of transactions is not guaranteed even with the setting 1, and in the worst case a power outage can even corrupt the InnoDB database. Using a battery-backed disk cache in the SCSI disk controller or in the disk itself speeds up file flushes, and makes the operation safer. You can also try using the Unix command hdparm to disable the caching of disk writes in hardware caches, or use some other command specific to the hardware vendor.”

http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit

And for more details on the how “some disk hardware fool the flush-to-disk operation”, then please see the following links:

http://peter-zaitsev.livejournal.com/12639.html
http://brad.livejournal.com/2116715.html?thread=10292331
http://bugs.mysql.com/bug.php?id=29221

For reference, here is the full error log snippet (after starting up mysqld after the outage):

111026 22:44:35 mysqld_safe Starting mysqld daemon with databases from /mysql/data
111026 22:44:35 [Note] PrimeBase XT (PBXT) Engine 1.0.11-7 Pre-GA loaded...
111026 22:44:35 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
111026 22:44:35 [Note] The server was not shutdown correctly, recovery required
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
111026 22:44:37  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 363485996068
111026 22:44:37  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 363487670956
111026 22:44:38  InnoDB: Error: page 4 log sequence number 363490119646
InnoDB: is in the future! Current system log sequence number 363487670956.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: for more information.
111026 22:44:38  InnoDB: Assertion failure in thread 139838283589376 in file log/log0recv.c line 1094
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
111026 22:44:38 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=202
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1683122 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
111026 22:44:38  InnoDB: Error: page 147457 log sequence number 363487671024
InnoDB: is in the future! Current system log sequence number 363487670956.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: for more information.
Fatal signal 11 while backtracing
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x99199e]
/usr/sbin/mysqld(handle_segfault+0x353) [0x5e9563]
/lib64/libpthread.so.0(+0xf490) [0x7f3514595490]
/lib64/libc.so.6(gsignal+0x35) [0x7f3513881905]
/lib64/libc.so.6(abort+0x175) [0x7f35138830e5]
/usr/sbin/mysqld() [0x89ee6f]
/usr/sbin/mysqld() [0x8a0322]

 

All in all, hope you never encounter this, but if you do, hopefully this helps you get on the right track sooner than later. :)

 
 
 


Period Panties by Period Panteez Menstrual Underwear Menstruation PMS Panty