innodb_ruby icon indicating copy to clipboard operation
innodb_ruby copied to clipboard
verified publisher icon jeremycole

Add support for MySQL 8.0

Open jeremycole opened this issue 4 years ago • 5 comments

Add support for MySQL 8.0. There are a number of changes needed to support MySQL 8.0, and some new data structures to implement:

  • [x] Support new headers in FSP pages
  • [x] Support for reconstructing full BLOB content
  • [ ] Support new page types:
    • [ ] UNKNOWN (13)
    • [ ] COMPRESSED (14)
    • [ ] ENCRYPTED (15)
    • [ ] COMPRESSED_AND_ENCRYPTED (16)
    • [ ] ENCRYPTED_RTREE (17)
    • [x] SDI_BLOB (18) (aka BLOB)
    • [ ] SDI_ZBLOB (19) (aka ZBLOB)
    • [ ] LEGACY_DBLWR (20)
    • [ ] RSEG_ARRAY (21)
    • [ ] LOB_INDEX (22)
    • [ ] LOB_DATA (23)
    • [ ] LOB_FIRST (24)
    • [ ] ZLOB_FIRST (25)
    • [ ] ZLOB_DATA (26)
    • [ ] ZLOB_INDEX (27)
    • [ ] ZLOB_FRAG (28)
    • [ ] ZLOB_FRAG_ENTRY (29)
    • [x] SDI (17853) (aka INDEX)
    • [ ] RTREE (17854)
  • [x] Support for new data dictionary structures:
    • [x] Read SDI information from InnoDB tablespace files
    • [x] Using SDI to make record describers
    • [x] Refactor current data dictionary to support old and new
  • [ ] Support new mysql data dictionary tables
    • [x] https://github.com/jeremycole/innodb_ruby/issues/205
    • [x] https://github.com/jeremycole/innodb_ruby/issues/207
  • [ ] Support for LOB
  • [ ] Support for Compressed LOB (ZLOB)
  • [ ] Support for online-added columns?
  • [ ] Support for undo tablespaces
  • [ ] ???

Sub-issue list:

  • [x] https://github.com/jeremycole/innodb_ruby/issues/203
  • [x] https://github.com/jeremycole/innodb_ruby/issues/185
  • [ ] https://github.com/jeremycole/innodb_ruby/issues/186
  • [ ] https://github.com/jeremycole/innodb_ruby/issues/187

PR list:

  • [x] https://github.com/jeremycole/innodb_ruby/pull/184
  • [x] https://github.com/jeremycole/innodb_ruby/pull/188
  • [x] https://github.com/jeremycole/innodb_ruby/pull/189
  • [x] https://github.com/jeremycole/innodb_ruby/pull/190
  • [x] https://github.com/jeremycole/innodb_ruby/pull/191
  • [x] https://github.com/jeremycole/innodb_ruby/pull/202
  • [x] https://github.com/jeremycole/innodb_ruby/pull/206
  • [x] https://github.com/jeremycole/innodb_ruby/pull/208
  • [x] https://github.com/jeremycole/innodb_ruby/pull/209
  • [x] https://github.com/jeremycole/innodb_ruby/pull/213

See also: https://github.com/jeremycole/innodb_ruby/issues/41, https://github.com/jeremycole/innodb_ruby/issues/44.

jeremycole avatar Jul 18 '21 20:07 jeremycole

Related?

# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
	16: from /usr/local/bin/innodb_space:23:in `<main>'
	15: from /usr/local/bin/innodb_space:23:in `load'
	14: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
	13: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
	12: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
	11: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
	10: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
	 9: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
	 8: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
	 7: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
	 6: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
	 5: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
	 4: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
	 3: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
	 2: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
	 1: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055a140fed580> (NoMethodError)

8.0.28

utdrmac avatar Sep 23 '22 16:09 utdrmac 

[root@MySQL80 mysql]# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
	16: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `<main>'
	15: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `load'
	14: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
	13: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
	12: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
	11: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
	10: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
	 9: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
	 8: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
	 7: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
	 6: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
	 5: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
	 4: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
	 3: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
	 2: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
	 1: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055b0ae47b968> (NoMethodError)
[root@MySQL80 mysql]#

8.0.18

lhsaq2009 avatar May 15 '23 07:05 lhsaq2009

I filed MySQL Bug 111079: InnoDB SDI index description has fields out-of-order as implementing SDI caused me to find a subtle issue with the fields being out of order internally.

jeremycole avatar May 22 '23 20:05 jeremycole

I filed MySQL Bug 111112: JSON SDI default_value in columns object includes trailing zero bytes as implementing SDI caused me to notice that the included default_value field includes trailing zero bytes (seems to be encoding a raw buffer from C++).

jeremycole avatar May 22 '23 20:05 jeremycole