Home > CentOS, Linux, MySQL > Upgrade MySQL cluster severalnines install from 7.2.10-1 to 7.3.2-1

Upgrade MySQL cluster severalnines install from 7.2.10-1 to 7.3.2-1

In short, I have one MySQL Cluster of 2 managament and 2 data nodes installed by http://www.severalnines.com/ script generated at their install script generator at http://www.severalnines.com/cluster-configurator/

The NDB version was 7.2.10-1

Some new clients requested an update , in a view to make use of the new bug fixes, features and optimizations.
More info : Changes in MySQL Cluster NDB 7.3.2 (5.6.11-ndb-7.3.2)

So, the steps were:

0. THE MOST IMPORTANT: make a backup 🙂
1. get the RPMS from http://dev.mysql.com/downloads/cluster/


MySQL-Cluster-test-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-devel-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm
MySQL-Cluster-embedded-gpl-7.3.2-1.el6.x86_64.rpm

2. Upload them to the first head node, note from there I have ssh password-less sudo access to the other nodes.
3. Make sure the cmon service is stopped at all nodes before to stop any of the ndb_mgmd or ndbd services.

$ service cmon stop
$ service cmon status

4. Start the rolling stop, upgrade , restart, mysql_upgrade of the ndb_mgmd nodes, one by one.
4.1 ndb_mgmd node 1:

# stop ndb_mgmd service:
$ service ndb_mgmd stop
$ service ndb_mgmd status

# check what to be updated : 
$ yum search | grep MySQL-Cluster
MySQL-Cluster-client-gpl.x86_64 : MySQL Cluster - Client
MySQL-Cluster-server-gpl.x86_64 : MySQL: a very fast and reliable SQL database server
MySQL-Cluster-shared-compat-gpl.x86_64 : MySQL shared client libraries for MySQL 5.1.67-1, 5.0.96-1, 4.1.23-0, 4.0.27-0
MySQL-Cluster-shared-gpl.x86_64 : MySQL Cluster - Shared libraries
$ yum  info MySQL-Cluster-server-gpl
Loaded plugins: product-id, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Installed Packages
Name        : MySQL-Cluster-server-gpl
Arch        : x86_64
Version     : 7.2.10
Release     : 1.el6
Size        : 470 M
Repo        : installed

# go to the directory with the new packages: 
$ cd ~/mysqlcluster-72-rpm/cluster/repo/MySQL-Cluster-7.3.2-1

# execute the update
$ sudo yum upgrade MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64.rpm  MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64.rpm MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64.rpm

# Now the update will face an script error:
$ sudo yum upgrade MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64.rpm  MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64.rpm MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64.rpm b
Loaded plugins: product-id, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
epel/metalink                                                                                                                                                                 |  21 kB     00:00     
epel                                                                                                                                                                          | 4.2 kB     00:00     
epel/primary_db                                                                                                                                                               | 5.3 MB     00:09     
epg                                                                                                                                                                           | 1.2 kB     00:00     
puppetlabs-deps                                                                                                                                                               | 1.9 kB     00:00     
puppetlabs-deps/primary_db                                                                                                                                                    |  20 kB     00:00     
puppetlabs-products                                                                                                                                                           | 1.9 kB     00:00     
puppetlabs-products/primary_db                                                                                                                                                |  93 kB     00:00     
vmware-tools                                                                                                                                                                  |  951 B     00:00     
Setting up Upgrade Process
Examining MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64.rpm: MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64
Marking MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64.rpm as an update to MySQL-Cluster-client-gpl-7.2.10-1.el6.x86_64
Examining MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm: MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64
Marking MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm as an update to MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64
Examining MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64.rpm: MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64
Marking MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64.rpm as an update to MySQL-Cluster-shared-gpl-7.2.10-1.el6.x86_64
Examining MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64.rpm: MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64
Marking MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64.rpm as an update to MySQL-Cluster-shared-compat-gpl-7.2.10-1.el6.x86_64
No Match for argument: b
No package b available.
Resolving Dependencies
--> Running transaction check
---> Package MySQL-Cluster-client-gpl.x86_64 0:7.2.10-1.el6 will be updated
---> Package MySQL-Cluster-client-gpl.x86_64 0:7.3.2-1.el6 will be an update
---> Package MySQL-Cluster-server-gpl.x86_64 0:7.2.10-1.el6 will be updated
---> Package MySQL-Cluster-server-gpl.x86_64 0:7.3.2-1.el6 will be an update
---> Package MySQL-Cluster-shared-compat-gpl.x86_64 0:7.2.10-1.el6 will be updated
---> Package MySQL-Cluster-shared-compat-gpl.x86_64 0:7.3.2-1.el6 will be an update
---> Package MySQL-Cluster-shared-gpl.x86_64 0:7.2.10-1.el6 will be updated
---> Package MySQL-Cluster-shared-gpl.x86_64 0:7.3.2-1.el6 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================================================
 Package                                              Arch                        Version                             Repository                                                                Size
=====================================================================================================================================================================================================
Updating:
 MySQL-Cluster-client-gpl                             x86_64                      7.3.2-1.el6                         /MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64                              81 M
 MySQL-Cluster-server-gpl                             x86_64                      7.3.2-1.el6                         /MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64                             594 M
 MySQL-Cluster-shared-compat-gpl                      x86_64                      7.3.2-1.el6                         /MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64                       11 M
 MySQL-Cluster-shared-gpl                             x86_64                      7.3.2-1.el6                         /MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64                             8.4 M

Transaction Summary
=====================================================================================================================================================================================================
Upgrade       4 Package(s)

Total size: 694 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64                                                                                                                                1/8 
  Updating   : MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64                                                                                                                                       2/8 
  Updating   : MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64                                                                                                                                       3/8 
Error in PREIN scriptlet in rpm package MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64

******************************************************************
A MySQL server package (MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64) is installed.

Upgrading directly from MySQL 7.2 to MySQL 7.3 may not
be safe in all cases.  A manual dump and restore using mysqldump is
recommended.  It is important to review the MySQL manual's Upgrading
section for version-specific incompatibilities.

A manual upgrade is required.

- Ensure that you have a complete, working backup of your data and my.cnf
  files
- Shut down the MySQL server cleanly
- Remove the existing MySQL packages.  Usually this command will
  list the packages you should remove:
  rpm -qa | grep -i '^mysql-'

  You may choose to use 'rpm --nodeps -ev <package-name>' to remove
  the package which contains the mysqlclient shared library.  The
  library will be reinstalled by the MySQL-Cluster-shared-compat package.
- Install the new MySQL packages supplied by Oracle and/or its affiliates
- Ensure that the MySQL server is started
- Run the 'mysql_upgrade' program

This is a brief description of the upgrade process.  Important details
can be found in the MySQL manual, in the Upgrading section.
******************************************************************
error: %pre(MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping MySQL-Cluster-server-gpl-7.3.2-1.el6
  Cleanup    : MySQL-Cluster-shared-compat-gpl-7.2.10-1.el6.x86_64                                                                                                                               5/8 
  Cleanup    : MySQL-Cluster-client-gpl-7.2.10-1.el6.x86_64                                                                                                                                      6/8 
  Cleanup    : MySQL-Cluster-shared-gpl-7.2.10-1.el6.x86_64                                                                                                                                      7/8 
Installed products updated.
  Verifying  : MySQL-Cluster-shared-gpl-7.3.2-1.el6.x86_64                                                                                                                                       1/8 
  Verifying  : MySQL-Cluster-client-gpl-7.3.2-1.el6.x86_64                                                                                                                                       2/8 
  Verifying  : MySQL-Cluster-shared-compat-gpl-7.3.2-1.el6.x86_64                                                                                                                                3/8 
  Verifying  : MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64                                                                                                                                       4/8 
MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64 was supposed to be removed but is not!
  Verifying  : MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64                                                                                                                                      5/8 
  Verifying  : MySQL-Cluster-shared-gpl-7.2.10-1.el6.x86_64                                                                                                                                      6/8 
  Verifying  : MySQL-Cluster-shared-compat-gpl-7.2.10-1.el6.x86_64                                                                                                                               7/8 
  Verifying  : MySQL-Cluster-client-gpl-7.2.10-1.el6.x86_64                                                                                                                                      8/8 

Updated:
  MySQL-Cluster-client-gpl.x86_64 0:7.3.2-1.el6                  MySQL-Cluster-shared-compat-gpl.x86_64 0:7.3.2-1.el6                  MySQL-Cluster-shared-gpl.x86_64 0:7.3.2-1.el6                 

Failed:
  MySQL-Cluster-server-gpl.x86_64 0:7.2.10-1.el6                                                    MySQL-Cluster-server-gpl.x86_64 0:7.3.2-1.el6                                                   

Complete!

# fix : manually remove the MySQL-Cluster-server before the install
$ sudo rpm -e MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64
$ sudo yum localinstall MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm

# start the mysqld
$ service mysql start

IMPORTANT: It will fail with the following error in case you have engine-condition-pushdown set at my.cnf :
[ERROR] /usr/sbin/mysqld: unknown variable ‘engine-condition-pushdown=1’
Details :

2013-07-17 11:04:55 25507 [Note] InnoDB: Tablespace and datafile system tables created.
2013-07-17 11:04:55 25507 [Note] InnoDB: Waiting for purge to start
2013-07-17 11:04:55 25507 [Note] InnoDB: 5.6.11 started; log sequence number 304036700452
2013-07-17 11:04:55 25507 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2013-07-17 11:04:55 25507 [Note] NDB: NodeID is 23, management server '10.95.109.216:1186'
2013-07-17 11:04:56 25507 [Note] NDB[0]: NodeID: 23, all storage nodes connected
2013-07-17 11:04:56 25507 [Note] NDB[1]: NodeID: 25, all storage nodes connected
2013-07-17 11:04:57 25507 [Note] NDB[2]: NodeID: 5, all storage nodes connected
2013-07-17 11:04:58 25507 [Note] NDB[3]: NodeID: 6, all storage nodes connected
2013-07-17 11:04:59 25507 [Note] NDB[4]: NodeID: 7, all storage nodes connected
2013-07-17 11:04:59 25507 [Note] NDB[5]: NodeID: 8, all storage nodes connected
2013-07-17 11:04:59 25507 [Note] Starting Cluster Binlog Thread
2013-07-17 11:04:59 25507 [ERROR] /usr/sbin/mysqld: unknown variable 'engine-condition-pushdown=1'
2013-07-17 11:04:59 25507 [ERROR] Aborting

# comment that engine-condition-pushdown at /etc/my.cnf and start the service again:


$ service mysql start
$ service mysql status 

# execute mysql_update in a view to implement the changes at MySQL v 5.6
$ mysql_update -p 
Enter password: 
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck' with connection arguments: '--socket=/var/lib/mysql/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/var/lib/mysql/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.ndb_apply_status                             OK
mysql.ndb_binlog_index                             OK
mysql.ndb_index_stat_head                          OK
mysql.ndb_index_stat_sample                        OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Running 'mysql_fix_privilege_tables'...
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/var/lib/mysql/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/var/lib/mysql/mysql.sock' 
Warning: Using a password on the command line interface can be insecure.
cmon.alarm                                         OK
cmon.alarm_hosts                                   OK
cmon.alarm_log                                     OK
cmon.backup                                        OK
cmon.backup_log                                    OK
cmon.backup_schedule                               OK
cmon.cluster                                       OK
cmon.cluster_config                                OK
cmon.cluster_configuration                         OK
cmon.cluster_configuration_templates               OK
cmon.cluster_event_types                           OK
cmon.cluster_log                                   OK
cmon.cluster_severity_types                        OK
cmon.cluster_state                                 OK
cmon.cluster_statistics                            OK
cmon.cluster_statistics_history                    OK
cmon.cmon_cluster_counters                         OK
cmon.cmon_cluster_graphs                           OK
cmon.cmon_configuration                            OK
cmon.cmon_daily_job                                OK
cmon.cmon_galera_counters                          OK
cmon.cmon_host_log                                 OK
cmon.cmon_job                                      OK
cmon.cmon_job_message                              OK
cmon.cmon_local_mysql_job                          OK
cmon.cmon_mysql_counters                           OK
cmon.cmon_mysql_grants                             OK
cmon.cmon_mysql_graphs                             OK
cmon.cmon_mysql_manual_grants                      OK
cmon.cmon_mysql_users                              OK
cmon.cmon_schema_uploads                           OK
cmon.cmon_status                                   OK
cmon.cmon_sw_package                               OK
cmon.cmon_uploads                                  OK
cmon.cmon_user                                     OK
cmon.configurator_nodemap                          OK
cmon.cpu_info                                      OK
cmon.cpu_stats                                     OK
cmon.cpu_stats_history                             OK
cmon.database_conf                                 OK
cmon.db_growth                                     OK
cmon.db_growth_hashmap                             OK
cmon.db_notifications                              OK
cmon.disk_stats                                    OK
cmon.disk_stats_history                            OK
cmon.diskdata                                      OK
cmon.diskdata_history                              OK
cmon.email_notification                            OK
cmon.ext_proc                                      OK
cmon.galera_garbd_proc                             OK
cmon.galera_status                                 OK
cmon.haproxy_server                                OK
cmon.hosts                                         OK
cmon.license                                       OK
cmon.mailserver                                    OK
cmon.memcache_statistics                           OK
cmon.memory_usage                                  OK
cmon.memory_usage_history                          OK
cmon.mysql_advisor                                 OK
cmon.mysql_advisor_history                         OK
cmon.mysql_advisor_reco                            OK
cmon.mysql_backup                                  OK
cmon.mysql_explains                                OK
cmon.mysql_global_statistics                       OK
cmon.mysql_global_statistics_history               OK
cmon.mysql_innodb_status                           OK
cmon.mysql_master_status                           OK
cmon.mysql_memory_usage                            OK
cmon.mysql_performance_meta                        OK
cmon.mysql_performance_probes                      OK
cmon.mysql_performance_results                     OK
cmon.mysql_processlist                             OK
cmon.mysql_query_histogram                         OK
cmon.mysql_repl_bw                                 OK
cmon.mysql_repl_link                               OK
cmon.mysql_replication_recovery                    OK
cmon.mysql_server                                  OK
cmon.mysql_slave_status                            OK
cmon.mysql_slow_queries                            OK
cmon.mysql_states                                  OK
cmon.mysql_statistics                              OK
cmon.mysql_statistics_history                      OK
cmon.mysql_variables                               OK
cmon.ndbinfo_diskpagebuffer                        OK
cmon.ndbinfo_logbuffers                            OK
cmon.ndbinfo_logbuffers_history                    OK
cmon.ndbinfo_logspaces                             OK
cmon.ndbinfo_logspaces_history                     OK
cmon.net_stats                                     OK
cmon.net_stats_history                             OK
cmon.node_state                                    OK
cmon.node_statistics                               OK
cmon.node_statistics_history                       OK
cmon.processes                                     OK
cmon.ram_stats                                     OK
cmon.ram_stats_history                             OK
cmon.restore                                       OK
cmon.restore_log                                   OK
cmon.schema_object                                 OK
cmon.table_growth                                  OK
cmon.table_growth_hashmap                          OK
cmon.top                                           OK
cmon.user_event_categories                         OK
cmon.user_events                                   OK
connect.auth_group                                 OK
connect.auth_group_permissions                     OK
connect.auth_message                               OK
connect.auth_permission                            OK
connect.auth_user                                  OK
connect.auth_user_groups                           OK
connect.auth_user_user_permissions                 OK
connect.crm_adminprofile                           OK
connect.devices                                    OK
connect.django_admin_log                           OK
connect.django_content_type                        OK
connect.django_session                             OK
connect.registration_attempts                      OK
connect.users                                      OK
ndbinfo.blocks                                     OK
ndbinfo.config_params                              OK
OK

#  now restart the ndb_mgmd service
$ service ndb_mgmd start
$ service ndb_mgmd status

4.2 ndb_mgmd node 2:
Exactly the same install, including the my.cnf fix

4.3 ndbd node 1:
Here we need only to update the MySQL-cluster-server package

# stop the local ndbd service:
$ service ndbd_3 stop
$ service ndbd_3 status

# go to the directory with the new packages: 
$ cd ~/mysqlcluster-72-rpm/cluster/repo/MySQL-Cluster-7.3.2-1
$ sudo rpm -e MySQL-Cluster-server-gpl-7.2.10-1.el6.x86_64
$ sudo yum localinstall MySQL-Cluster-server-gpl-7.3.2-1.el6.x86_64.rpm

# start the local ndbd service:
$ service ndbd_3 start
$ service ndbd_3 status

# at a management node start the updated ndbd node

[ndbd(NDB)]     2 node(s)
id=3    @10.95.109.195  (mysql-5.6.11 ndb-7.3.2, not started)
id=4    @10.95.109.196  (mysql-5.5.29 ndb-7.2.10, Nodegroup: 0)

[ndb_mgmd(MGM)] 2 node(s)
id=1    @10.95.109.216  (mysql-5.6.11 ndb-7.3.2)
id=2    @10.95.109.217  (mysql-5.6.11 ndb-7.3.2)
ndb_mgm> 3 start 
ndb_mgm> Node 3: Start initiated (version 7.3.2)

4.4 ndbd node 2:
Same procedure here, after its start, the status should be like :

Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=3    @10.95.109.195  (mysql-5.6.11 ndb-7.3.2, Nodegroup: 0)
id=4    @10.95.109.196  (mysql-5.6.11 ndb-7.3.2, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 2 node(s)
id=1    @10.95.109.216  (mysql-5.6.11 ndb-7.3.2)
id=2    @10.95.109.217  (mysql-5.6.11 ndb-7.3.2)

[mysqld(API)]   21 node(s)

5. Now start all the cmon monitors at all the 4 nodes


$ for crap in 10.95.109.195 10.95.109.196 10.95.109.216 10.95.109.217; do  ssh -T -p 22 $crap sudo /etc/init.d/cmon restart; done
Stopping cmon:  ok
Starting cmon:  ok
Stopping cmon:  ok
Starting cmon:  ok
Stopping cmon:  ok
Starting cmon:  ok
Stopping cmon:  ok
Starting cmon:  ok

6. If its appropriate, restore the full production database from the backup
For the backups I use my own script at bitbucket

cheers folks, a ver si ebay habla con Solo

SlackWare4Life !

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: