Archive

Archive for the ‘OpenStack’ Category

Compile at CentOS 6.5 the new MySQL webscalesql-5.6.17 branch by Facebook, Google, LinkedIn, and Twitter

March 31, 2014 Leave a comment

http://webscalesql.org/

yeah , big buzz around that one 🙂

So I decided to check the install process:

1. Clone the repo from

root@webscalesql-5.6.clean:[Mon Mar 31 11:37:11][~]$ cd /opt/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:15][/opt]$ mkdir installs
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:17][/opt]$ cd installs/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:19][/opt/installs]$ git clone https://github.com/webscalesql/webscalesql-5.6.git
Initialized empty Git repository in /opt/installs/webscalesql-5.6/.git/
remote: Counting objects: 30397, done.
remote: Compressing objects: 100% (12678/12678), done.
remote: Total 30397 (delta 18716), reused 27620 (delta 16936)
Receiving objects: 100% (30397/30397), 47.99 MiB | 460 KiB/s, done.
Resolving deltas: 100% (18716/18716), done.

2. Update the CentOs install just in case before to play with it:

root@webscalesql-5.6.clean:[Mon Mar 31 11:40:35][/opt/installs]$ yum update -y 

3. Compile it :
Note : The final part of the prompt here “[webscalesql-5.6.17]” id the actual git branch version.

root@webscalesql-5.6.clean:[Mon Mar 31 12:23:22][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-bash: cmake: command not found

3.1 No Cmake ? The CentOS install was done without cmake etc dev tools as at production they are not needed.
Now, we need it: Install cmake

root@webscalesql-5.6.clean:[Mon Mar 31 12:23:34][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install cmake -y 
....
Installed:
  cmake.x86_64 0:2.6.4-5.el6  
root@webscalesql-5.6.clean:[Mon Mar 31 12:24:10][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake --version
cmake version 2.6-patch 4                                                                                                                                                               

3.2 Try to compile it again?

root@webscalesql-5.6.clean:[Mon Mar 31 12:24:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Running cmake version 2.6.4
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error: your C compiler: "CMAKE_C_COMPILER-NOTFOUND" was not found.   Please set CMAKE_C_COMPILER to a valid compiler path or name.
CMake Error: your CXX compiler: "CMAKE_CXX_COMPILER-NOTFOUND" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.

No luck, the development tools are missing
3.3 Install ‘Development Tools’ group

root@webscalesql-5.6.clean:[Mon Mar 31 12:24:54][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum groupinstall 'Development Tools'
Installed:
  autoconf.noarch 0:2.63-5.1.el6      automake.noarch 0:1.11.1-4.el6      bison.x86_64 0:2.4.1-5.el6                       byacc.x86_64 0:1.9.20070509-7.el6   cscope.x86_64 0:15.6-6.el6           
  ctags.x86_64 0:5.8-2.el6            diffstat.x86_64 0:1.51-2.el6        doxygen.x86_64 1:1.6.1-6.el6                     flex.x86_64 0:2.5.35-8.el6          gcc.x86_64 0:4.4.7-4.el6             
  gcc-c++.x86_64 0:4.4.7-4.el6        gcc-gfortran.x86_64 0:4.4.7-4.el6   indent.x86_64 0:2.2.10-7.el6                     intltool.noarch 0:0.41.0-1.1.el6    libtool.x86_64 0:2.2.6-15.5.el6      
  patchutils.x86_64 0:0.3.1-3.1.el6   rcs.x86_64 0:5.7-37.el6             redhat-rpm-config.noarch 0:9.0.3-42.el6.centos   rpm-build.x86_64 0:4.8.0-37.el6     subversion.x86_64 0:1.6.11-10.el6_5  
  swig.x86_64 0:1.3.40-6.el6          systemtap.x86_64 0:2.3-4.el6_5     

Dependency Installed:
  apr.x86_64 0:1.3.9-5.el6_2              apr-util.x86_64 0:1.3.9-3.el6_0.1            cloog-ppl.x86_64 0:0.15.7-1.2.el6      cpp.x86_64 0:4.4.7-4.el6       gettext-devel.x86_64 0:0.17-16.el6      
  gettext-libs.x86_64 0:0.17-16.el6       kernel-devel.x86_64 0:2.6.32-431.11.2.el6    libart_lgpl.x86_64 0:2.3.20-5.1.el6    libgcj.x86_64 0:4.4.7-4.el6    libstdc++-devel.x86_64 0:4.4.7-4.el6    
  mpfr.x86_64 0:2.4.1-6.el6               neon.x86_64 0:0.29.3-3.el6_4                 pakchois.x86_64 0:0.4-3.2.el6          ppl.x86_64 0:0.10.2-11.el6     systemtap-client.x86_64 0:2.3-4.el6_5   
  systemtap-devel.x86_64 0:2.3-4.el6_5   

3.4 Try to compile … one more time 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:30:37][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Running cmake version 2.6.4
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at cmake/build_configurations/mysql_release.cmake:43 (MESSAGE):
        aio is required on Linux, you need to install the required library:

            Debian/Ubuntu:              apt-get install libaio-dev
            RedHat/Fedora/Oracle Linux: yum install libaio-devel
            SuSE:                       zypper install libaio-devel

          If you really do not want it, pass -DIGNORE_AIO_CHECK to cmake.
          
Call Stack (most recent call first):
  CMakeLists.txt:94 (INCLUDE)
-- Configuring incomplete, errors occurred!

So install libaio libs ..

root@webscalesql-5.6.clean:[Mon Mar 31 12:32:43][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install libaio-devel

Installed:
  libaio-devel.x86_64 0:0.3.107-10.el6                                                                                                                                                               

3.5 Try to compile … again
Now the CMakeCache.txt file has to be removed in a view to start form scratch:

root@webscalesql-5.6.clean:[Mon Mar 31 12:33:23][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:33:57][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
--
---
-- Could NOT find Curses  (missing:  CURSES_LIBRARY CURSES_INCLUDE_PATH)
CMake Error at cmake/readline.cmake:82 (MESSAGE):
  Curses library not found.  Please install appropriate package,

      remove CMakeCache.txt and rerun cmake.On Debian/Ubuntu, package name is libncurses5-dev, on Redhat and derivates it is ncurses-devel.
Call Stack (most recent call first):
  cmake/readline.cmake:125 (FIND_CURSES)
  cmake/readline.cmake:194 (FIND_SYSTEM_LIBEDIT)
  CMakeLists.txt:397 (MYSQL_CHECK_READLINE)
-- Configuring incomplete, errors occurred!

So follow up the hint and installed the libs:

root@webscalesql-5.6.clean:[Mon Mar 31 12:34:25][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install ncurses-devel

Installed:
  ncurses-devel.x86_64 0:5.7-3.20090208.el6                                                                                                                                                          

3.6 Try to compile … 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:35:44][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:35:51][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
...
....
.....
-- Found Curses: /usr/lib64/libcurses.so
-- Looking for tputs in /usr/lib64/libcurses.so
-- Looking for tputs in /usr/lib64/libcurses.so - found
-- Looking for include files HAVE_READLINE_HISTORY_H
-- Looking for include files HAVE_READLINE_HISTORY_H - not found.
CMake Error at cmake/readline.cmake:186 (MESSAGE):
  Cannot find libreadline!
Call Stack (most recent call first):
  cmake/readline.cmake:194 (FIND_SYSTEM_LIBEDIT)
  CMakeLists.txt:397 (MYSQL_CHECK_READLINE)
-- Configuring incomplete, errors occurred!

readline-devel was needed indeed even we installed ncurses libs

root@webscalesql-5.6.clean:[Mon Mar 31 12:37:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install readline-devel

Installed:
  readline-devel.x86_64 0:6.0-4.el6                                                                                                                                                                  

3.7 Try to compile … 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:37:38][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:37:47][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Successfully downloaded http://googlemock.googlecode.com/files/gmock-1.6.0.zip to /opt/installs/webscalesql-5.6/source_downloads
CMake Error: Problem with tar_extract_all(): Invalid argument
CMake Error: Problem extracting tar: /opt/installs/webscalesql-5.6/source_downloads/gmock-1.6.0.zip
-- Library mysqlserver depends on OSLIBS -lpthread;m;rt;crypt;dl;aio
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6

3.8 Fix CMake Error: Problem with tar_extract_all(): Invalid argument at extracting the gmock-1.6.0.zip archive.

root@webscalesql-5.6.clean:[Mon Mar 31 12:46:05][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cd source_downloads/
root@webscalesql-5.6.clean:[Mon Mar 31 12:46:08][/opt/installs/webscalesql-5.6/source_downloads] [webscalesql-5.6.17]$ ls
gmock-1.6.0.zip
root@webscalesql-5.6.clean:[Mon Mar 31 12:46:08][/opt/installs/webscalesql-5.6/source_downloads] [webscalesql-5.6.17]$ unzip -a gmock-1.6.0.zip 

3.9 Try to compile it …. 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:47:10][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:47:12][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
 -- GMOCK_SOURCE_DIR:/opt/installs/webscalesql-5.6/source_downloads/gmock-1.6.0
-- Performing Test HAVE_NO_NULL
-- Performing Test HAVE_NO_NULL - Failed
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS - Failed
-- GTEST_LIBRARIES:gmock;gtest
-- Library mysqlserver depends on OSLIBS -lpthread;m;rt;crypt;dl;aio
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 12:47:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ make
.
..
...
Linking C static library libmysys.a
[ 15%] Built target mysys
Scanning dependencies of target dbug
[ 15%] Building C object dbug/CMakeFiles/dbug.dir/dbug.c.o
Linking C static library libdbug.a
[ 15%] Built target dbug
Scanning dependencies of target mysys_ssl
[ 15%] Building CXX object mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o
cc1plus: error: unrecognized command line option "-std=c++11"
make[2]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o] Error 1
make[1]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/all] Error 2
make: *** [all] Error 2

4 Fix -std=c++11 error.
This error is ude to old gcc c++. That is fixable by installing the CentOS devtools:

root@webscalesql-5.6.clean:[Mon Mar 31 12:47:25]:[/opt/installs/webscalesql-5.6]$ yum info  gcc-c++
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
base                                                                                                                                                                          | 3.7 kB     00:00     
extras                                                                                                                                                                        | 3.4 kB     00:00     
testing-1.1-devtools-6                                                                                                                                                        |  951 B     00:00     
updates                                                                                                                                                                       | 3.4 kB     00:00     
Installed Packages
Name        : gcc-c++
Arch        : x86_64
Version     : 4.4.7
Release     : 4.el6
Size        : 11 M
Repo        : installed
From repo   : base
Summary     : C++ support for GCC
root@webscalesql-5.6.clean:[Mon Mar 31 12:48:39][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ g++ --version
g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@webscalesql-5.6.clean:[Mon Mar 31 12:50:11][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cd /etc/yum.repos.d/
root@webscalesql-5.6.clean:[Mon Mar 31 12:50:30][/etc/yum.repos.d]$ wget  http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo
--2014-03-31 12:50:38--  http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo
Resolving people.centos.org... 204.15.73.242
Connecting to people.centos.org|204.15.73.242|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 174 [text/plain]
Saving to: “devtools-1.1.repo”

100%[===========================================================================================================================================================>] 174         --.-K/s   in 0s      

2014-03-31 12:50:39 (19.9 MB/s) - “devtools-1.1.repo” saved [174/174]

root@webscalesql-5.6.clean:[Mon Mar 31 12:50:39][/etc/yum.repos.d]$ yum repolist
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
testing-1.1-devtools-6                                                                                                                                                        |  951 B     00:00     
testing-1.1-devtools-6/primary                                                                                                                                                |  13 kB     00:00     
testing-1.1-devtools-6                                                                                                                                                                         42/42
repo id                                                                                   repo name                                                                                            status
base                                                                                      CentOS-6 - Base                                                                                      6,367
extras                                                                                    CentOS-6 - Extras                                                                                       14
testing-1.1-devtools-6                                                                    testing 1.1 devtools for CentOS 6                                                                       42
updates                                                                                   CentOS-6 - Updates                                                                                     713
repolist: 7,136

root@webscalesql-5.6.clean:[Mon Mar 31 12:51:08][/etc/yum.repos.d]$ yum install devtoolset-1.1

Install      19 Package(s)

Total download size: 85 M
Installed size: 303 M
Installed:
  devtoolset-1.1.noarch 0:1-13.el6                                                                                                                                                                   

Dependency Installed:
  devtoolset-1.1-binutils.x86_64 0:2.23.51.0.3-3.el6                 devtoolset-1.1-dwz.x86_64 0:0.7-1.el6                            devtoolset-1.1-elfutils.x86_64 0:0.154-6.el6                   
  devtoolset-1.1-elfutils-libelf.x86_64 0:0.154-6.el6                devtoolset-1.1-elfutils-libs.x86_64 0:0.154-6.el6                devtoolset-1.1-gcc.x86_64 0:4.7.2-5.el6                        
  devtoolset-1.1-gcc-c++.x86_64 0:4.7.2-5.el6                        devtoolset-1.1-gcc-gfortran.x86_64 0:4.7.2-5.el6                 devtoolset-1.1-gdb.x86_64 0:7.5.0.20120926-26.el6              
  devtoolset-1.1-libquadmath-devel.x86_64 0:4.7.2-5.el6              devtoolset-1.1-libstdc++-devel.x86_64 0:4.7.2-5.el6              devtoolset-1.1-oprofile.x86_64 0:0.9.7-6.el6                   
  devtoolset-1.1-runtime.noarch 0:1-13.el6                           devtoolset-1.1-systemtap.x86_64 0:1.8-8.el6                      devtoolset-1.1-systemtap-client.x86_64 0:1.8-8.el6             
  devtoolset-1.1-systemtap-devel.x86_64 0:1.8-8.el6                  devtoolset-1.1-systemtap-runtime.x86_64 0:1.8-8.el6              devtoolset-1.1-valgrind.x86_64 1:3.8.1-3.2.el6                 

Complete!
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:29][/etc/yum.repos.d]$ cd /opt/installs/webscalesql-5.6/
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:48][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y   
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:56][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 13:02:33][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ make 
..
Linking C static library libdbug.a
[ 15%] Built target dbug
[ 15%] Building CXX object mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o
cc1plus: error: unrecognized command line option "-std=c++11"
make[2]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o] Error 1
make[1]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/all] Error 2
make: *** [all] Error 2

After the install of devtoos you have to enable the devtoolset-1.1 bash:

root@webscalesql-5.6.clean:[Mon Mar 31 13:04:41][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ scl enable devtoolset-1.1 bash
[root@webscalesql-5 webscalesql-5.6]# export PS1="\u@\H:[\d \t]:[\w]$ "
root@webscalesql-5.6.clean:[Mon Mar 31 13:05:50]:[/opt/installs/webscalesql-5.6]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 13:06:04]:[/opt/installs/webscalesql-5.6]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 13:07:46]:[/opt/installs/webscalesql-5.6]$ make

Linking CXX executable my_safe_process
[100%] Built target my_safe_process
root@webscalesql-5.6.clean:[Mon Mar 31 13:36:58]:[/opt/installs/webscalesql-5.6]$ make install

The compile and the installation is done ! Finally .. 🙂
Done

5. Now, the post install:

root@webscalesql-5.6.clean:[Mon Mar 31 13:41:56]:[/opt/installs/webscalesql-5.6]$ cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql.server
root@webscalesql-5.6.clean:[Mon Mar 31 13:42:17]:[/opt/installs/webscalesql-5.6]$ mysql
bash: mysql: command not found

root@webscalesql-5.6.clean:[Mon Mar 31 13:42:36]:[/opt/installs/webscalesql-5.6]$ export PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/scripts
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:06]:[/opt/installs/webscalesql-5.6]$ mysql --version
mysql  Ver 14.14 Distrib 5.6.17, for Linux (x86_64) using readline 6.0
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:10]:[/opt/installs/webscalesql-5.6]$ cd /var/lib/mysql/
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:25]:[/var/lib/mysql]$ ls
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:26]:[/var/lib/mysql]$ chown -R mysql .
chown: invalid user: `mysql'
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:33]:[/var/lib/mysql]$ groupadd mysql
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:40]:[/var/lib/mysql]$ useradd -r -g mysql mysql
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:44]:[/var/lib/mysql]$ chown -R mysql .
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:48]:[/var/lib/mysql]$ mysql_install_db --basedir=/usr/local/mysql --user=mysql --datadir=/var/lib/mysql
Installing MySQL system tables...2014-03-31 13:43:58 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-03-31 13:43:58 25002 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-03-31 13:43:58 25002 [Note] InnoDB: The InnoDB memory heap is disabled
2014-03-31 13:43:58 25002 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-03-31 13:43:58 25002 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-03-31 13:43:58 25002 [Note] InnoDB: Using Linux native AIO
2014-03-31 13:43:58 25002 [Note] InnoDB: Not using CPU crc32 instructions
2014-03-31 13:43:58 25002 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-03-31 13:43:58 25002 [Note] InnoDB: Completed initialization of buffer pool
2014-03-31 13:43:58 25002 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2014-03-31 13:43:58 25002 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2014-03-31 13:43:58 25002 [Note] InnoDB: Database physically writes the file full: wait...
2014-03-31 13:43:59 25002 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2014-03-31 13:44:04 25002 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2014-03-31 13:44:09 25002 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2014-03-31 13:44:09 25002 [Warning] InnoDB: New log files created, LSN=45781
2014-03-31 13:44:09 25002 [Note] InnoDB: Doublewrite buffer not found: creating new
2014-03-31 13:44:09 25002 [Note] InnoDB: Doublewrite buffer created
2014-03-31 13:44:09 25002 [Note] InnoDB: 128 rollback segment(s) are active.
2014-03-31 13:44:09 25002 [Warning] InnoDB: Creating foreign key constraint system tables.
2014-03-31 13:44:09 25002 [Note] InnoDB: Foreign key constraint system tables created
2014-03-31 13:44:09 25002 [Note] InnoDB: Creating tablespace and datafile system tables.
2014-03-31 13:44:10 25002 [Note] InnoDB: Tablespace and datafile system tables created.
2014-03-31 13:44:10 25002 [Note] InnoDB: Waiting for purge to start
2014-03-31 13:44:10 25002 [Note] InnoDB: 5.6.17 started; log sequence number 0
2014-03-31 13:44:14 25002 [Note] Binlog end
2014-03-31 13:44:14 25002 [Note] unregister_replicator OK
2014-03-31 13:44:14 25002 [Note] InnoDB: FTS optimize thread exiting.
2014-03-31 13:44:14 25002 [Note] InnoDB: Starting shutdown...
2014-03-31 13:44:15 25002 [Note] InnoDB: Shutdown completed; log sequence number 1625977
OK

Filling help tables...2014-03-31 13:44:15 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-03-31 13:44:15 25024 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-03-31 13:44:15 25024 [Note] InnoDB: The InnoDB memory heap is disabled
2014-03-31 13:44:15 25024 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-03-31 13:44:15 25024 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-03-31 13:44:15 25024 [Note] InnoDB: Using Linux native AIO
2014-03-31 13:44:15 25024 [Note] InnoDB: Not using CPU crc32 instructions
2014-03-31 13:44:15 25024 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-03-31 13:44:15 25024 [Note] InnoDB: Completed initialization of buffer pool
2014-03-31 13:44:15 25024 [Note] InnoDB: Highest supported file format is Barracuda.
2014-03-31 13:44:15 25024 [Note] InnoDB: 128 rollback segment(s) are active.
2014-03-31 13:44:15 25024 [Note] InnoDB: Waiting for purge to start
2014-03-31 13:44:15 25024 [Note] InnoDB: 5.6.17 started; log sequence number 1625977
2014-03-31 13:44:15 25024 [Note] Binlog end
2014-03-31 13:44:15 25024 [Note] unregister_replicator OK
2014-03-31 13:44:15 25024 [Note] InnoDB: FTS optimize thread exiting.
2014-03-31 13:44:15 25024 [Note] InnoDB: Starting shutdown...
2014-03-31 13:44:17 25024 [Note] InnoDB: Shutdown completed; log sequence number 1625987
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

  /usr/local/mysql/bin/mysqladmin -u root password 'new-password'
  /usr/local/mysql/bin/mysqladmin -u root -h webscalesql-5.6.clean password 'new-password'

Alternatively you can run:

  /usr/local/mysql/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:

  cd . ; /usr/local/mysql/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl

  cd mysql-test ; perl mysql-test-run.pl

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /usr/local/mysql/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

WARNING: Default config file /etc/my.cnf exists on the system
This file will be read by default by the MySQL server
If you do not want to use this, either remove it, or use the
--defaults-file argument to mysqld_safe when starting the server


root@webscalesql-5.6.clean:[Mon Mar 31 13:44:17]:[/var/lib/mysql]$ chkconfig --list mysql.server
service mysql.server supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add mysql.server')

root@webscalesql-5.6.clean:[Mon Mar 31 13:45:31]:[/var/lib/mysql]$ chkconfig --add  mysql.server
root@webscalesql-5.6.clean:[Mon Mar 31 13:45:36]:[/var/lib/mysql]$ chkconfig --list mysql.server
mysql.server   	0:off	1:off	2:on	3:on	4:on	5:on	6:off

root@webscalesql-5.6.clean:[Mon Mar 31 13:47:53]:[/usr/local/mysql/support-files]$ service mysql.server status
MySQL is not running                                       [FAILED]
root@webscalesql-5.6.clean:[Mon Mar 31 13:48:04]:[/usr/local/mysql/support-files]$ 

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:04]:[/usr/local/mysql/support-files]$ service mysql.server start
Starting MySQL.                                            [  OK  ]
root@webscalesql-5.6.clean:[Mon Mar 31 13:48:13]:[/usr/local/mysql/support-files]$ service mysql.server status
MySQL running (25273)                                      [  OK  ]

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:17]:[/usr/local/mysql/support-files]$ mysql 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:26]:[/usr/local/mysql/support-files]$ mysql -h 127.0.0.1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.17 MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql root@127.0.0.1:[Mon Mar 31 13:49:21 2014][mysql]> select user,password,host from user;
+------+----------+-----------------------+
| user | password | host                  |
+------+----------+-----------------------+
| root |          | localhost             |
| root |          | webscalesql-5.6.clean |
| root |          | 127.0.0.1             |
| root |          | ::1                   |
|      |          | localhost             |
|      |          | webscalesql-5.6.clean |
+------+----------+-----------------------+
6 rows in set (0.00 sec)

mysql> set password=password('webscalesql');
Query OK, 0 rows affected (0.00 sec)

mysql> Bye
root@webscalesql-5.6.clean:[Mon Mar 31 13:51:17]:[/usr/local/mysql/support-files]$ nano /root/.my.cnf
root@webscalesql-5.6.clean:[Mon Mar 31 13:51:52]:[/usr/local/mysql/support-files]$ cat /root/.my.cnf 
[mysql]
user=root
password=webscalesql
prompt=mysql\_\u@\h:[\D][\d]>\_
host=127.0.0.1
root@webscalesql-5.6.clean:[Mon Mar 31 13:52:21]:[/usr/local/mysql/support-files]$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.17 MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql root@127.0.0.1:[Mon Mar 31 13:52:26 2014][(none)]> 

All that was done inspired by the SkySQL::Global Leaders in MariaDB & MySQL blog of Mr. Kolbe Kegel at WebScaleSQL! Will it build?

Slackware4Life!

Advertisements

Daily backup OpenStack single MySQL with Percona innobackupex including the /etc directory

October 17, 2013 1 comment

this is a short script for daily backup of the OpenStack MySQL and the /etc direcotry of the control node

[root@dev-epg-rhos-01 BACKUP]# cat /root/bin/epg.innobackupex.openstack.sh
#!/bin/sh
# done for epgmad4@tid.es
# Barcelona Thu Oct 17 16:30:36 CEST 2013

days_to_keep=7
NFS=/BACKUP
exportDate=`date +%Y-%m-%d.%H.%M.%S`
export_DIR=${NFS}/${HOSTNAME}.${exportDate}
test ! -d "${export_DIR}" && echo "$(date) : creating ${export_DIR}" && mkdir -p "${export_DIR}"
export_MySQL_DIR=${export_DIR}/mysql.bckp
export_ETC_DIR=${export_DIR}/etc.bckp
rsync -avh /etc ${export_ETC_DIR}
echo "=========================================================================================================" >> ${export_DIR}/README.restore.with.innobackupex
echo "HOW to restore this FULL mysql backup" >> ${export_DIR}/README.restore.with.innobackupex
echo "=========================================================================================================" >> ${export_DIR}/README.restore.with.innobackupex
echo "DO IT IN A SCREEN, as it involves some file movements:" >> ${export_DIR}/README.restore.with.innobackupex
echo "screen -S mysqlRestore" >> ${export_DIR}/README.restore.with.innobackupex
echo "service stop mysql" >> ${export_DIR}/README.restore.with.innobackupex
echo "ps aux | grep mysql" >> ${export_DIR}/README.restore.with.innobackupex
echo "rsync -avh /var/lib/mysql /var/lib/mysql.BAD" >> ${export_DIR}/README.restore.with.innobackupex
echo "rm -rf /var/lib/mysql" >>  ${export_DIR}/README.restore.with.innobackupex
echo "mkdir -p /var/lib/mysql && chown -R mysql:mysql /var/lib/mysql" >>  ${export_DIR}/README.restore.with.innobackupex
echo "innobackupex --copy-back ${export_MySQL_DIR}" >>  ${export_DIR}/README.restore.with.innobackupex
echo "=========================================================================================================" >> ${export_DIR}/README.restore.with.innobackupex
echo "more info at http://www.percona.com/doc/percona-xtrabackup/2.1/innobackupex/restoring_a_backup_ibk.html:" >>  ${export_DIR}/README.restore.with.innobackupex
cat /root/bin/README.restore.with.innobackupex >> ${export_DIR}/README.restore.with.innobackupex
innobackupex --ibbackup=xtrabackup --no-timestamp ${export_MySQL_DIR}
test $? -gt 0 && echo "$(date) : xtrabackup failed at ${export_MySQL_DIR}" && exit 0
innobackupex --apply-log ${export_MySQL_DIR}
find ${NFS}/ -daystart -maxdepth 1 -ctime +${days_to_keep} -type d -name "*${HOSTNAME}*" -exec /bin/rm -rf {} \;

#Slackware4Life!

the README.restore.with.innobackupex content:

cat /root/bin/README.restore.with.innobackupex
Restoring a Full Backup with innobackupex
For convenience, innobackupex has a –copy-back option,
which performs the restoration of a backup to the server’s datadir

$ innobackupex –copy-back /path/to/BACKUP-DIR
It will copy all the data-related files back to the server’s datadir,
determined by the server’s my.cnf configuration file.
You should check the last line of the output for a success message:
==================================================
=> innobackupex: Finished copying back files. <=
=> 111225 01:08:13 innobackupex: completed OK! <=
==================================================

Note The datadir must be empty; Percona XtraBackup innobackupex –copy-back option will not copy over existing files.
Also it’s important to note that MySQL server needs to be shut down before restore is performed.
You can’t restore to a datadir of a running mysqld instance (except when importing a partial backup).
As files’ attributes will be preserved, in most cases you will need to change the files’ ownership
to mysql before starting the database server, as they will be owned by the user who created the backup:

$ chown -R mysql:mysql /var/lib/mysql
Also note that all of these operations will be done as the user calling innobackupex,
you will need write permissions on the server’s datadir.

Public DNS resolve for VM instances hosted at OpenStack nova compute grizzly edition

July 25, 2013 1 comment

Issue – the OpenStack instances hostnames are not resolved at the moment by their floating IPs.

Fast solution – use PowerDNS
Short story:
1. Boot an instance , assign it a floating IP, open tcp ports 22 8001 and udp port 53.
2. Install PowerDNS, the particular yum package is named pdns.
3. install the mysql backend for PowerDNS, its available as yum package as well.
4. Install MariaDB , set the proper configuration, the initialize it at the default location.
5. Set the pdns user, create the pdns database and create the tables needed.
6. Configure the pdns service to use the gmysql backend.
7. Create and user at the OpenStack MySQL able to connect from the floating IP subnet.
8. Grant that user select permissions on all nova.* tables.
9. At the PowerDNS instance, set simple python / php / bash script to query the nova database and to inject the hostnames and the floating IPs as DNS records for the particular OpenStack domains at the pdns db at the pdns OpenStack instance.
In fact, what I do is to clean up all the available OpenStack DNS records, and to inject them again in one transaction, in a view to clean up the deleted and changed hostnames / floating IPs.
10. Set up the external DNS server to forward all queries related to the OpenStack domains to the PowerDNS floating UP.

In general, its good to start a python daemon, in a view to keep the recent status of the OpenStack DNS as a simple hash,
and only when there are changes in OpenStack instances / assigned floating IPs, to push the updates to the RowerDNS db backend.
I hate when the database is used in a lazy way.

All that can be done by hacking the nova python code, as PowerDNS has restful API as well,
but I know there are incoming new OpenStack modules which will do that.

The long story with examples and links TBD.

Update : the source code is here : https://github.com/seikath/openstack-external-dns

One more update to be pushed to the git source:
In a view to push the VM hostname changes without recreating the VM, I decided to use short_description field instead of the hostname,
as the nova rename instance command changes only that field, and the hostname remains intact.

The production code is changed and tested, just not pushed to the public repo.

Yum clean up / rollback RHEL 6 / CentOS packages to the initial status.

July 20, 2013 Leave a comment

In short, playing with RDO – the RedHat OpenStack install script /puppet based .. yummm / I needed several times to rollback the RHEL installed machines to their initial state.

So, in short, in a view to clean everything to the “after install” status :

# if you are brave heart, execute it right away 🙂 
yum history list all | sed  '/D, E/d;/Erase/d' | awk '/^ *[[:digit:]]/ {print $1}' | sed  '/^1$/d;' | while read crap; do echo yum history undo $crap -y;done
# check what has been done : 
yum history list all