Archive

Archive for the ‘AWS’ Category

Redis backup script

August 7, 2015 3 comments
#!/bin/bash 
## Fri, 07 Aug 2015 14:04:57 +0300 
## redis backup every 15 minutes 
## */15 * * * * redis.backup.maintenance.sh >> /var/log/redis.backup.log 2>&1 
## at /etc/rc.local : 
## test -d /var/run/redis.backup.lock.dir && rm -rf /var/run/redis.backup.lock.dir 
## watch the job: 
## tail -f /var/log/redis.backup.log 

#redis-cli LASTSAVE | awk '{print $1}' | { read gmt ; date "+%Y-%m-%d %H:%M:%S" -d "@$gmt" ; } 
# 2015-08-07 01:25:54 

lockf="/var/run/redis.backup.lock.dir"

# check for running script start 
if [ -d "${lockf}" ]
then
 echo "$(date +%Y-%m-%d.%H.%M.%S) : ${lockf} exists, exiting"
 exit 0
else
 mkdir "${lockf}" && echo "$(date +%Y-%m-%d.%H.%M.%S) : created lock at ${lockf}"
fi


echo "$(date +%Y-%m-%d.%H.%M.%S) : redis backup start"
echo "$(date +%Y-%m-%d.%H.%M.%S) : cleanup the /redis_backups and leave the last 6 backups"
find /redis_backups -maxdepth 1 -type f -name "dump.rdb.*" | sort -r | sed '7,$!d' | while read to_be_deleted; do rm -f ${to_be_deleted} && echo "$(date +%Y-%m-%d.%H.%M.%S) : deleted ${to_be_deleted}";done

last_save=$(redis-cli LASTSAVE | awk '{print $1}')
echo -n "$(date +%Y-%m-%d.%H.%M.%S) : executing redis-cli BGSAVE : "
redis-cli BGSAVE
while true
do
 if [ $(redis-cli LASTSAVE | awk '{print $1}') -eq ${last_save} ]
 then
 echo -n ". "
 sleep 2
 else
 echo ""
 echo "$(date +%Y-%m-%d.%H.%M.%S) : start ionice -c2 -n0 cp -vv /opt/redis/dump.rdb to /redis_backups/"
 ionice -c2 -n0 cp -vv /opt/redis/dump.rdb /redis_backups/dump.rdb.$(date +%Y-%m-%d.%H.%M.%S) && echo "$(date +%Y-%m-%d.%H.%M.%S) : backup comleted"
 break
 fi
done


if [ -d "${lockf}" ]
then
 echo "$(date +%Y-%m-%d.%H.%M.%S) : removing the lock"
 rm -rf "${lockf}"
fi
Advertisements
Categories: AWS, bash, NoSQL Tags: , ,

MySQL backup and cleanup bash scripts with mydumper

June 28, 2014 1 comment

1. Backup script

#!/bin/sh
# Fri Jun 27 10:44:49 2014
# done by dragkh
# usage: 
# cat /etc/cron.d/backupmysql 
# 0  3  *  *  *       root    /root/bin/clean.backup.hyperion.mysql.mydumper.daily.sh >>  /var/log/clean.backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1
# 35  3  *  *  *       root    /root/bin/backup.hyperion.mysql.mydumper.daily.sh >> /var/log/backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1

ROOT_BACKUP_DIR="/home/mydumper"

seik_date () {
if [ -z $1 ]
then
# cdate=`date +%Y-%m-%d\ %H:%M:%S\ %Z`; export cdate; echo $cdate
cdate=`date -R`; export cdate; echo $cdate
else

if [ -z ${2} ]
then
cdate=`date +%Y-%m-%d.%H.%M.%S`; export cdate; echo $cdate
else
cdate=`date "+%Y-%m-%d %H:%M:%S"`; export cdate; echo $cdate
fi

fi
}


function check_dir {
 test ! -d "${1}" && mkdir -p "${1}"
}


function set_cpu_threads {
    # set the threads one less than the existing 
    threads=$(cat /proc/cpuinfo  |  grep processor | tail -1 | awk '{print $3}')
    test $threads -lt 1 && threads=1
}

function dump_schema {
    mysqldump -d --dump-date --all-databases > ${DATA_DIR}/${HOSTNAME}.only.sql
}


function dump_data {
    echo "$(seik_date f) : executing : mydumper -o $DATA_DIR --long-query-guard 120 -r 100000 -c -e -m -L ${DATA_DIR}/mysql-backup.log -t ${threads} -v 3"
    mydumper -o $DATA_DIR --long-query-guard 120 -r 100000 -c -e -m -L ${DATA_DIR}/mysql-backup.log -t ${threads} -v 3
}

DATA_DIR="${ROOT_BACKUP_DIR}/$(seik_date d)"
check_dir "${DATA_DIR}" && echo "$(seik_date f) : ${DATA_DIR} is missing, creating it now .."
set_cpu_threads
echo "$(seik_date f) : star dumping the schema at ${DATA_DIR}.."
dump_schema && echo "$(seik_date f) : end dumping the schema at ${DATA_DIR} .."
echo "$(seik_date f) : start dumping the data at ${DATA_DIR} via ${threads} parallel threads .."
dump_data && echo "$(seik_date f) : end dumping the data at ${DATA_DIR} via ${threads} parallel threads .."

2. Clean up script keeping always backup directories intact

#!/bin/bash
# Sat Jun 28 03:16:38 EEST 2014
# done by dragkh
# usage: 
# cat /etc/cron.d/backupmysql
# 0       3       *       *       *       root    /root/bin/clean.backup.hyperion.mysql.mydumper.daily.sh >> /var/log/clean.backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1
# 35      3      *       *       *       root    /root/bin/backup.hyperion.mysql.mydumper.daily.sh >> /var/log/backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1

ROOT_BACKUP_DIR="/home/mydumper"

seik_date () {
if [ -z $1 ]
then
cdate=`date -R`; export cdate; echo $cdate
else

if [ -z ${2} ]
then
cdate=`date +%Y-%m-%d.%H.%M.%S`; export cdate; echo $cdate
else
cdate=`date "+%Y-%m-%d %H:%M:%S"`; export cdate; echo $cdate
fi

fi
}

day_limit=7;  
ls -t ${ROOT_BACKUP_DIR} | \
while read dir
do 
    ((dir_num++))
    test $dir_num -gt $day_limit && test -d "${ROOT_BACKUP_DIR}/${dir}" &&  rm -rf "${dir}" && echo "$(seik_date d) : removed [${dir_num}]::[${dir}]" && continue 
    test -d "${ROOT_BACKUP_DIR}/${dir}" && echo "$(seik_date d) : skipping [${dir_num}]::[${dir}]"
done

Compile at CentOS 6.5 the new MySQL webscalesql-5.6.17 branch by Facebook, Google, LinkedIn, and Twitter

March 31, 2014 Leave a comment

http://webscalesql.org/

yeah , big buzz around that one 🙂

So I decided to check the install process:

1. Clone the repo from

root@webscalesql-5.6.clean:[Mon Mar 31 11:37:11][~]$ cd /opt/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:15][/opt]$ mkdir installs
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:17][/opt]$ cd installs/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:19][/opt/installs]$ git clone https://github.com/webscalesql/webscalesql-5.6.git
Initialized empty Git repository in /opt/installs/webscalesql-5.6/.git/
remote: Counting objects: 30397, done.
remote: Compressing objects: 100% (12678/12678), done.
remote: Total 30397 (delta 18716), reused 27620 (delta 16936)
Receiving objects: 100% (30397/30397), 47.99 MiB | 460 KiB/s, done.
Resolving deltas: 100% (18716/18716), done.

2. Update the CentOs install just in case before to play with it:

root@webscalesql-5.6.clean:[Mon Mar 31 11:40:35][/opt/installs]$ yum update -y 

3. Compile it :
Note : The final part of the prompt here “[webscalesql-5.6.17]” id the actual git branch version.

root@webscalesql-5.6.clean:[Mon Mar 31 12:23:22][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-bash: cmake: command not found

3.1 No Cmake ? The CentOS install was done without cmake etc dev tools as at production they are not needed.
Now, we need it: Install cmake

root@webscalesql-5.6.clean:[Mon Mar 31 12:23:34][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install cmake -y 
....
Installed:
  cmake.x86_64 0:2.6.4-5.el6  
root@webscalesql-5.6.clean:[Mon Mar 31 12:24:10][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake --version
cmake version 2.6-patch 4                                                                                                                                                               

3.2 Try to compile it again?

root@webscalesql-5.6.clean:[Mon Mar 31 12:24:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Running cmake version 2.6.4
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error: your C compiler: "CMAKE_C_COMPILER-NOTFOUND" was not found.   Please set CMAKE_C_COMPILER to a valid compiler path or name.
CMake Error: your CXX compiler: "CMAKE_CXX_COMPILER-NOTFOUND" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.

No luck, the development tools are missing
3.3 Install ‘Development Tools’ group

root@webscalesql-5.6.clean:[Mon Mar 31 12:24:54][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum groupinstall 'Development Tools'
Installed:
  autoconf.noarch 0:2.63-5.1.el6      automake.noarch 0:1.11.1-4.el6      bison.x86_64 0:2.4.1-5.el6                       byacc.x86_64 0:1.9.20070509-7.el6   cscope.x86_64 0:15.6-6.el6           
  ctags.x86_64 0:5.8-2.el6            diffstat.x86_64 0:1.51-2.el6        doxygen.x86_64 1:1.6.1-6.el6                     flex.x86_64 0:2.5.35-8.el6          gcc.x86_64 0:4.4.7-4.el6             
  gcc-c++.x86_64 0:4.4.7-4.el6        gcc-gfortran.x86_64 0:4.4.7-4.el6   indent.x86_64 0:2.2.10-7.el6                     intltool.noarch 0:0.41.0-1.1.el6    libtool.x86_64 0:2.2.6-15.5.el6      
  patchutils.x86_64 0:0.3.1-3.1.el6   rcs.x86_64 0:5.7-37.el6             redhat-rpm-config.noarch 0:9.0.3-42.el6.centos   rpm-build.x86_64 0:4.8.0-37.el6     subversion.x86_64 0:1.6.11-10.el6_5  
  swig.x86_64 0:1.3.40-6.el6          systemtap.x86_64 0:2.3-4.el6_5     

Dependency Installed:
  apr.x86_64 0:1.3.9-5.el6_2              apr-util.x86_64 0:1.3.9-3.el6_0.1            cloog-ppl.x86_64 0:0.15.7-1.2.el6      cpp.x86_64 0:4.4.7-4.el6       gettext-devel.x86_64 0:0.17-16.el6      
  gettext-libs.x86_64 0:0.17-16.el6       kernel-devel.x86_64 0:2.6.32-431.11.2.el6    libart_lgpl.x86_64 0:2.3.20-5.1.el6    libgcj.x86_64 0:4.4.7-4.el6    libstdc++-devel.x86_64 0:4.4.7-4.el6    
  mpfr.x86_64 0:2.4.1-6.el6               neon.x86_64 0:0.29.3-3.el6_4                 pakchois.x86_64 0:0.4-3.2.el6          ppl.x86_64 0:0.10.2-11.el6     systemtap-client.x86_64 0:2.3-4.el6_5   
  systemtap-devel.x86_64 0:2.3-4.el6_5   

3.4 Try to compile … one more time 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:30:37][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Running cmake version 2.6.4
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at cmake/build_configurations/mysql_release.cmake:43 (MESSAGE):
        aio is required on Linux, you need to install the required library:

            Debian/Ubuntu:              apt-get install libaio-dev
            RedHat/Fedora/Oracle Linux: yum install libaio-devel
            SuSE:                       zypper install libaio-devel

          If you really do not want it, pass -DIGNORE_AIO_CHECK to cmake.
          
Call Stack (most recent call first):
  CMakeLists.txt:94 (INCLUDE)
-- Configuring incomplete, errors occurred!

So install libaio libs ..

root@webscalesql-5.6.clean:[Mon Mar 31 12:32:43][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install libaio-devel

Installed:
  libaio-devel.x86_64 0:0.3.107-10.el6                                                                                                                                                               

3.5 Try to compile … again
Now the CMakeCache.txt file has to be removed in a view to start form scratch:

root@webscalesql-5.6.clean:[Mon Mar 31 12:33:23][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:33:57][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
--
---
-- Could NOT find Curses  (missing:  CURSES_LIBRARY CURSES_INCLUDE_PATH)
CMake Error at cmake/readline.cmake:82 (MESSAGE):
  Curses library not found.  Please install appropriate package,

      remove CMakeCache.txt and rerun cmake.On Debian/Ubuntu, package name is libncurses5-dev, on Redhat and derivates it is ncurses-devel.
Call Stack (most recent call first):
  cmake/readline.cmake:125 (FIND_CURSES)
  cmake/readline.cmake:194 (FIND_SYSTEM_LIBEDIT)
  CMakeLists.txt:397 (MYSQL_CHECK_READLINE)
-- Configuring incomplete, errors occurred!

So follow up the hint and installed the libs:

root@webscalesql-5.6.clean:[Mon Mar 31 12:34:25][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install ncurses-devel

Installed:
  ncurses-devel.x86_64 0:5.7-3.20090208.el6                                                                                                                                                          

3.6 Try to compile … 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:35:44][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:35:51][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
...
....
.....
-- Found Curses: /usr/lib64/libcurses.so
-- Looking for tputs in /usr/lib64/libcurses.so
-- Looking for tputs in /usr/lib64/libcurses.so - found
-- Looking for include files HAVE_READLINE_HISTORY_H
-- Looking for include files HAVE_READLINE_HISTORY_H - not found.
CMake Error at cmake/readline.cmake:186 (MESSAGE):
  Cannot find libreadline!
Call Stack (most recent call first):
  cmake/readline.cmake:194 (FIND_SYSTEM_LIBEDIT)
  CMakeLists.txt:397 (MYSQL_CHECK_READLINE)
-- Configuring incomplete, errors occurred!

readline-devel was needed indeed even we installed ncurses libs

root@webscalesql-5.6.clean:[Mon Mar 31 12:37:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ yum install readline-devel

Installed:
  readline-devel.x86_64 0:6.0-4.el6                                                                                                                                                                  

3.7 Try to compile … 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:37:38][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:37:47][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Successfully downloaded http://googlemock.googlecode.com/files/gmock-1.6.0.zip to /opt/installs/webscalesql-5.6/source_downloads
CMake Error: Problem with tar_extract_all(): Invalid argument
CMake Error: Problem extracting tar: /opt/installs/webscalesql-5.6/source_downloads/gmock-1.6.0.zip
-- Library mysqlserver depends on OSLIBS -lpthread;m;rt;crypt;dl;aio
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6

3.8 Fix CMake Error: Problem with tar_extract_all(): Invalid argument at extracting the gmock-1.6.0.zip archive.

root@webscalesql-5.6.clean:[Mon Mar 31 12:46:05][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cd source_downloads/
root@webscalesql-5.6.clean:[Mon Mar 31 12:46:08][/opt/installs/webscalesql-5.6/source_downloads] [webscalesql-5.6.17]$ ls
gmock-1.6.0.zip
root@webscalesql-5.6.clean:[Mon Mar 31 12:46:08][/opt/installs/webscalesql-5.6/source_downloads] [webscalesql-5.6.17]$ unzip -a gmock-1.6.0.zip 

3.9 Try to compile it …. 🙂

root@webscalesql-5.6.clean:[Mon Mar 31 12:47:10][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 12:47:12][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
 -- GMOCK_SOURCE_DIR:/opt/installs/webscalesql-5.6/source_downloads/gmock-1.6.0
-- Performing Test HAVE_NO_NULL
-- Performing Test HAVE_NO_NULL - Failed
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS
-- Performing Test HAVE_NO_UNUSED_TYPEDEFS - Failed
-- GTEST_LIBRARIES:gmock;gtest
-- Library mysqlserver depends on OSLIBS -lpthread;m;rt;crypt;dl;aio
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 12:47:20][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ make
.
..
...
Linking C static library libmysys.a
[ 15%] Built target mysys
Scanning dependencies of target dbug
[ 15%] Building C object dbug/CMakeFiles/dbug.dir/dbug.c.o
Linking C static library libdbug.a
[ 15%] Built target dbug
Scanning dependencies of target mysys_ssl
[ 15%] Building CXX object mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o
cc1plus: error: unrecognized command line option "-std=c++11"
make[2]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o] Error 1
make[1]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/all] Error 2
make: *** [all] Error 2

4 Fix -std=c++11 error.
This error is ude to old gcc c++. That is fixable by installing the CentOS devtools:

root@webscalesql-5.6.clean:[Mon Mar 31 12:47:25]:[/opt/installs/webscalesql-5.6]$ yum info  gcc-c++
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
base                                                                                                                                                                          | 3.7 kB     00:00     
extras                                                                                                                                                                        | 3.4 kB     00:00     
testing-1.1-devtools-6                                                                                                                                                        |  951 B     00:00     
updates                                                                                                                                                                       | 3.4 kB     00:00     
Installed Packages
Name        : gcc-c++
Arch        : x86_64
Version     : 4.4.7
Release     : 4.el6
Size        : 11 M
Repo        : installed
From repo   : base
Summary     : C++ support for GCC
root@webscalesql-5.6.clean:[Mon Mar 31 12:48:39][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ g++ --version
g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@webscalesql-5.6.clean:[Mon Mar 31 12:50:11][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cd /etc/yum.repos.d/
root@webscalesql-5.6.clean:[Mon Mar 31 12:50:30][/etc/yum.repos.d]$ wget  http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo
--2014-03-31 12:50:38--  http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo
Resolving people.centos.org... 204.15.73.242
Connecting to people.centos.org|204.15.73.242|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 174 [text/plain]
Saving to: “devtools-1.1.repo”

100%[===========================================================================================================================================================>] 174         --.-K/s   in 0s      

2014-03-31 12:50:39 (19.9 MB/s) - “devtools-1.1.repo” saved [174/174]

root@webscalesql-5.6.clean:[Mon Mar 31 12:50:39][/etc/yum.repos.d]$ yum repolist
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
testing-1.1-devtools-6                                                                                                                                                        |  951 B     00:00     
testing-1.1-devtools-6/primary                                                                                                                                                |  13 kB     00:00     
testing-1.1-devtools-6                                                                                                                                                                         42/42
repo id                                                                                   repo name                                                                                            status
base                                                                                      CentOS-6 - Base                                                                                      6,367
extras                                                                                    CentOS-6 - Extras                                                                                       14
testing-1.1-devtools-6                                                                    testing 1.1 devtools for CentOS 6                                                                       42
updates                                                                                   CentOS-6 - Updates                                                                                     713
repolist: 7,136

root@webscalesql-5.6.clean:[Mon Mar 31 12:51:08][/etc/yum.repos.d]$ yum install devtoolset-1.1

Install      19 Package(s)

Total download size: 85 M
Installed size: 303 M
Installed:
  devtoolset-1.1.noarch 0:1-13.el6                                                                                                                                                                   

Dependency Installed:
  devtoolset-1.1-binutils.x86_64 0:2.23.51.0.3-3.el6                 devtoolset-1.1-dwz.x86_64 0:0.7-1.el6                            devtoolset-1.1-elfutils.x86_64 0:0.154-6.el6                   
  devtoolset-1.1-elfutils-libelf.x86_64 0:0.154-6.el6                devtoolset-1.1-elfutils-libs.x86_64 0:0.154-6.el6                devtoolset-1.1-gcc.x86_64 0:4.7.2-5.el6                        
  devtoolset-1.1-gcc-c++.x86_64 0:4.7.2-5.el6                        devtoolset-1.1-gcc-gfortran.x86_64 0:4.7.2-5.el6                 devtoolset-1.1-gdb.x86_64 0:7.5.0.20120926-26.el6              
  devtoolset-1.1-libquadmath-devel.x86_64 0:4.7.2-5.el6              devtoolset-1.1-libstdc++-devel.x86_64 0:4.7.2-5.el6              devtoolset-1.1-oprofile.x86_64 0:0.9.7-6.el6                   
  devtoolset-1.1-runtime.noarch 0:1-13.el6                           devtoolset-1.1-systemtap.x86_64 0:1.8-8.el6                      devtoolset-1.1-systemtap-client.x86_64 0:1.8-8.el6             
  devtoolset-1.1-systemtap-devel.x86_64 0:1.8-8.el6                  devtoolset-1.1-systemtap-runtime.x86_64 0:1.8-8.el6              devtoolset-1.1-valgrind.x86_64 1:3.8.1-3.2.el6                 

Complete!
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:29][/etc/yum.repos.d]$ cd /opt/installs/webscalesql-5.6/
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:48][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y   
root@webscalesql-5.6.clean:[Mon Mar 31 13:01:56][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 13:02:33][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ make 
..
Linking C static library libdbug.a
[ 15%] Built target dbug
[ 15%] Building CXX object mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o
cc1plus: error: unrecognized command line option "-std=c++11"
make[2]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/crypt_genhash_impl.cc.o] Error 1
make[1]: *** [mysys_ssl/CMakeFiles/mysys_ssl.dir/all] Error 2
make: *** [all] Error 2

After the install of devtoos you have to enable the devtoolset-1.1 bash:

root@webscalesql-5.6.clean:[Mon Mar 31 13:04:41][/opt/installs/webscalesql-5.6] [webscalesql-5.6.17]$ scl enable devtoolset-1.1 bash
[root@webscalesql-5 webscalesql-5.6]# export PS1="\u@\H:[\d \t]:[\w]$ "
root@webscalesql-5.6.clean:[Mon Mar 31 13:05:50]:[/opt/installs/webscalesql-5.6]$ rm CMakeCache.txt 
rm: remove regular file `CMakeCache.txt'? y
root@webscalesql-5.6.clean:[Mon Mar 31 13:06:04]:[/opt/installs/webscalesql-5.6]$ cmake . -DBUILD_CONFIG=mysql_release -DENABLE_DOWNLOADS=1
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/installs/webscalesql-5.6
root@webscalesql-5.6.clean:[Mon Mar 31 13:07:46]:[/opt/installs/webscalesql-5.6]$ make

Linking CXX executable my_safe_process
[100%] Built target my_safe_process
root@webscalesql-5.6.clean:[Mon Mar 31 13:36:58]:[/opt/installs/webscalesql-5.6]$ make install

The compile and the installation is done ! Finally .. 🙂
Done

5. Now, the post install:

root@webscalesql-5.6.clean:[Mon Mar 31 13:41:56]:[/opt/installs/webscalesql-5.6]$ cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql.server
root@webscalesql-5.6.clean:[Mon Mar 31 13:42:17]:[/opt/installs/webscalesql-5.6]$ mysql
bash: mysql: command not found

root@webscalesql-5.6.clean:[Mon Mar 31 13:42:36]:[/opt/installs/webscalesql-5.6]$ export PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/scripts
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:06]:[/opt/installs/webscalesql-5.6]$ mysql --version
mysql  Ver 14.14 Distrib 5.6.17, for Linux (x86_64) using readline 6.0
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:10]:[/opt/installs/webscalesql-5.6]$ cd /var/lib/mysql/
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:25]:[/var/lib/mysql]$ ls
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:26]:[/var/lib/mysql]$ chown -R mysql .
chown: invalid user: `mysql'
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:33]:[/var/lib/mysql]$ groupadd mysql
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:40]:[/var/lib/mysql]$ useradd -r -g mysql mysql
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:44]:[/var/lib/mysql]$ chown -R mysql .
root@webscalesql-5.6.clean:[Mon Mar 31 13:43:48]:[/var/lib/mysql]$ mysql_install_db --basedir=/usr/local/mysql --user=mysql --datadir=/var/lib/mysql
Installing MySQL system tables...2014-03-31 13:43:58 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-03-31 13:43:58 25002 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-03-31 13:43:58 25002 [Note] InnoDB: The InnoDB memory heap is disabled
2014-03-31 13:43:58 25002 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-03-31 13:43:58 25002 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-03-31 13:43:58 25002 [Note] InnoDB: Using Linux native AIO
2014-03-31 13:43:58 25002 [Note] InnoDB: Not using CPU crc32 instructions
2014-03-31 13:43:58 25002 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-03-31 13:43:58 25002 [Note] InnoDB: Completed initialization of buffer pool
2014-03-31 13:43:58 25002 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2014-03-31 13:43:58 25002 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2014-03-31 13:43:58 25002 [Note] InnoDB: Database physically writes the file full: wait...
2014-03-31 13:43:59 25002 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2014-03-31 13:44:04 25002 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2014-03-31 13:44:09 25002 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2014-03-31 13:44:09 25002 [Warning] InnoDB: New log files created, LSN=45781
2014-03-31 13:44:09 25002 [Note] InnoDB: Doublewrite buffer not found: creating new
2014-03-31 13:44:09 25002 [Note] InnoDB: Doublewrite buffer created
2014-03-31 13:44:09 25002 [Note] InnoDB: 128 rollback segment(s) are active.
2014-03-31 13:44:09 25002 [Warning] InnoDB: Creating foreign key constraint system tables.
2014-03-31 13:44:09 25002 [Note] InnoDB: Foreign key constraint system tables created
2014-03-31 13:44:09 25002 [Note] InnoDB: Creating tablespace and datafile system tables.
2014-03-31 13:44:10 25002 [Note] InnoDB: Tablespace and datafile system tables created.
2014-03-31 13:44:10 25002 [Note] InnoDB: Waiting for purge to start
2014-03-31 13:44:10 25002 [Note] InnoDB: 5.6.17 started; log sequence number 0
2014-03-31 13:44:14 25002 [Note] Binlog end
2014-03-31 13:44:14 25002 [Note] unregister_replicator OK
2014-03-31 13:44:14 25002 [Note] InnoDB: FTS optimize thread exiting.
2014-03-31 13:44:14 25002 [Note] InnoDB: Starting shutdown...
2014-03-31 13:44:15 25002 [Note] InnoDB: Shutdown completed; log sequence number 1625977
OK

Filling help tables...2014-03-31 13:44:15 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-03-31 13:44:15 25024 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-03-31 13:44:15 25024 [Note] InnoDB: The InnoDB memory heap is disabled
2014-03-31 13:44:15 25024 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-03-31 13:44:15 25024 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-03-31 13:44:15 25024 [Note] InnoDB: Using Linux native AIO
2014-03-31 13:44:15 25024 [Note] InnoDB: Not using CPU crc32 instructions
2014-03-31 13:44:15 25024 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-03-31 13:44:15 25024 [Note] InnoDB: Completed initialization of buffer pool
2014-03-31 13:44:15 25024 [Note] InnoDB: Highest supported file format is Barracuda.
2014-03-31 13:44:15 25024 [Note] InnoDB: 128 rollback segment(s) are active.
2014-03-31 13:44:15 25024 [Note] InnoDB: Waiting for purge to start
2014-03-31 13:44:15 25024 [Note] InnoDB: 5.6.17 started; log sequence number 1625977
2014-03-31 13:44:15 25024 [Note] Binlog end
2014-03-31 13:44:15 25024 [Note] unregister_replicator OK
2014-03-31 13:44:15 25024 [Note] InnoDB: FTS optimize thread exiting.
2014-03-31 13:44:15 25024 [Note] InnoDB: Starting shutdown...
2014-03-31 13:44:17 25024 [Note] InnoDB: Shutdown completed; log sequence number 1625987
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

  /usr/local/mysql/bin/mysqladmin -u root password 'new-password'
  /usr/local/mysql/bin/mysqladmin -u root -h webscalesql-5.6.clean password 'new-password'

Alternatively you can run:

  /usr/local/mysql/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:

  cd . ; /usr/local/mysql/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl

  cd mysql-test ; perl mysql-test-run.pl

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /usr/local/mysql/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

WARNING: Default config file /etc/my.cnf exists on the system
This file will be read by default by the MySQL server
If you do not want to use this, either remove it, or use the
--defaults-file argument to mysqld_safe when starting the server


root@webscalesql-5.6.clean:[Mon Mar 31 13:44:17]:[/var/lib/mysql]$ chkconfig --list mysql.server
service mysql.server supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add mysql.server')

root@webscalesql-5.6.clean:[Mon Mar 31 13:45:31]:[/var/lib/mysql]$ chkconfig --add  mysql.server
root@webscalesql-5.6.clean:[Mon Mar 31 13:45:36]:[/var/lib/mysql]$ chkconfig --list mysql.server
mysql.server   	0:off	1:off	2:on	3:on	4:on	5:on	6:off

root@webscalesql-5.6.clean:[Mon Mar 31 13:47:53]:[/usr/local/mysql/support-files]$ service mysql.server status
MySQL is not running                                       [FAILED]
root@webscalesql-5.6.clean:[Mon Mar 31 13:48:04]:[/usr/local/mysql/support-files]$ 

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:04]:[/usr/local/mysql/support-files]$ service mysql.server start
Starting MySQL.                                            [  OK  ]
root@webscalesql-5.6.clean:[Mon Mar 31 13:48:13]:[/usr/local/mysql/support-files]$ service mysql.server status
MySQL running (25273)                                      [  OK  ]

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:17]:[/usr/local/mysql/support-files]$ mysql 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

root@webscalesql-5.6.clean:[Mon Mar 31 13:48:26]:[/usr/local/mysql/support-files]$ mysql -h 127.0.0.1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.17 MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.


mysql root@127.0.0.1:[Mon Mar 31 13:49:21 2014][mysql]> select user,password,host from user;
+------+----------+-----------------------+
| user | password | host                  |
+------+----------+-----------------------+
| root |          | localhost             |
| root |          | webscalesql-5.6.clean |
| root |          | 127.0.0.1             |
| root |          | ::1                   |
|      |          | localhost             |
|      |          | webscalesql-5.6.clean |
+------+----------+-----------------------+
6 rows in set (0.00 sec)

mysql> set password=password('webscalesql');
Query OK, 0 rows affected (0.00 sec)

mysql> Bye
root@webscalesql-5.6.clean:[Mon Mar 31 13:51:17]:[/usr/local/mysql/support-files]$ nano /root/.my.cnf
root@webscalesql-5.6.clean:[Mon Mar 31 13:51:52]:[/usr/local/mysql/support-files]$ cat /root/.my.cnf 
[mysql]
user=root
password=webscalesql
prompt=mysql\_\u@\h:[\D][\d]>\_
host=127.0.0.1
root@webscalesql-5.6.clean:[Mon Mar 31 13:52:21]:[/usr/local/mysql/support-files]$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.17 MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql root@127.0.0.1:[Mon Mar 31 13:52:26 2014][(none)]> 

All that was done inspired by the SkySQL::Global Leaders in MariaDB & MySQL blog of Mr. Kolbe Kegel at WebScaleSQL! Will it build?

Slackware4Life!

Drupal install of commerce_kickstart-7.x-1.8-core at AWS on two PostgreSQL 9.1.4 replicated nodes with Nginx and php-fpm

July 22, 2012 Leave a comment

I decided to try Drupal on two PostgreSQL nodes replicated as master and readonly slave.
The usual choice for webserver is nginx

Follow the http://drupal.org/documentation/install and
http://drupal.org/project/commerce_kickstart
http://wiki.nginx.org/Drupal

# get the source and install it at the proper location like /usr/share/nginx/drupal
wget  http://ftp.drupal.org/files/projects/disable_messages-7.x-1.0.tar.gz

#set the proper nginx config 
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 15:36:27]:[/var/lib/pgsql]$ cat /etc/nginx/conf.d/drupal.conf
server {
        server_name darkstar.gotdns.org;
#        root /var/lib/pgsql9/xfs.data/drupal; 
        root /usr/share/nginx/drupal; ## <-- Your only path reference.
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        # This matters if you use drush
        location = /backup {
                deny all;
        }
 
        # Very rarely should these ever be accessed outside of your lan
        location ~* \.(txt|log)$ {
                allow 10.210.213.55/32;
                deny all;
        }
 
        location ~ \..*/.*\.php$ {
                return 403;
        }
 
        location / {
                # This is cool because no php is touched for static content
                try_files $uri @rewrite;
        }
 
        location @rewrite {
                # Some modules enforce no slash (/) at the end of the URL
                # Else this rewrite block wouldn't be needed (GlobalRedirect)
                rewrite ^/(.*)$ /index.php?q=$1;
        }
 
        location ~ \.php$ {
	        fastcgi_pass   127.0.0.1:9000;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                #NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
		#fastcgi_pass   unix:/var/run/php-fpm.sock;
        	fastcgi_index  index.php;
		fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;

        }
 
        # Fighting with ImageCache? This little gem is amazing.
        location ~ ^/sites/.*/files/imagecache/ {
                try_files $uri @rewrite;
        }
        # Catch image styles for D7 too.
        location ~ ^/sites/.*/files/styles/ {
                try_files $uri @rewrite;
        }
 
        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                expires max;
                log_not_found off;
        }
}

#check the nginx config :
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 15:42:02]:[/var/lib/pgsql]$ nginx -t -c /etc/nginx/nginx.conf
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

# and reload it
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 15:44:05]:[/var/lib/pgsql]$ service  nginx reload
Reloading nginx:  

# create pgsql user and db and set the pemirssions
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 11:55:33]:[~]$ createuser -U postgres -h 127.0.0.1 --pwprompt --encrypted --no-createrole --no-createdb drupal 
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 12:06:25]:[~]$ createdb -U postgres -h 127.0.0.1 --encoding=UTF8 --owner=drupal drupal

# set the psql user permissions at the slave with same password you set to the drupal user at the master:
root@ip-10-28-82-97-node-2:[Sun Jul 22 16:01:59]:[/var/lib/pgsql/data]$ cat  /var/lib/pgsql/data/pg_hba.conf | grep -i drupal
hostnossl       drupal          drupal             masterIP/32	 password

# ativate the user permissions 
root@ip-10-28-82-97-node-2:[Sun Jul 22 14:23:31]:[/var/lib/pgsql/data]$ service postgresql reload

#check the connection to slave:
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 16:03:33]:[/usr/share/nginx]$ echo "select version();" | psql -h slaveIP -U drupal drupal
Password for user drupal: 
                                                   version                                                   
-------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.1.4 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.4.6 20110731 (Red Hat 4.4.6-3), 64-bit
(1 row)


#start the install process hitting the index.php
#in case you get <strong>Error not found</strong>  change the permissions of the drupal root directory to 0755:   
chmod -R 0755  /usr/share/nginx/drupal

# install some of the php modules requested by the install as gd, dom and mbstring:
yum install php-soap php-mbstring php-xml  -y
#then restart the php-fpm 
root@domU-12-31-39-09-D2-CD-node-1:[Sun Jul 22 15:09:19]:[/var/lib/pgsql]$ service  php-fpm restart
Stopping php-fpm:                                          [  OK  ]
Starting php-fpm:                                          [  OK  ]

if you get an Warning message like this one:
Compilation failed: unknown option bit(s) set at offset 0 in user_validate_name() (line 645 of /usr/share/nginx/drupal/modules/user/user.module).

this is due to php PECL pcre extension update
there will be some more similar errors indicated at
+1606 /usr/share/nginx/drupal/includes/bootstrap.inc
+3726 /usr/share/nginx/drupal/includes/common.inc
+645 /usr/share/nginx/drupal/modules/user/user.module

the fastest way to fix it at AWS is to do yum update of pcre.x86_64

in general with AWS is better to do just yum update

then, everything works as expected,

do not forget to set the permissions of ./sites/default/settings.php to 0755 after the install

how to set the drupal to use the slave :
thank to padraig o’sullivan who posted
The settings.php file for the Drupal site needs to be updated to know about this slave database.

end of the story, now we have possible web cluster as the second node could be used for second web server, just some db connection settings have to be adjusted. and NFS with CacheFS for the web writable part of the drupal.

The drupal admin is owned by my daughter now … she is about to create the online store.

Bees with machine guns and EC2 at AWS

July 8, 2012 5 comments

I tried yesterday bees with machine guns tool after I read http://aws.amazon.com/releasenotes/0814290369059292

1. install bees as described at https://github.com/newsapps/beeswithmachineguns

2. Configuring EC2 credentials : http://code.google.com/p/boto/wiki/BotoConfig

Note : if you dont know what aws_access_key_id and aws_secret_access_key values are,
go to https://console.aws.amazon.com/iam/home and create at least one admin user
there you will get the values needed

create the .boto and .awsecret files :

dragkh@darkstar:[Sun Jul 08 16:04:31]:[~]$ echo "[Credentials]
aws_access_key_id = ${aws_access_key_id}
aws_secret_access_key = ${aws_secret_access_key}
[Boto]
ec2_region_name = eu-west-1
ec2_region_endpoint = ec2.eu-west-1.amazonaws.com
elb_region_name = eu-west-1
elb_region_endpoint = elasticloadbalancing.eu-west-1.amazonaws.com
" > ~/.boto

dragkh@darkstar:[Sun Jul 08 16:05:31]:[~]$ echo "${aws_access_key_id}
${aws_secret_access_key}
" > .awssecret

dragkh@darkstar:[Sun Jul 08 16:06:31]:[~]$ bees 
Usage: 
bees COMMAND [options]

Bees with Machine Guns

A utility for arming (creating) many bees (small EC2 instances) to attack
(load test) targets (web applications).

commands:
  up      Start a batch of load testing servers.
  attack  Begin the attack on a specific url.
  down    Shutdown and deactivate the load testing servers.
  report  Report the status of the load testing servers.

Note, you have to create Kay pair /named dragkh for example/ in the ec2_region set at .boto config
its good to have general security group ready, like allowing ssh from 0.0.0.0/0, for example named “allssh

then dragkh.pem file has to be properly chmoded as 0600 and placed in the user’s .ssh directory


dragkh@darkstar:[Sun Jul 08 17:06:31]:[~]$ bees up -s 4  -k dragkh   -i ami-instance-per.choice -z eu-west-1b -g allssh  -l ec2-user
Connecting to the hive.
Attempting to call up 4 bees.
Waiting for bees to load their machine guns...
.
.
.
.
.
Bee i-813c41c9 is ready for the attack.
.
Bee i-833c41cb is ready for the attack.
.
Bee i-853c41cd is ready for the attack.
.
Bee i-873c41cf is ready for the attack.
The swarm has assembled 4 bees.
dragkh@darkstar:[Sun Jul 08 18:06:31]:[~]$ bees report
Read 4 bees from the roster.
Bee i-813c41c9: running @ 176.34.79.154
Bee i-833c41cb: running @ 46.137.8.43
Bee i-853c41cd: running @ 54.247.142.79
Bee i-873c41cf: running @ 46.137.63.221

start 4 instances accessible with the key dragkh , used instance ami-instance-per.choice in zone eu-west-1b and implement security group allssh , default login ec2-user

how to check for real you are ready

# get one of the reported ips
dragkh@darkstar:[Sun Jul 08 18:10:31]:[~]$ ssh -i ~/ssh/dragkh.pem ec2-user@176.34.79.154
ssh: connect to host 176.34.79.154 port 22: Connection refused
# try again, its very possible the instance is still warming up ...

dragkh@darkstar:[Sun Jul 08 19:06:31]:[~]$ ssh -i ~/ssh/dragkh.pem ec2-user@176.34.79.154
The authenticity of host '176.34.79.154 (176.34.79.154)' can't be established.
RSA key fingerprint is ef:90:71:f4:f4:28:a7:ed:b0:61:a0:2e:e4:24:73:d4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '176.34.79.154' (RSA) to the list of known hosts.
Last login: Sun Jul  8 13:20:15 2012 from 33.pool85-61-183.dynamic.orange.es

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 5 security update(s) out of 33 total update(s) available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-10-228-169-103 ~]$ sudo su - 
# Ctrl+d to go back to your laptop
[root@ip-10-228-169-103 install]# logout
[ec2-user@ip-10-228-169-103 ~]$ logout
Connection to 176.34.79.154 closed.

dragkh@darkstar:[Sun Jul 08 21:06:31]:[~]$

start the swarm attack:


dragkh@darkstar:[Sun Jul 08 22:06:31]:[~]$ bees attack  -n 10000 -c 400 -u http://your.test.public.server/
Read 4 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 4 bees will fire 2500 rounds, 100 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 2 is joining the swarm.
Bee 3 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 2 is firing his machine gun. Bang bang!
Bee 3 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 0 is out of ammo.
Bee 2 is out of ammo.
Bee 1 is out of ammo.
Bee 3 is out of ammo.
Offensive complete.
     Complete requests:		10000
     Requests per second:	35.960000 [#/sec] (mean)
     Time per request:		11125.877250 [ms] (mean)
     50% response time:		6539.750000 [ms] (mean)
     90% response time:		25720.500000 [ms] (mean)
Mission Assessment: Swarm annihilated target.
The swarm is awaiting new orders.

in case you get some like Apparently your bees are peace-loving hippies.


dragkh@darkstar:[Sun Jul 08 23:06:31]:[~]$ bees attack  -n 10000 -c 100 -u http://your.test.public.server/
Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 5000 rounds, 50 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Offensive complete.
     2 of your bees didn't make it to the action. They might be taking a little longer than normal to find their machine guns, or may have been terminated without using "bees down".
     No bees completed the mission. Apparently your bees are peace-loving hippies.
The swarm is awaiting new orders.

its VERY possible the ab apache tool is missing from the bees instances
one solution is to log at them and to install it , for example with yum:

yum install httpd-tools -y

the other solution is, create an AMI with the installed ab tool and use it
I created one for me with ab apache benchmark tool and apache-jmeter-2.7 installed:
ami-c33d39b7 : 228348162977/bee-slave-jmeter

well, dont forget to shut down the swarm with bees down:


dragkh@darkstar:[Sun Jul 08 29:06:31]:[~]$ bees down 
Read 4 bees from the roster.
Connecting to the hive.
Calling off the swarm.
Stood down 4 bees.
dragkh@darkstar:[Sun Jul 08 29:06:31]:[~]$ bees report
No bees have been mobilized.

this is all, but please, as is written at https://github.com/newsapps/beeswithmachineguns

If you decide to use the Bees, please keep in mind the following important caveat: they are, more-or-less a distributed denial-of-service attack in a fancy package and, therefore, if you point them at any server you don’t own you will behaving unethically, have your Amazon Web Services account locked-out, and be liable in a court of law for any downtime you cause.

You have been warned.

Slackware4LiFe!

Categories: Apache, AWS, bash, Linux Tags: , ,

Compile and install PostgreSQL 9.2 beta 1 at AWS micro instance (EC2)

May 21, 2012 Leave a comment

first things first – take some time and read http://www.postgresql.org/about/news/1395/

Compilation:
get the sources at the install directory and compile it :


cd /opt/installs/
wget http://ftp.postgresql.org/pub/source/v9.2.0beta1/postgresql-9.2beta1.tar.bz2
tar xvf postgresql-9.2beta1.tar.bz2
# prepare the configure script:
vi postgresql-9.2beta1.seikath.conf
cat  /opt/installs/postgresql-9.2beta1.seikath.conf
#!/bin/sh
here=`pwd`

cd /opt/installs/postgresql-9.2beta1
./configure  \
--with-ossp-uuid \
--with-libxml \
--with-openssl \
--with-pam \
--with-krb5 \
--with-python \
--with-perl  \
--with-tcl  \
--with-pgport=6543 \

if you get this error

checking for flags to link embedded Perl... Can't locate ExtUtils/Embed.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
BEGIN failed--compilation aborted.
# you are missing some perl libraries
# install them:
yum install perl-YAML* -y
yum install perl-ExtUtils* -y

if you get this error

checking for library containing readline... no
configure: error: readline library not found
If you have readline already installed, see config.log for details on the
failure.  It is possible the compiler isn't looking in the proper directory.
Use --without-readline to disable readline support.
# you are missing readline libraries
# install them:
yum install readline* -y

if you get this error

checking for pam_start in -lpam... no
configure: error: library 'pam' is required for PAM
# you are missing pam libraries
# install them:
yum install pam-devel -y

if you get this error

checking for uuid_export in -lossp-uuid... no
checking for uuid_export in -luuid... no
configure: error: library 'ossp-uuid' or 'uuid' is required for OSSP-UUID
# you are missing uuid libraries
# install them:
yum install uuid-devel -y

if you get this error

checking for tclConfig.sh... no
configure: error: file 'tclConfig.sh' is required for Tcl
# you are missing tclConfig.sh
# install it:
yum install tcl-devel -y

if you get this error

configure: error: header file <Python.h> is required for Python
# you are missing Python libraries
# install them:
yum install python-devel -y

use gmake to compile as recommended

echo "gmake -j3 | tee /tmp/postgresql-9.2beta1.install.log" | at now 
tail -F /tmp/postgresql-9.2beta1.install.log
...

All of PostgreSQL successfully made. Ready to install.
echo "gmake -j3 install | tee -a /tmp/postgresql-9.2beta1.install.log" | at now
tail -F /tmp/postgresql-9.2beta1.install.log
...
PostgreSQL installation complete.

# DONE 🙂  
root@domU-12-31-39-07-39-AE-node-1:[Tue May 22 03:38:50]:[/opt/installs/postgresql-9.2beta1]$ /usr/local/pgsql/bin/psql -V
psql (PostgreSQL) 9.2beta1
contains support for command-line editing

Post install check of plproxy-2.4 on replicated PostgreSQL 9.1.3 at Amazon Web Services (EC2 , S3)

May 18, 2012 Leave a comment

I got an error on the post install check of plproxy:


root@domU-12-31-39-07-39-AE-node-1:[Fri May 18 18:37:24]:[/opt/installs/plproxy-2.4]$ make installcheck postgres
/usr/lib64/pgsql/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=. --psqldir='/usr/bin'   --dbname=regression --inputdir=test --dbname=regression plproxy_init plproxy_test plproxy_select plproxy_many plproxy_errors plproxy_clustermap plproxy_dynamic_record plproxy_encoding plproxy_split plproxy_target plproxy_sqlmed
(using postmaster on Unix socket, default port)
============== dropping database "regression"         ==============
DROP DATABASE
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test plproxy_init             ... ok
test plproxy_test             ... FAILED
test plproxy_select           ... ok
test plproxy_many             ... ok
test plproxy_errors           ... ok
test plproxy_clustermap       ... ok
test plproxy_dynamic_record   ... ok
test plproxy_encoding         ... FAILED (test process exited with exit code 2)
test plproxy_split            ... ok
test plproxy_target           ... ok
test plproxy_sqlmed           ... ok

=======================
 2 of 11 tests failed. 
=======================

The differences that caused some tests to fail can be viewed in the
file "/opt/installs/plproxy-2.4/regression.diffs".  A copy of the test summary that you see
above is saved in the file "/opt/installs/plproxy-2.4/regression.out".

make: *** [installcheck] Error 1

how to fix plproxy_test:
Edit the plproxy Make file and add to REGRESS_OPTS proper values in a view to use the correct user and host


root@domU-12-31-39-07-39-AE-node-1:[Fri May 18 19:10:12]:[/opt/installs/plproxy-2.4]$ fgrep REGRESS_OPTS  Makefile
REGRESS_OPTS = --dbname=regression --inputdir=test --user=postgres --host=127.0.0.1
# now we have :

root@domU-12-31-39-07-39-AE-node-1:[Fri May 18 19:05:51]:[/opt/installs/plproxy-2.4]$ make installcheck 
/usr/lib64/pgsql/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=. --psqldir='/usr/bin'   --dbname=regression --inputdir=test --user=postgres --host=127.0.0.1 --no-locale --dbname=regression plproxy_init plproxy_test plproxy_select plproxy_many plproxy_errors plproxy_clustermap plproxy_dynamic_record plproxy_encoding plproxy_split plproxy_target plproxy_sqlmed
(using postmaster on 127.0.0.1, default port)
============== dropping database "regression"         ==============
DROP DATABASE
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test plproxy_init             ... ok
test plproxy_test             ... ok
test plproxy_select           ... ok
test plproxy_many             ... ok
test plproxy_errors           ... ok
test plproxy_clustermap       ... ok
test plproxy_dynamic_record   ... ok
test plproxy_encoding         ... FAILED (test process exited with exit code 2)
test plproxy_split            ... ok
test plproxy_target           ... ok
test plproxy_sqlmed           ... ok

=======================
 1 of 11 tests failed. 
=======================