Shell CheatSheet :Languages: :PROPERTIES: :type: language :export_file_name: cheatsheet-shell-A4.pdf :END:

#+BEGIN_HTML

#+END_HTML

PDF Link: [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/cheatsheet-shell-A4.pdf][cheatsheet-shell-A4.pdf]], Category: [[https://cheatsheet.dennyzhang.com/category/languages][languages]]
Blog URL: https://cheatsheet.dennyzhang.com/cheatsheet-shell-A4
Related posts: [[https://cheatsheet.dennyzhang.com/cheatsheet-vim-A4][Vim CheatSheet]], [[https://github.com/topics/denny-cheatsheets][#denny-cheatsheets]]

File me [[https://github.com/dennyzhang/cheatsheet.dennyzhang.com/issues][Issues]] or star [[https://github.com/dennyzhang/cheatsheet.dennyzhang.com][this repo]]. ** Basic | Name | Comment | |----------------------------------------+-----------------------------------------------------------------------| | Redirect stdout/stderr | =ls /tmp >/dev/null 2>&1= | | Deal with filename | =basename $f=, =dirname $f= | | Use timeout: avoid command hang | timeout 10 sh -c 'ls -lt' | | Restart shell without killing terminal | =exec -l $SHELL= | | Run sub-shell | =echo $BASH_SUBSHELL; ( echo "Running in subshell: $BASH_SUBSHELL" )= | | Set pipefail | =set -ueoxv pipefail=, =set +ueoxv pipefail= | | Shell match regexp | =echo $str, then grep "$regexp"= | | Shell match regexp | =expr match "$date" "^[0-9]{8}" >/dev/null && echo yes= | | Run static code check | [[https://www.dennyzhang.com/shellcheck][Link: shellcheck]] | | Show date in utc | =date -u= | | Check file type of a given type | =file /etc/hosts= | | Check command type of a command | =type echo= | ** Shell script | Name | Comment | |----------------------------------------+---------------------------------------------| | Trap exit signal | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/trap-exit.sh][code/trap-exit.sh]] | | Shell retry | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/shell-retry.sh][code/shell-retry.sh]] | | Check if a string contains a substring | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/string-contains.sh][code/string-contains.sh]] | | Check if a string in a list | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/string-in-list.sh][code/string-in-list.sh]], [[https://stackoverflow.com/questions/8063228/how-do-i-check-if-a-variable-exists-in-a-list-in-bash][Link: stackoverflow]] | | Log with timestamp | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/log-with-timestamp.sh][code/log-with-timestamp.sh]] | | Quit if current user is not root | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/assert-user-root.sh][code/assert-user-root.sh]] | | Set -x on fly | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/restore-debug-output.sh][code/restore-debug-output.sh]] | | Shell run curl check | [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/curl-test.sh][code/curl-test.sh]] | ** Environment variables | Name | Comment | |------------------------------------------------+------------------------------------| | List all environment variables | =export= | | Define a new env variable | export NAME1="value1" | | Define a variable with default value of others | export MYVAR="${MYVAR:-$OTHERVAR}" | ** iterm | Name | Comment | |---------------+---------| | Delete a word | Ctrl+w | ** zsh | Name | Comment | |-------------------------------+-----------------------------------------------------| | Disable all zsh's autocorrect | In ~/.zshrc, =unsetopt correct_all= | | Disable a given autocorrect | In ~/.zshrc, alias ssh='nocorrect ssh'. [[https://github.com/dennyzhang/cheatsheet-shell-A4/blob/master/code/restore-debug-output.sh][zsh_disable]] | ** GNU tools *** Echo string | Name | Comment | |----------------------+---------------------------------------------------------| | Echo red text | echo -e "hello,\e[0;31m there \e[0;31m" | | Echo multiple lines | echo -e "hello,\ndenny" | | Echo bold text | echo -e hello, "\033[1mThis is bold text.\033[0m" | | Echo underlined text | echo -e hello, "\033[4mThis is underlined text.\033[0m" | ** Shell Basic *** cd | Name | Comment | |--------------------------+---------------------------------------------------------| | Go to given folder | =cd /var/log/= | | Go to folder in subshell | =(cd /var/log/ && ls)= After this, PWD won't be changed | | Go to home | =cd= | | Go to parent folder | =cd ..= | | Go to previous folder | =cd -= | #+BEGIN_HTML #+END_HTML *** Numeric | Name | Comment | |------+------------------------------------------------| | * | =expr 5 * 4= | | + | =let z=x+y, z=$x+$y= | | == | =int1 -eq int2=, =[ $? -eq 0 ] && echo "good"= | | >= | =int1 -ge =int2= | | > | =int1 -gt =int2= | | <= | =int1 -le =int2= | | < | =int1 -lt =int2= | | != | =int1 -ne =int2= | *** xargs #+BEGIN_SRC sh

Run grep for files filtered by find

find /var/log -name "*.log" | xargs grep -i error

Loop with pipes

cat /etc/passwd | awk -F':' '{print $1}' | xargs -I{} sudo -l -U {} | grep -v "not allowed to" #+END_SRC ** Scripts

Compare command output #+BEGIN_SRC sh [ 0 -eq $(find ./data -name "*.txt" -type f -print | wc -l) ] #+END_SRC
get ip from eth0 #+BEGIN_SRC sh /sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}' #+END_SRC ** More Resources License: Code is licensed under [[https://www.dennyzhang.com/wp-content/mit_license.txt][MIT License]]. #+BEGIN_HTML

<img align="bottom"src="https://www.dennyzhang.com/wp-content/uploads/sns/github.png" alt="github" /> #+END_HTML

org-mode configuration :noexport: #+STARTUP: overview customtime noalign logdone showall #+DESCRIPTION: #+KEYWORDS: #+LATEX_HEADER: \usepackage[margin=0.6in]{geometry} #+LaTeX_CLASS_OPTIONS: [8pt] #+LATEX_HEADER: \usepackage[english]{babel} #+LATEX_HEADER: \usepackage{lastpage} #+LATEX_HEADER: \usepackage{fancyhdr} #+LATEX_HEADER: \pagestyle{fancy} #+LATEX_HEADER: \fancyhf{} #+LATEX_HEADER: \rhead{Updated: \today} #+LATEX_HEADER: \rfoot{\thepage\ of \pageref{LastPage}} #+LATEX_HEADER: \lfoot{\href{https://github.com/dennyzhang/cheatsheet-shell-A4}{GitHub: https://github.com/dennyzhang/cheatsheet-shell-A4}} #+LATEX_HEADER: \lhead{\href{https://cheatsheet.dennyzhang.com/cheatsheet-shell-A4}{Blog URL: https://cheatsheet.dennyzhang.com/cheatsheet-shell-A4}} #+AUTHOR: Denny Zhang #+EMAIL: [email protected] #+TAGS: noexport(n) #+PRIORITIES: A D C #+OPTIONS: H:3 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc #+EXPORT_EXCLUDE_TAGS: exclude noexport #+SEQ_TODO: TODO HALF ASSIGN | DONE BYPASS DELEGATE CANCELED DEFERRED #+LINK_UP: #+LINK_HOME:
--8<-------------------------- separator ------------------------>8-- :noexport:
[#A] Difficulties in shell/bash :noexport:Coding:Personal: :PROPERTIES: :type: Linux_Language :END:

[[https://www.dennyzhang.com/wp-content/uploads/2014/04//blog_bash.png]] | Item | Summary | |----------------------------------------------------------------+-------------------------------------------------------------| | ${cmds}/$(cmd) | 在当前shell中执行命令/在子shell中执行命令 | | ${x}y/$xy | 将变量x连上字符y/变量xy | | $*/$@ | 当某个输入变量含空格时, 两者是不同的 | |----------------------------------------------------------------+-------------------------------------------------------------| | type | 查看命令的类型(内置命令,别名,函数,可执行文件等) | | (sh /tmp/1.sh ; echo "error code $?") >/tmp/1.log | 追加sh运行后的$?于log文件 | | read -e variablename | 读取用户输入 | | eval __value="$$env" | eval命令在处理命令行时,先执行所有的shell替换,然后执行命令行 | | /bin/sh -xe /tmp/hudson366585559507073478.sh | echo shell | | declare -a PRE_ROLLING_SCRIPT=cat $output_file | 申明变量 | ** 测试操作 -- test :noexport:

test命令会修改$?的值

| Item | Summary | |---------------------------+-----------------------| | test -z str | 判断字符串str是否为空 | | str1 = str2 | str1是否与str2相同 | | -d | 是否一个目录 | | -e | 文件是否存在 | | [ "" == "$diff_content" ] | |

[ 75 == svn info ./manifests/localnet/setting.pp.sample | grep Revision | awk -F': ' '{print $2}' ] *** 示例代码 #+BEGIN_EXAMPLE myPath="/var/log/httpd/" myFile="/var /log/httpd/access.log"

#这里的-x 参数判断$myPath是否存在并且是否具有可执行权限 if [ ! -x "$myPath"]; then mkdir "$myPath" fi

#这里的-d 参数判断$myPath是否存在 if [ ! -d "$myPath"]; then mkdir "$myPath" fi

#这里的-f参数判断$myFile是否存在 if [ ! -f "$myFile" ]; then touch "$myFile" fi

#其他参数还有-n,-n是判断一个变量是否是否有值 if [ ! -n "$myVar" ]; then echo "$myVar is empty" exit 0 fi

#两个变量判断是否相等 if [ "$var1" = "$var2" ]; then echo '$var1 eq $var2' else echo '$var1 not eq $var2' fi #+END_EXAMPLE *** shell中常用系统变量和条件判断 http://blog.chinaunix.net/u/28814/showart_1422464.html\ #+begin_example shell中常用系统变量和条件判断常用系统变量 $0 当前程序的名称 $n 当前程序的第n个参数,n=1,2,...9 $* 当前程序的所有参数(不包括程序本身) $# 当前程序的参数个数(不包括程序本身) $$ 当前程序的PID $! 执行上一个子进程的PID $? 执行上一个指令的返回值条件判断:expression为字符串操作 -n str 字符串str是否不为空 -z str 字符串str是否为空 str1 =str2 str1是否与str2相同 str1!=str2 str1是否与str2不同条件判断:expression为整数操作 expr1 -a expr2 如果 expr1 和 expr2 评估为真,则为真 expr1 -o expr2 如果 expr1 或 expr2 评估为真,则为真条件判断:expression为bool操作 int1 -eq int2 如果int1等于int2,则为真 int1 -ge int2 如果int1大于或等于int2,则为真 int1 -gt int2 如果int1大于int2 ,则为真 int1 -le int2 如果int1小于或等于int2 ,则为真 int1 -lt int2 如果int1小于int2 ,则为真 int1 -ne int2 如果int1不等于int2 ,则为真条件判断:expression为文件操作 -b 是否块文件 -p 文件是否为一个命名管道 -c 是否字符文件 -r 文件是否可读 -d 是否一个目录 -s 文件的长度是否不为零 -e 文件是否存在 -S 是否为套接字文件 -f 是否普通文件 -x 文件是否可执行,则为真 -g 是否设置了文件的 SGID 位 -u 是否设置了文件的 SUID 位 -G 文件是否存在且归该组所有 -w 文件是否可写,则为真 -k 文件是否设置了的粘贴位 -t fd fd 是否是一个与终端相连的打开的文件描述符(fd 默认为 1) -O 文件是否存在且归该用户所有 #+end_example ** 任务操作 -- Job :noexport: | 命令 | 含义 | |-------------+--------------------------------------------------------------| | bg | 启动被终止的后台作业 | | fg | 将后台作业调到前台 | | jobs | 列出所有正在运行的作业 | | kill | 向指定作业发送kill信号 | | ^Z(Ctrl-Z) | 终止(挂起)作业.屏幕上将出现提示符 | |-------------+--------------------------------------------------------------| | stop | 挂起一个后台作业 | | stty tostop | 当一个后台作业向终端发送输出时就挂起它 | | wait[n] | 等待一个指定的作业并返回它的退出状态,这里n是一个PID或作业号 |

jobs命令的参数 | jobs命令的参数 | 含义 | |----------------+----------------------| | %n | 作业号n | | %string | 以string开头的作业名 | | %?string | 作业名包含string | | %% | 当前作业 | | %+ | 当前作业 | | %- | 当前作业前的一个作业 | | -r | 列出所有运行的作业 | | -s | 列出所有挂起的作业 | *** basic use #+begin_example
shell的作业控制: C-z, bg, fg sleep 1000 #起一个前端作业 ^Z # 将前端作业转到后台 jobs -l #查看作业列表 bg #启动被终止的后台作业 jobs -l #查看作业列表 fg #将后台作业调到前台

jobs -r:列出所有运行的作业 jobs -s:列出所有暂停的作业 bg %2:启动后台运行中的第二个作业 #+end_example *** DONE 特殊变量:$!最后一个后台作业ID, $?上一条命令的退出值, $$当前shell的PID CLOSED: [2010-03-04 星期四 13:02] | 变量 | 含义 | | '$' | 当前shell的PID | | '-' | 当前的sh选项设置 | | '?' | 已执行的上一条命令的退出值 | | '!' | 最后一个进入后台的作业的PID | #+begin_src sh echo The pid of this shell is $$ echo The options for this shell are $- grep ls /etc/passwd echo $? sleep 25 & echo $! #+end_src ** # --8<-------------------------- separator ------------------------>8-- ** [question] Difference between the two shell commands: load_builtin_options VS $(load_builtin_options) ** [question] shell中all-servers不是合法变量名? *** console shot: :noexport: #+begin_example ,----------- /sshx:[email protected]:/home/zhangwei/hudsonecae/health_check/parse_conf.sh | all-servers="127.0.0.1" | ecae-servers="127.0.0.1" | embeded-ecae-services="ecae router webapp mongo memcache scm" | mysql-server="127.0.0.1" # TODO: what if multiple zookeeper servers `-----------

///c0f958ce1a270dc022d431edf2eff580#$./ecae-health-check.sh [2012-03-28 06:50:41] Parse configuration ./parse_conf.sh: line 14: all-servers=127.0.0.1: command not found ./parse_conf.sh: line 15: ecae-servers=127.0.0.1: command not found ./parse_conf.sh: line 16: embeded-ecae-services=ecae router webapp mongo memcache scm: command not found ./parse_conf.sh: line 17: mysql-server=127.0.0.1: command not found ./parse_conf.sh: line 19: zk-servers=127.0.0.1: command not found #+end_example ** [question] [ test=ab ]; echo $? 输出为什么为0 ** [question] 如何在shell文件内部,将stdout重定向 ** [question] 解释下面umask的行为 #+begin_example ///c98c8cb7008ad33b849ed394ba96520d#$umask 0; mkdir /tmp/usr/local/ecae-health-check/report/t7 ///c98c8cb7008ad33b849ed394ba96520d#$ls -lt /tmp/usr/local/ecae-health-check/report/ total 36 drwxrwxrwx 2 root root 4096 Jul 12 11:32 t7 -rw-rw-rw- 1 root root 0 Jul 12 11:32 t5 ----rw-rw- 1 root root 0 Jul 12 11:32 t4 ---------- 1 root root 0 Jul 12 11:31 t3 d--------- 2 root root 4096 Jul 12 11:31 t2 d--------- 2 root root 4096 Jul 12 11:30 tt #+end_example ** [question] 下面ls命令输出之间的间隔符是什么 #+begin_example ///60b852829888cc1252b5498d239e247c#$ls /tmp/elmar client data deps ebin if include priv rel src ///60b852829888cc1252b5498d239e247c#$ls /tmp/elmar > ./test ///60b852829888cc1252b5498d239e247c#$cat ./test client data deps ebin if include priv rel src ///60b852829888cc1252b5498d239e247c#$ #+end_example ** [question] shell将如下代码定义成函数 #+begin_src sh while [ -z "$all_servers" ] ; do echo -ne "Enter the ip list for all nodes in the system, separated by whitespace. Sample: 192.168.75.101 192.168.75.102\n> " read -e all_servers update_option_in_conf "$System_file" "all_servers" "$all_servers" done #+end_src ** [question] 解释下面shell执行的行为 #+begin_example /sshx:[email protected]: #$ command='ps -ef | grep "run_erl.*elmar" | grep -v grep | awk -F''' ''' '''{print $2}''' | xargs lsof -p | wc -l' /sshx:[email protected]: #$ echo $command ps -ef | grep "run_erl.*elmar" | grep -v grep | awk -F' ' '{print $2}' | xargs lsof -p | wc -l /sshx:[email protected]: #$ ps -ef | grep "run_erl.*elmar" | grep -v grep | awk -F' ' '{print $2}' | xargs lsof -p | wc -l 12 /sshx:[email protected]: #$ command /sshx:[email protected]: #$ $(command) /sshx:[email protected]: #$ echo $? 0 /sshx:[email protected]: #$ echo $command ps -ef | grep "run_erl.*elmar" | grep -v grep | awk -F' ' '{print $2}' | xargs lsof -p | wc -l #+end_example ** # --8<-------------------------- separator ------------------------>8-- ** Common misunderstood -- 常见疑点 *** ${x}与$x的区别: ${x}y表示将变量x的值连接上字符y, $xy表示变量xy http://unix-school.blogspot.in/2011/07/difference-between-x-and-x.html\ *** $*与$@的区别 *** Internal commands VS External commands http://unix-school.blogspot.in/2012/03/internal-vs-external-commands.html\

Internal commands are something which is built into the shell.
When an external command has to be executed, a new process has to be spawned and the command gets executed. | Item | Comment | |----------------------------------------------------+----------| | get the list of Internal commands | help | | find out whether a command is internal or external | type cat | #+begin_src bash

For performance issue, better use internal commands, compared to external commands

Say to add 2 numbers say x & y:

z=expr $x+$y # Not good

for a given list, return a new list with items re-arranged by random algorithm

function generate_random_list() { local lists=${1?} local lists_ret array count lists_ret="" while [ ! -z "$lists" ] do array=($lists) count=${#array[*]} # item counts of the list index=$((RANDOM % $count)) item=${array[$index]} lists=echo $lists | sed 's/'$item'//g' # remove the chosen item lists=${lists%% } # remove tailing whitespace lists_ret=$lists_ret" "$item #echo "count:"$count"index:"$index" item:"$item" lists:"$lists" lists_ret:"$lists_ret done echo $lists_ret } org_list="ab cd ef hb ed af" new_list=$(generate_random_list "$org_list") echo -e "org_list:"$org_list"\nnew_list:"$new_list #+end_src ** 目录相关操作 -- Directory | Item | Summary | |--------------------+------------------------------------------------------------------| | cd - | 回退当上一次的目录位置 | | (cd ./deps; ls af) | 进入目录后, 运行shell某个命令, 如果失败, 那么不修改当前的目录 | ** 文件操作 -- File | Item | Summary | |--------------------------------------------------------+----------------------------------| | sudo bash -c "echo hello >>/tmp/hosts" | 向权限不够的文件尾部追加一些内容 | | echo "test" ｜ tee -a /test.log | 向权限不够的文件尾部追加一些内容 | | (cat /etc/hosts; echo "append") ｜ sudo tee ./test.log | 向权限不够的文件尾部追加一些内容 | *** DONE globbing文件名替换:ls[!f-z]??? :IMPORTANT: CLOSED: [2010-03-04 星期四 11:44] globbing就是将元字符展开为文件名的过程.

ls .bak ls a?c? ls[!f-z]??? ls f{oo,aa,umble} ** 字符串操作 -- string | 表达式 | 功能 | |---------------+------------------------------------------------------------| | ${变量%模式} | 将变量值的尾部与模式进行最小匹配,并将匹配的部分删除 | | ${变量%%模式} | 将变量值的尾部与模式进行最大匹配,并将匹配的部分删除 | | ${变量#模式} | 将变量值的头部与模式进行最小匹配,并将匹配的部分删除 | | ${变量##模式} | 将变量值的头部与模式进行最大匹配,并将匹配的部分删除 | | ${#变量} | 替换为变量中的字符个数.如果是或@,长度则是位置参量的个数 | *** string compare #+BEGIN_EXAMPLE http://www.linux286.com/index.php/action_viewthread_tid_15502.html\ shell中常用系统变量和条件判断 #+end_example *** 示例代码 #+begin_src sh pathname="/usr/bin/local/bin" echo ${pathname%/bin*} # /usr/bin/local echo ${pathname%%/bin*} # /usr/

pathname=/home/liliput/jake/.bashrc echo ${pathname#/home} # /liliput/jake/.bashrc

pathname=/home/liliput/jake/.bashrc echo ${pathname##*/} # .bashrc

name="Ebenezer Scrooge" echo ${#name} # 16 ;; -------------------------- separator --------------------------

An IP range is like "10.32.182.181-184" or just a single IP address

function __util_parse_ip_range() { local range=${1?} local start_ip network start end

if [[ $range == - ]]; then start_ip=${range%-} end=${range##-} util_check_ip start_ip && util_check_num end || { log_error "Invalid IP range '$range'" return 1 }

network=${start_ip%.} start=${start_ip##.} util_check_num start || { log_error "Invalid IP range '$range'" return 1 }

((start <= end)) || { log_error "Invalid IP range '$range': end number '$end' is less"
"than start number '$start'" return 1 }

local x for ((x = start; x <= end; x++)); do echo -n "$network.$x " done else util_check_ip range || { log_error "Invalid IP range '$range' - neither a range nor an"
"single address" return 1 } echo -n "$range " fi

return 0 } #+end_src *** TODO 从下面CONARY_NEW_VERSION变量中提取出1.2.3.32049 CONARY_NEW_VERSION=/maui.lss.emc.com@emc:atmos-1.2-auto/1.2.3.32049-1-1 *** DONE bash strip whitespace: ${var##( )} CLOSED: [2016-04-15 Fri 07:47] http://www.cyberciti.biz/faq/bash-remove-whitespace-from-string/ str2=$(echo "${str}" |sed -e 's/^[ \t]//g') str2=$(echo "${str}" |sed -e 's/[ \t]*$//g')

output=" This is a test" output="${output##( )}" echo "=${output}=" ** 随机 -- random | Item | Summary | |-----------------------+------------------------| | echo $((RANDOM % 10)) | 产生指定区间内的随机数 | ** shell的位置参量 -- 命令行参数 | 位置参量 | 指代对象 | |----------+-----------------------------------------------| | $0 | 脚本名 | | $# | 位置参量的个数 | | $ | 所有的位置参量 | | $@ | 未加双引号时,与$含义相同 | | "$" | 扩展为单个变量(例如: "$1 $2 $3" | | "$@" | 扩展为多个单独的变量(例如:"$1", "$2", "$3") | | $1...$9 | 最多可引用9个位置参量 |

带参数的set命令将重置位置参量. 位置参量一旦被重置,原来的参量列表就会丢失
要想清除所有的位置参量,可使用set --
比较$与$@的异同 *** 拿到命令行参数 #+begin_src sh version=${1:-"1.4.0"} mkdir -p atmos-$version newest_applance=$(ls -Xl atmos-auto-$version |tail -n 1 |awk '{print $9}') if [[ ! -f atmos-$version/$newest_applance ]] then ln $newest_applance atmos-$version/ fi echo $newest_applance |cut -d'-' -f3 #+end_src *** DONE shell: 怎样通过命令行参数把含有回车符的字符串传给python程序程序中的\是可以转换了.而传入的参数中的\是不转义的.如果你使用repr()看一下它们的内部表示就知道了:

对于传入的"a\nb\n"在python运行时其实是"a\nb\n",与内部的"a\nb\n"是不同的. **** code :noexport: #+BEGIN_SRC -t python ttt.py $'a\nb\nc'

;; -------------------------- separator -------------------------- [root@mail ~]# A="a

b c " [root@mail ~]# python ttt.py $A a b c a a ;; -------------------------- separator -------------------------- [root@mail ~]# python ttt.py "a b c " a b c a b c

a b c ;; -------------------------- separator -------------------------- #+END_SRC **** useful link http://bbs.chinaunix.net/archiver/tid-913541.html\ 怎样通过命令行参数把含有回车符"\n"的字符串传给python程序 *** DONE shell中的if是指命令成功执行为真,即$?为0 CLOSED: [2010-03-15 星期一 11:44] #+BEGIN_EXAMPLE if fun; then if [[ $(fun) -ne 1 ]]; then ;; -------------------------- separator --------------------------

如下代码的输出为 here2 here4 here7 ;; -------------------------- separator --------------------------

function fun() { return 1 } UPGRADE_SKIP_INSTALL="true" if $UPGRADE_SKIP_INSTALL; then if fun; then echo "here1" else echo "here2" fi else echo "here3" fi

if [[ $(fun) -ne 1 ]]; then echo "here4" else echo "here5" fi

if [[ $(fun) == 1 ]]; then echo "here6" else echo "here7" fi #+end_example *** DONE shell的shift命令 CLOSED: [2010-02-26 星期五 15:37] shift命令指定参数时,会将参量列表左移指定的次数. 没有给定参数时,shift命令仅把参量列表左移一次. 一旦列表被移动,左端那个参数就被永远删除了. ** 变量扩展修饰符 :IMPORTANT:HARD: http://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion

${EDITOR:-/bin/vi}

修饰符首先提供一个简单的条件测试,用来检查某个变量是否已经被设置,然后根据测试结果给变量赋一个值. | 修饰符 | 值 | |----------------------------+-------------------------------------------------------------------------------------------------------------------------------------| | ${variable:-word} | 如果变量variable已被设置且非空,则代入它的值.否则,代入word | | ${variable:=word} | 已被设置且值非空,就代入它的值.否则,将variable的值设为word.始终代入varialble的值.位置参量不能用这种方式赋值. | | ${varialble:+word} | 如果变量variable已被设置且值非空,代入word.否则,什么都不代入(代入空值) | | ${varialble:?word} | 如果变量variable已被设置且值非空,就代入它的值.否则,输出word并且从shell退出.如果省略了word,就会显示信息:parameter null or not set | | ${varialble:offset} | 获得变量variable值中位置从offset开始的子串,偏移为从0到串的末尾. | | ${varialble:offset:length} | 获得变量variable值中位置从offset开始长度为length的子串. |

echo $EDITOR

echo ${EDITOR:-/bin/vi}

local timeout=${1?"timeout required"} # in second *** DONE shell语法:local extra_tr=${@:-""} 当参数为空时,赋予默认值 CLOSED: [2010-03-04 星期四 15:11] *** DONE shell:local ip=${1?} 参数的非空检查 CLOSED: [2010-03-04 星期四 15:10] **** codesnippet #+begin_src sh __mail_subject_prefix="${1?'mail subject prefix required'}" ;; -------------------------- separator -------------------------- function util_is_ip() { local ip=${1?}

[[ $ip =~ ^<0-9>{1,3}.<0-9>{1,3}.<0-9>{1,3}.<0-9>{1,3}$ ]] return $? } #+end_src ** 重复执行命令(Event Designators,命令指示符): | Item | Summary | |-------------+--------------------------------------------------------| | !! | 重复执行最近一条命令 | | !str | 重复执行最近一条以str开头的命令 | | !?str? | 重复执行最近一条包含str的命令 | | !N | 重复执行第N条命令 | | !-N | 重复执行倒数第N条命令 | | ^str1^str2^ | 重复执行最近一条命令,但将命令中的字符串str1替换为str2 | ** 示例代码 *** 遍历文件夹 #+begin_src sh function ergodic(){ for file in ls $1 do if [ -d $1"/"$file ] then ergodic $1"/"$file else echo $1"/"$file fi done } INIT_PATH="/etc/mysql" ergodic $INIT_PATH #+end_src *** 检查脚本运行用户(是否为root) #+begin_src sl check_run_use() { if [ "$UID" -ne "$ROOT_UID"] then echo "please su to root" exit 0 fi } #+end_src *** 检查字符串是否为字母 #+begin_src sl is_alpha() { [ $# -eq 1 ] || return $FAILURE

case $1 in
*[!a-zA-Z]*|"") return $FAILURE
*) return $SUCCESS
esac

} #+end_src *** 把传递字符串修改为小写 #+begin_src sl to_lower() { if [ -z "$1" ] then echo "(null)" return fi echo "$@" | tr A-Z a-z return } #+end_src *** 完整模板示例 :noexport: #+begin_src sh #filename #author : tito #email : [email protected] #since : 2009-1-11 #modify : 2009-1-12 #deploy : 124.74.193.211 #function : clean log dirtory

log_dir="/var/log/httpd" this_date=date +'%Y%m%d' back_file_name="back."this_date".tar.gz" #clean up dir clean_up_dir(){ #$1 is enter dirctory if [ -d "$1" ] then echo "$1 is not a dictory" exit 0 fi cd $1 #back up the log file tar -zxvf $back_file_name $1/* cp $back_file_name /opt #rm the log file in is dictory rm -f "$1/*" return 0 }

clean_up_dir() $log_dir

exit 0 #+end_src *** 检查参数 #+begin_src sl #check enter args usage () { if [ -z "$1" ] then msg=filename else msg=$@ fi

echo "`basename	$0`"$msg""

} #+end_src *** DONE shell 如何打印出标准错误输出后, 退出 CLOSED: [2011-01-27 Thu 20:37] #+begin_src sh out=$(ssh_do $master_node_ip $cmd || { log_error " Failed to sync time of maui nodes." return 1 } ) #+end_src *** DONE [#A] shell中, 如何向文件中添加一行, 如果该行已经存在, 则不添加 CLOSED: [2011-10-13 Thu 16:43] #+BEGIN_EXAMPLE

保证./test.cfg中, key1的值为value1, 如果key1不存在, 则追加一行key1 = value1 (grep "key1 +=" ./test.cfg && sudo sed -i 's/key1 +=.*/key1 = value1/' ./test.cfg) || ((cat ./test.cfg; echo "key1 = value1") | sudo tee ./test.cfg)

--8<-------------------------- §separator§ ------------------------>8--

sudo sed -i 's/xmodmap ~/.xmodmap 2>/dev/null//' /tmp/hosts

echo xmodmap ~/.xmodmap 2>/dev/null | sudo tee /tmp/hosts

sudo sed -i 's/xmodmap ~/.xmodmap 2>/dev/null//' /tmp/hosts #+end_example *** DONE shell中function使用返回值 - return/echo CLOSED: [2010-03-04 星期四 11:09] increment(){ sum=expr $1 + 1 return $sum }

echo -n "This sum is " increment 5

echo $? echo $sum

变量sum虽然是在函数increment中定义的,但它的作用域却是全局的. 因而可以在调用函数的脚本中被识别. *** 逻辑与的示例代码 #+begin_src sh if test -d /home -a -d /tmp2 ; then echo "two directories exist" else echo "don't exist" fi; #+end_src ** basic use *** DONE 单引号内不被解释,双引号内会被解释 CLOSED: [2010-03-04 星期四 13:33] #+BEGIN_EXAMPLE 单引号能保护所有元字符不被解释. 双引号允许对它所括的内容进行变量替换和命令替换.

echo Don't you need '$5.00?' echo 'Mother yelled, "Time to eat!"'

name=Jody echo "Hi $name, I'm glad to meet you!" echo "Hey $name, the time is $(date)"

创建脚本时,脚本的第一行通常称为shbang(#!)行. 当脚本启动后,UNIX内核检查文件的第一行以决定将要执行的程序类型. #+end_example *** DONE shell调试:set -x开启echo, sh -n解释不执行命令 CLOSED: [2010-03-04 星期四 15:39] | 命令 | 选项 | 含义 | | sh -x scriptname | Echo选项 | 在变量替换之后,执行之前显示脚本的每一行 | | sh -v scriptname | Verbose选项 | 执行之前显示脚本的每一行,与键入脚本中的一样 | | sh -n scriptname | Noexec选项 | 解释但不执行命令 | | set -u | 未绑定变量 | 尚未设置的标志变量 | | set -x | 开启echo | 跟踪脚本执行 | | set +x | 关闭echo | 关闭跟踪 | ** useful link http://www.hsrl.rutgers.edu/ug/shell_help.html\ How to write a shell script http://www.freeos.com/guides/lsst/\ Linux Shell Scripting Tutorial v1.05r3 http://www.cyberciti.biz/tips/how-to-generating-print-range-sequence-of-numbers.html#comments\ Shell Scripting: Generate or Print Range of Numbers ( Sequence of Numbers for Loop ) ** DONE Shell: Argument list too long CLOSED: [2011-01-27 Thu 22:58] #+begin_example http://www.linuxjournal.com/article/6060\ "Argument list too long": Beyond Arguments and Limitations | Linux Journal http://richmegginson.livejournal.com/11296.html\ richmegginson: Fun with bourne shell: Argument list too long http://en.kioskea.net/faq/1086-unable-to-delete-file-argument-list-too-long\ Unable to delete file: language.org http://www.unix.com/shell-programming-scripting/25666-argument-list-too-long-shell-error.html\ Argument list too long - Shell error - The UNIX and Linux Forums #+end_example ** CANCELED Why below shell script fail :noexport: CLOSED: [2011-03-02 Wed 13:34] To trigger the recovery proces, please run below command for each SS disk in this node: mauisvcmgr -s mauicc -c trigger_cc_rcvrtask -a 'queryStr=:,act=DiskRecover,taskId='

replace with the name of the node. replace <diskID> with the id of the SS disk -- the id is order of the output "grep mauiss /etc/fstab", starting from 1. replace with a random UUID -- you can generate it from uuidgen.

Denny, could you help to write a shell script to automate this step? Thanks. *** code snippet :noexport: #!/bin/sh

Trigger cc recovery task in all SS disk of current node

current_hostname=hostname disk_id=1

Iterate each ss disk

ss_disk_count=grep mauiss /etc/fstab | wc -l for ss_disk in $(seq $ss_disk_count); do random_uuid=uuidgen echo "mauisvcmgr -s mauicc -c trigger_cc_rcvrtask -a 'queryStr=$current_hostname:$disk_id,act=DiskRecover,taskId=$random_uuid'" mauisvcmgr -s mauicc -c trigger_cc_rcvrtask -a 'queryStr=$current_hostname:$disk_id,act=DiskRecover,taskId=$random_uuid' disk_id=expr $disk_id + 1 echo "=========" done echo "Finish of the script" ** shell function sample #+begin_src sh #!/bin/sh srv_name=${1?"service name required"} node_name=${2?"node name required"} srv_port=${3?"service port required"} function create_zk_node() { local path=${1?} local data=${2?} zk_tool -c query -p $path || zk_tool -c create -p $path -d $data } create_zk_node "/services" "service" create_zk_node "/services/${srv_name}" "service" create_zk_node "/services/${srv_name}/instances" "instances" create_zk_node "/services/${srv_name}/global_conf" "global_conf" create_zk_node "/services/${srv_name}/global_conf/port" $srv_port create_zk_node "/services/${srv_name}/instances/${node_name}" "instance" create_zk_node "/services/${srv_name}/instances/${node_name}/instance_conf" "instance" #+end_src ** DONE [#A] sample: shell on-line help CLOSED: [2012-03-28 Wed 11:49] *** console shot: :noexport: #+begin_example ///52f4795d9c60ad26bf18beed5c43d11b#$cat /usr/local/bin/thrift_client #!/bin/bash

PWD=/usr/local/sanity/lib/sanity-1 NODE_NAME="$(basename $0 .sh)" VERSION=1.0

check() { exec erl -noshell -noinput -pa "$PWD/ebin"
+fnu
+K true
-setcookie random
-s sanity_checker $1 $2 $3 }

usage() { echo "${NODE_NAME}: missing argument Try `${NODE_NAME} --help' for more information." exit 0 }

help() { cat <<EOF ${NODE_NAME} version ${VERSION} Usage: ${NODE_NAME} [ -s|--server server ] [ -p|--port port ] [ -c|--command command ]

sanity check for those services which implemented ecae thrift service interface Mandatory arguments to long options are mandatory for short options too. -s, --server server ip address or hostname [default = localhost] -p, --port server port [default = 9105] -c, --command thrift interface method [default = sanityCheck] -v, --version output version information -h, --help display this help Example: thrift_client -c sanityCheck -s 192.168.75.108 -p 9200 EOF exit 0 }

ARGS=getopt -a -o s:p:c:vh -l server:,port:,command:,version,help -- "$@"

[ $? -ne 0 ] && usage

eval set -- "${ARGS}"

while true do case "$1" in -s|--server) server="$2" shift ;; -p|--port) port="$2" shift ;; -c|--command) command="$2" shift ;; -v|--version) echo "${NODE_NAME} ver. ${VERSION}" shift exit 0 ;; -h|--help) help shift exit 0 ;; --) shift break ;; esac shift done

[ "x" = "x$command" ] && command=sanityCheck [ "x" = "x$server" ] && server=localhost [ "x" = "x$port" ] && port=9105

check $command $server "$port" #+end_example ** # --8<-------------------------- §separator§ ------------------------>8-- ** TODO shell将如下代码定义成函数 #+begin_src sh while [ -z "$all_servers" ] ; do echo -ne "Enter the ip list for all nodes in the system, separated by whitespace. Sample: 192.168.75.101 192.168.75.102\n> " read -e all_servers update_option_in_conf "$System_file" "all_servers" "$all_servers" done #+end_src ** TODO shell为什么没有方便的自增 ///0352ea91def77fd8e5d305f770ca1f70#$failure=0 ///0352ea91def77fd8e5d305f770ca1f70#$failure=expr $failure + 1 ** TODO Linux exec command http://www.linuxjournal.com/content/bash-redirections-using-exec\ Bash Redirections Using Exec | Linux Journal *** code src/mgmt/tools/pgdbsetup #+BEGIN_SRC -t sub closeMongrelFd { my @res = lsof -i TCP:3000 | grep mongrel | awk '{print \$4}' | sed 's/[a-zA-Z]//'; my $mongrelFd = $res[0]; chomp ($mongrelFd); if($mongrelFd) { my $cmd = "exec $mongrelFd>&-"; print $cmd; $cmd } } #+END_SRC *** code /tmp/tmp:

apples pears bananas pleaches plums #+BEGIN_SRC -t #!/bin/sh

Scriptname: speller

Purpose: Check and fix spelling errors in a file

exec < /tmp/tmp while read line do echo $line echo -n "Is this word correct? [ Y/N]" read answer < /dev/tty case "$answer" in [Yy]* ) continue;; *) echo "What is the correct spelling?" read word < /dev/tty sed "s/$line/$word/g" /tmp/tmp >error mv error tmp echo $line has been changed to $word. esac done #+END_SRC *** TODO Shell: Exec命令 #+BEGIN_EXAMPLE 使用exec命令,不需要创建子shell,就能打开或关闭标准输入和标准输出.

exec命令常被用来打开文件(根据文件名或文件描述符)以供读写.注意,文件描述符0,1和2已预留给标准输入标准输出和标准错误输出.文件打开后,将得到下一个可用的文件描述符. #+end_example ** TODO shell中. ./1.sh与sh ./1.sh的区别是什么 ** CDPATH: 对于访问经常去的目录,　节省cd输入目录路的时间 ** How to find the length of a variable http://unix-school.blogspot.in/2010_04_01_archive.html\\ | Item | Summary | |--------------------------------------------------------+---------| | echo ${#VAR} | | | echo -n $VAR ｜ wc -c | | | printf $VAR ｜ wc -c | | | expr $VAR : '.*' | | | echo $VAR ｜ awk '{print length ;}' | | | echo $VAR ｜ perl -ne 'chop; print length($_) . "\n";' | | ** [#A] shell的启动配置文件执行顺序 :IMPORTANT: http://zhidao.baidu.com/question/366827280.html #+begin_example ~/.bash_profile 是交互式login 方式进入 bash 运行的 ~/.bashrc 是交互式 non-login 方式进入 bash 运行的 shell的分类: 交互式登录shell:以一个用户的身份登陆进去,输入命令的格式的shell 读取配置文件的顺序: /etc/profile-->/etc/profile.d/-->~/.bash_profile-->~/.bashrc-->/etc/bashrc 非交互式登录shell:只在一个脚本的,不需要用户登录,自动运行读取配置文件的顺序: /.bashrc-->/etc/bashrc-->/etc/profile.d/ #+end_example ** TODO shell输出多行内容时的重定向 ** TODO shell hook函数, 在返回时调用一个方法 ** su命令和su -命令的区别 http://www.ha97.com/4001.html #+begin_example su命令和su -命令最大的本质区别就是:前者只是切换了root身份,但Shell环境仍然是普通用户的Shell;而后者连用户和Shell环境一起切换成root身份了.只有切换了Shell环境才不会出现PATH环境变量错误.su切换成root用户以后,pwd 一下,发现工作目录仍然是普通用户的工作目录;而用su -命令切换以后,工作目录变成root的工作目录了.用echo $PATH命令看一下su和su -以后的环境变量有何不同.以此类推,要从当前用户切换到其它用户也一样,应该使用su -命令 #+end_example ** [ -e /etc/*.conf ] 是不正确的,如果是判断满足某个pattern的文件是否存在,需要用find -name来查找 ** [#A] bash执行带单引号的命令, $command方式是不行的,需要用eval $command :IMPORTANT: ** Here document 是 shell 的一个非常常用的功能: << EOF ... EOF http://ylinux.org/246 #+begin_example 如何结束字符用引号括起来,那么 Here 文档中的变量是不会被替换的: $ cat > /tmp/t_here.txt << "EOF" UID=$UID EOF $ cat /tmp/t_here.txt UID=$UID

如果结束字符没有用引号括起来:

$ cat > /tmp/t_here.txt << EOF UID=$UID EOF $ cat /tmp/t_here.txt UID=1000

#+end_example ** 判断数据类型为int #+begin_src sh ,----------- | @@ -91,7 +91,7 @@ | print_testcase_info "$testcase_name" "$testcase_desc" | show_output="t" | result=$(remote_command $hostname "ps -ef | grep '$process_regex' | grep -v grep | awk -F' ' '{print $2}' | xargs ps -o vsz= -p") | - if [ $? -ne 0 ]; then | + if [ $? -ne 0 ] || [ -z "${result##[!0-9]}" ]; then | log_failure "$scenario_name" "$testcase_name"
| "Output:($result), Fail to get memory for ($process_regex) in ($hostname).\n\n" | else `----------- #+end_src ** [#B] on-line help usage #+begin_src sh #!/bin/bash ##-------------------------------------------------------------------

@copyright 2013

File : xzb_update_user_html.sh

Author : filebat [email protected]

Description : Update posts info to mysql

--

Created : <2013-01-31>

Updated: Time-stamp: <2013-01-31 20:08:58>

##------------------------------------------------------------------- . ./utility_xzb.sh

BIN_NAME="$(basename $0 .sh)"

function update_user_html() { user_dir=${1?"user website directory is required"} userid=${2?"userid is required"}

## TODO automatically get date, since this may not be mandotary
date=${3?"date is required"}
index_html="$user_dir/$(echo $date | tr -d -).html"

python_script="import jinja_html; jinja_html.generate_list_user_post(\"$userid\", \"$date\", \"$index_html\")"

command="(cd $XZB_HOME/code/smarty_html; python -c '${python_script}')"
eval $command
if [ $? -ne 0 ]; then
    log "[$BIN_NAME.sh] Generate $index_html failed."
    exit 1
else
    log "[$BIN_NAME.sh] Generate $index_html is done."
fi

python_script="import jinja_html; jinja_html.generate_user_all_posts(\"$userid\", \"$date\", \"$user_dir\")"
command="(cd $XZB_HOME/code/smarty_html; python -c '${python_script}')"
eval $command
if [ $? -ne 0 ]; then
    log "[$BIN_NAME.sh] Generate html files of user posts failed."
    exit 1
else
    log "[$BIN_NAME.sh] Generate html files of user posts is done."
fi

}

ensure_variable_isset ensure_is_root

VERSION=0.1

help() { cat <<EOF Usage: ${BIN_NAME} [OPTION]

Sample: sudo xzb_update_user_html.sh --user denny --date 2013-01-24 --vhostdir /home/wwwroot/denny.youwen.im +----------------------+ +------------------------------+ | | | | | generate index html +---+ generate html files of posts + | | | | +----------------------+ +------------------------------+

${BIN_NAME} is a shell script to generate a user's html files for a given date

Optional arguments: -h, --help display this help --user username --date date --vhostdir root directory for the vhost -v, --version output version information EOF exit 0 }

ARGS=getopt -a -o hv -l user:,date:,vhostdir:,version,help -- "$@" [ $? -ne 0 ] && help eval set -- "${ARGS}"

while true do case "$1" in -v|--version) echo "${BIN_NAME} version ${VERSION}" shift exit 0 ;; -h|--help) help shift exit 0 ;; --user) userid="$2" shift ;; --date) date="$2" shift ;; --vhostdir) vhostdir="$2" shift ;; --) shift break ;; esac shift done

if [ -z $vhostdir ]; then echo "vhostdir is a mandatory option" help exit 1 fi

if [ -z $userid ]; then echo "userid is a mandatory option" help exit 1 fi

if [ -z $date ]; then echo "date is a mandatory option" help exit 1 fi

update_user_html "$vhostdir" $userid $date

File : xzb_update_user_html.sh ends

#+end_src ** DONE shell . utility可能不存在: . $(dirname $0)/utility_xzb.sh (这个需要在同一级目录才行) CLOSED: [2013-02-09 Sat 00:01] ** DONE crontab与terminal的shell使用的environment parameter可能是不一样的 CLOSED: [2013-02-19 Tue 14:15] http://unix.stackexchange.com/questions/27289/how-can-i-run-a-cron-command-with-existing-environmental-variables

#+begin_example In the crontab, before you command, add . $HOME/.profile. For example:

0 5 * * * . $HOME/.profile; /path/to/command/to/run

Cron knows nothing about your shell; it is started by the system, so it has a minimal environment. If you want anything, you need to have that brought in yourself. #+end_example ** 运维小脚本 *** 发报警邮件 #+begin_src sh #!/bin/bash function notify_log_for_webcrawler() { err_msg=$(find /usr/local/snake_worker/log -name "erlang.log.*" -a -type f -print0 | xargs -0 -e grep -C 2 -inH -e "fail to run task") if ! [ -z "$err_msg" ]; then echo -e "========= ERROR: some tasks fail ======\n" cat <<EOF | msmtp -C /home/denny/.msmtprc -f [email protected] [email protected] Subject: webcrawler fail for some tasks

"$err_msg" EOF

fi;

}

notify_log_for_webcrawler #+end_src ** DONE bash : event not found: !前要加\做转义 CLOSED: [2013-05-30 Thu 00:03] bash-3.2$ curl -H 'Content-Type: text/xml' -d "<ToUserName></ToUserName><FromUserName></FromUserName><CreateTime>1369366098</CreateTime><MsgType></MsgType><Content></Content><MsgId>5881769154217771468</MsgId>" "http://0.0.0.0:8090/api?signature=7d0e888df44def04c76084d26e92c59fb15c1a50&timestamp=1369459087&nonce=1369763244" 2>/dev/null bash: ![CDATA[gh_05d5313dea46]]: event not found ** DONE bash fail to concat | CLOSED: [2013-06-13 Thu 15:09] 把${command_str}改在eval $command_str *** code #+begin_src sh command_str="ls | grep a" echo "$command_str" ${command_str} #+end_src **** console output #+begin_example bash-3.2$ sh ./test.sh ls | grep a ls: a: No such file or directory ls: grep: No such file or directory ls: |: No such file or directory #+end_example ** shell include with relateive direction: . $(dirname $0)/utility_xzb.sh . $(dirname $0)/utility_xzb.sh ** TODO bash -c 调用是什么意思 ** check whether variable is set https://github.com/openstack-dev/devstack.git --> functions #+begin_src sh function is_set() { local var=$"$1" eval "[ -n "$var" ]" # For ex.: sh -c "[ -n "$var" ]" would be better, but several exercises depends on this }

Prints backtrace info

filename:lineno:function

function die() { local exitcode=$? set +o xtrace local line=$1; shift if [ $exitcode == 0 ]; then exitcode=1 fi backtrace 2 err $line "$*" exit $exitcode }

Prints line number and "message" in error format

err $LINENO "message"

function err() { local exitcode=$? errXTRACE=$(set +o | grep xtrace) set +o xtrace local msg="[ERROR] ${BASH_SOURCE[2]}:$1 $2" echo $msg 1>&2; if [[ -n ${SCREEN_LOGDIR} ]]; then echo $msg >> "${SCREEN_LOGDIR}/error.log" fi $errXTRACE return $exitcode }

function die_if_not_set() { local exitcode=$? FXTRACE=$(set +o | grep xtrace) set +o xtrace local line=$1; shift local evar=$1; shift if ! is_set $evar || [ $exitcode != 0 ]; then die $line "$*" fi $FXTRACE }

die_if_not_set $LINENO AMI "Failure registering" #+end_src ** DONE shell script to update ini/cfg configuration file CLOSED: [2013-10-22 Tue 14:35] /Users/mac/backup/tech/large_sourcecode/openstack/openstack-dev/devstack/openstack-dev/functions #+begin_src sh

Set an option in an INI file

iniset config-file section option value

function iniset() { local file=$1 local section=$2 local option=$3 local value=$4

if ! grep -q "^\[$section\]" "$file" 2>/dev/null; then
    # Add section at the end
    echo -e "\n[$section]" >>"$file"
fi
if ! ini_has_option "$file" "$section" "$option"; then
    # Add it
    sed -i -e "/^\[$section\]/ a\\

$option = $value " "$file" else # Replace it sed -i -e "/^[$section]/,/^[.]/ s|^($option[ \t]=[ \t]).$|\1$value|" "$file" fi }

#+end_src ** DONE shell $SECONDS show how long the script runs: echo_summary "stack.sh completed in $SECONDS seconds." CLOSED: [2013-10-22 Tue 14:56] *** built-in environment: $SECONDS stack.sh completed in $SECONDS seconds. *** manual caculate /Users/mac/backup/tech/large_sourcecode/openstack/openstack-dev/devstack/openstack-dev/exercises/volumes.sh #+begin_src sh start_time=$(date +%s) cinder create --display_name $VOL_NAME --display_description "test volume: $VOL_NAME" $DEFAULT_VOLUME_SIZE ||
die $LINENO "Failure creating volume $VOL_NAME" if ! timeout $ACTIVE_TIMEOUT sh -c "while ! cinder list | grep $VOL_NAME | grep available; do sleep 1; done"; then die $LINENO "Volume $VOL_NAME not created" fi end_time=$(date +%s) echo "Completed cinder create in $((end_time - start_time)) seconds"

#+end_src ** shell function in function /Users/mac/backup/tech/large_sourcecode/openstack/openstack-dev/devstack/openstack-dev/lib/swift #+begin_src sh function test() { function test1() { echo "hello1" } test1 test1 test1 test1 } test test1 echo "end" #+end_src ** DONE grep command output: make sure detect the command is running correctly CLOSED: [2013-10-22 Tue 16:42] *** 方法: #+begin_example val=$(nova aggregate-list | grep -c " $aggregate_name ") if [[ $? -eq 0 ]] && [[ $val == 0 ]]; then #+end_example *** 相关信息 https://review.openstack.org/#/c/53233/1/exercises/aggregates.sh,unified #+begin_example

diff --git a/exercises/aggregates.sh b/exercises/aggregates.sh index e5fc7de..f8c60f9 100755 --- a/exercises/aggregates.sh +++ b/exercises/aggregates.sh @@ -48,24 +48,25 @@ 48 48 # Create an aggregate 49 49 # =================== 50 50 51 51 AGGREGATE_NAME=test_aggregate_$RANDOM 52 52 AGGREGATE2_NAME=test_aggregate_$RANDOM 53 53 AGGREGATE_A_ZONE=nova 54 54 55 55 exit_if_aggregate_present() { 56 56 aggregate_name=$1 57 57 58 - if [ $(nova aggregate-list | grep -c " $aggregate_name ") == 0 ]; then 58 + val=$(nova aggregate-list | grep -c " $aggregate_name ") 59 + if [[ $? -eq 0 ]] && [[ $val == 0 ]]; then 59 60 echo "SUCCESS $aggregate_name not present" 60 61 else 61 - die $LINENO "found aggregate: $aggregate_name" 62 + die $LINENO "fail to get aggregate or found aggregate: $aggregate_name" 62 63 exit -1 63 64 fi 64 65 } 65 66 66 67 exit_if_aggregate_present $AGGREGATE_NAME 67 68 68 69 AGGREGATE_ID=$(nova aggregate-create $AGGREGATE_NAME $AGGREGATE_A_ZONE | grep " $AGGREGATE_NAME " | get_field 1) 69 70 AGGREGATE2_ID=$(nova aggregate-create $AGGREGATE2_NAME $AGGREGATE_A_ZONE | grep " $AGGREGATE2_NAME " | get_field 1) 70 71 71 72 # check aggregate created #+end_example ** DONE shell contiune loop CLOSED: [2013-10-23 Wed 15:25] http://www.cyberciti.biz/faq/unix-linux-bsd-appleosx-continue-in-bash-loop/ #+begin_src sh for i in something do [ condition ] && continue cmd1 cmd2 done #+end_src ** DONE trueorfalse CLOSED: [2013-10-23 Wed 15:29] /Users/mac/backup/tech/large_sourcecode/openstack/openstack-dev/devstack/is-service-enabled/functions #+begin_src sh

Normalize config values to True or False

Accepts as False: 0 no No NO false False FALSE

Accepts as True: 1 yes Yes YES true True TRUE

VAR=$(trueorfalse default-value test-value)

function trueorfalse() { local default=$1 local testval=$2

[[ -z "$testval" ]] && { echo "$default"; return; }
[[ "0 no No NO false False FALSE" =~ "$testval" ]] && { echo "False"; return; }
[[ "1 yes Yes YES true True TRUE" =~ "$testval" ]] && { echo "True"; return; }
echo "$default"

} #+end_src ** DONE bash backtrace CLOSED: [2013-10-23 Wed 16:12] /Users/mac/backup/tech/large_sourcecode/openstack/openstack-dev/devstack/openstack-dev/functions #+begin_src sh

Prints backtrace info

filename:lineno:function

function backtrace { local level=$1 local deep=$((${#BASH_SOURCE[@]} - 1)) echo "[Call Trace]" while [ $level -le $deep ]; do echo "${BASH_SOURCE[$deep]}:${BASH_LINENO[$deep-1]}:${FUNCNAME[$deep-1]}" deep=$((deep - 1)) done } #+end_src ** DONE detech oom-killer CLOSED: [2013-10-30 Wed 19:05] /Users/mac/backup/tech/large_sourcecode/openstack/openstack-infra/config/modules/jenkins/files/slave_scripts/jenkins-oom-grep.sh #+begin_src sh #!/bin/bash

Copyright 2012 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License"); you may

not use this file except in compliance with the License. You may obtain

a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS, WITHOUT

WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the

License for the specific language governing permissions and limitations

under the License.

Find out if jenkins has triggered the out-of-memory killer by checking

the output of dmesg before and after a test run.

PATTERN=" invoked oom-killer: "

case "$1" in pre) rm -fr /tmp/jenkins-oom-log mkdir /tmp/jenkins-oom-log dmesg > /tmp/jenkins-oom-log/pre exit 0 ;; post) dmesg > /tmp/jenkins-oom-log/post diff /tmp/jenkins-oom-log/{pre,post}
| grep "^> " | sed "s/^> //" > /tmp/jenkins-oom-log/diff if grep -q "$PATTERN" /tmp/jenkins-oom-log/diff then cat /tmp/jenkins-oom-log/diff exit 1 fi ;; esac

#+end_src ** DONE shell:当$(fun)方式调用时,fun没法quit 当前shell script CLOSED: [2013-12-12 Thu 20:59] #+begin_src sh get_data() { local match_column=$(($1 + 1)) local regex="$2" local output_column=$(($3 + 1)) shift 3

result=$("$@" | \
       awk -F'|' \
           "! /^\+/ && \$${match_column} ~ \"^ *${regex} *\$\" \
            { print \$${output_column} }")
[[ -z "$result" ]] || exit -1

}

output=$(get_data 2 admin 1 keystone role-list) echo $output #+end_src ** TODO awk "length>50" /home/zhangwei/elmar/elmar/src/elmar.erl打印出文件名与行号数 ** DONE ask for confirmation CLOSED: [2013-12-31 Tue 23:59] #+begin_src sh #!/bin/bash ##-------------------------------------------------------------------

File : test.sh

Author : filebat [email protected]

Description :

--

Created : <2013-12-29>

Updated: Time-stamp: <2013-12-31 23:58:24>

##------------------------------------------------------------------- if [ "$1" = "-y" ] || [ "$1" = "--yes" ]; then SKIP_ASK=y shift fi

user_wants() { if [ "$SKIP_ASK" = "y" ]; then return 0; fi

while true; do
    read -n 1 -p "$1 " cont
    echo 1>&2
    case $cont in
        y|Y)
            return 0
            ;;
        n|N)
            return 1
            ;;
        *)
            ;;
    esac
done

}

if user_wants 'Drop Heat database tables?'; then echo "here" fi

File : test.sh ends

#+end_src ** DONE sh wrap OS difference CLOSED: [2014-01-01 Wed 00:17] /Users/mac/backup/tech/large_sourcecode/openstack/openstack/heat/heat/bin/heat-db-setup #+begin_src sh

Check for root privileges

if [[ $EUID -ne 0 ]] ; then echo "This operation requires superuser privileges, using sudo:" if sudo -l > /dev/null ; then ELEVATE="sudo" else exit 1 fi fi

case "$1" in rpm) echo "Installing on an RPM system." PACKAGE_INSTALL="$ELEVATE yum install" PACKAGE_STATUS="rpm -q" SERVICE_MYSQLD="mysqld" SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start" SERVICE_STATUS="service $SERVICE_MYSQLD status" SERVICE_ENABLE="$ELEVATE chkconfig" ;; deb) echo "Installing on a Debian system." PACKAGE_INSTALL="$ELEVATE apt-get install" PACKAGE_STATUS="dpkg-query -s" SERVICE_MYSQLD="mysql" SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start" SERVICE_STATUS="$ELEVATE service $SERVICE_MYSQLD status" SERVICE_ENABLE="" ;; *) usage ;; esac

#+end_src ** DONE sh check for root privileges, since the euid may not be 0, but still in the admin user group CLOSED: [2014-01-01 Wed 00:19] #+begin_src sh

Check for root privileges

if [[ $EUID -ne 0 ]] ; then echo "This operation requires superuser privileges, using sudo:" if sudo -l > /dev/null ; then ELEVATE="sudo" else exit 1 fi fi

#+end_src ** DONE shell comparision of float CLOSED: [2014-07-10 Thu 10:11] http://stackoverflow.com/questions/11541568/how-to-do-float-comparison-in-bash echo "10.2>10.1" | bc ** run_command_in_all_nodes.sh #+begin_src sh #!/bin/bash ##-------------------------------------------------------------------

File : run_command_in_all_nodes.sh

Author : Denny [email protected]

Description : Run command in a list of servers.

Here we assume ssh key is properly uploaded, thus we

don't need to input ssh password

--

Created : <2014-07-10>

Updated: Time-stamp: <2014-07-31 14:10:04>

##-------------------------------------------------------------------

Example1: Run a command in all nodes of prod env.

sh ./run_command_in_all_nodes.sh "ssh root@HOSTIP date"

Note: We assume ssh public key is uploaded properly

In above, HOSTIP will be replaced by server ip.

Example2: Run a command in some nodes

sh ./run_command_in_all_nodes.sh "ssh root@HOSTIP date" "172.20.16.14 172.20.16.12"

command=${1:-"ssh root@HOSTIP date"} server_list=${2:-"172.20.16.14 172.20.16.12 172.20.16.17 172.20.16.18 172.20.16.13 172.20.16.15 172.20.16.16 172.20.16.11 172.21.16.11 172.21.16.12 172.20.16.19 172.20.16.20 172.20.16.23 172.20.16.26 172.20.18.13 172.20.18.15 172.20.18.16"}

date=date +%Y-%m-%d_%H:%M has_error=false echo date +['%Y-%m-%d %H:%M:%S'] "Actions Begin\n"

for server in ${server_list[*]}; do actual_command=echo $command | sed "s/HOSTIP/$server/g" echo "\n========== On $server Run: $actual_command ==========" output=$actual_command if [ $? -ne 0 ]; then has_error=true echo "Error to run: $actual_command\n" else if (echo "$output" | grep -i error 1>/dev/null); then has_error=true echo "========== Action on $server failed ==========" fi; fi; echo "${output}\n" done;

echo date +['%Y-%m-%d %H:%M:%S']" Actions are done\n"

File : run_command_in_all_nodes.sh ends

#+end_src ** DONE [#A] bash do case :IMPORTANT: CLOSED: [2014-08-30 Sat 20:04] http://www.thegeekstuff.com/2010/07/bash-case-statement/ #+begin_src sh $ cat signal.sh #!/bin/bash

if [ $# -lt 2 ] then echo "Usage : $0 Signalnumber PID" exit fi

case "$1" in

echo "Sending SIGHUP signal" kill -SIGHUP $2 ;;
echo "Sending SIGINT signal" kill -SIGINT $2 ;;
echo "Sending SIGQUIT signal" kill -SIGQUIT $2 ;;
echo "Sending SIGKILL signal" kill -SIGKILL $2 ;; ) echo "Signal number $1 is not processed" ;; esac #+end_src ** DONE shell if... then..: if true ; then echo "hello" ; fi CLOSED: [2014-10-02 Thu 16:18] ** DONE shell substract 2 variable: COUNT=expr $FIRSTV - $SECONDV CLOSED: [2014-10-03 Fri 11:56] ** DONE mac get file modified timestamp: stat -f "%Sm" -t "%s" $flagfile CLOSED: [2014-10-03 Fri 11:41] http://stackoverflow.com/questions/12169710/formatted-modified-date-time-on-mac-bash ** DONE shell trim whitespace: echo "${output}" | sed -e 's/^[ \t]//' CLOSED: [2015-02-21 Sat 07:51] ** DONE Bash run with parameter CLOSED: [2015-03-19 Thu 11:42] /usr/local/bin/run_command_in_all_nodes.sh "ssh $ssh_opt -t -t root@HOSTIP curl -L http://sprepo.fluigidentity.com/fluig_share/chef_cmd/deployment/$deploy_config_file | skip_monitor_log=1 repo_server_ip_port=$repo_server_ip_port deploy_version=$branch_name bash -e"

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" ** DONE track timespent CLOSED: [2015-04-22 Wed 08:41] START=$(date +%s) backup_dir END=$(date +%s) DIFF=$(echo "$END - $START" | bc) log "Track time spent: Backup step takes $DIFF seconds" ** DONE bash -e and cron CLOSED: [2015-04-26 Sun 18:42] How to write a daily crontab script #+BEGIN_EXAMPLE root@ip-172-31-11-0:~/osc_it/backup_dir# vim /tmp/test.sh root@ip-172-31-11-0:~/osc_it/backup_dir# chmod 755 /tmp/test.sh root@ip-172-31-11-0:~/osc_it/backup_dir# bash /tmp/test.sh /etc/hosts ls: cannot access /etc/hosts2: No such file or directory /etc/hosts root@ip-172-31-11-0:~/osc_it/backup_dir# bash -e /tmp/test.sh /etc/hosts ls: cannot access /etc/hosts2: No such file or directory root@ip-172-31-11-0:~/osc_it/backup_dir# cat /tmp/test.sh #!/bin/bash -e ls /etc/hosts ls /etc/hosts2 ls /etc/hosts #+END_EXAMPLE ** DONE cat VS tac CLOSED: [2015-04-27 Mon 18:53] http://stackoverflow.com/questions/742466/how-can-i-reverse-the-order-of-lines-in-a-file ** DONE less vs more CLOSED: [2015-04-27 Mon 18:54] ** DONE [#C] bash array separate by new line, instead of whitespace CLOSED: [2015-07-09 Thu 07:57] http://mindspill.net/computing/linux-notes/using-the-bash-ifs-variable-to-make-for-loops-split-with-non-whitespace-characters/ http://stackoverflow.com/questions/19771965/split-bash-string-by-newline-characters

checks="check_gui_home -t 20 check_gui_login"

IFS=$'\n'

for a in $checks; do echo "--" "$a" "--" ; done unset IFS ** DONE bash divide numbers: echo 5/2 | bc -l: use bc or bash CLOSED: [2015-07-21 Tue 16:46] http://stackoverflow.com/questions/1088098/how-do-i-divide-in-the-linux-console

string=84415266816 echo "$string/(102410241024)" | bc -l

echo $(( 7 / 3 )) ** DONE [#A] bash filter string: echo $ram_capacity | sed -n 's/.(free.)}/\1/p' CLOSED: [2015-07-21 Tue 16:05] http://stackoverflow.com/questions/11568859/how-to-extract-text-from-a-string-using-sed MacPro:default mac$ echo $ram_capacity {u'used': 5064916008, u'total': 84415266816, u'quotaTotal': 84415266816, u'usedByData': 39025953, u'free': 79350350808}

ram_capacity="{u'used': 5064916008, u'total': 84415266816, u'quotaTotal': 84415266816, u'usedByData': 39025953, u'free': 79350350808}" echo $ram_capacity | sed -n 's/.(free.)}/\1/p' ** DONE [#A] bash function mute output :IMPORTANT: CLOSED: [2015-08-01 Sat 22:06] #+BEGIN_SRC sh #!/bin/bash -e function fun() { INSTANCE_ID="id1" echo "hello, world" }

echo "run0:" fun

val1=${fun} echo "run1: $val1"

val2=$(fun) echo "run2: $val2"

echo "INSTANCE_ID: $INSTANCE_ID". #+END_SRC

#+BEGIN_EXAMPLE MacPro:~ mac$ ./test.sh run0: hello, world run1: run2: hello, world INSTANCE_ID: id1. #+END_EXAMPLE ** DONE linux convert string to int CLOSED: [2015-08-02 Sun 08:20] http://stackoverflow.com/questions/11268437/how-to-convert-string-to-integer-in-unix

expr $d1 - $d2 You can also do:

echo $(( d1 - d2 ))

[ expr "22" - 23 -gt 0 ]; echo $? ** # --8<-------------------------- separator ------------------------>8-- ** DONE bash: from a list of directories, remove all except the first two: ls -lth /tmp/ | awk 'NR>2' CLOSED: [2015-08-13 Thu 08:16] #+BEGIN_EXAMPLE jenkins@1e7c2d4de548:/var/www/repo$ ls -lth | grep dev_code | awk -F' ' '{print $9}' dev_code_9d820f6df69cccc2b28be12336a0e4bf88c4d555 dev dev_code_b13a247cf739d75368d63180e5218d22c0c73fdd dev_code_4610882ea3b665908ba0726f186ce730ab8a0e9f dev_code_ba1965c75f3ccd28f0f37bff0916ed48154c90ed dev_code_71d197967f92bb0600afe2d307d1dd86315d5835 dev_code_0e45206e90a6c1603db676a085b6c531ed655e62 dev_code_0d23ccf8544ccb87e4511a3c65c8358c15d5325b dev_code_429b951934969a7055b72ec2aa9a5549b7778981 #+END_EXAMPLE ** DONE bash print line #2 and line #3: ls -lth /tmp/ | awk 'NR==2,NR==3' CLOSED: [2015-08-13 Thu 08:16] #+BEGIN_EXAMPLE MacPro:org_data mac$ ls -lth /tmp/ | awk 'NR==2,NR==3' drwx------ 3 root wheel 102B Aug 13 06:05 KSOutOfProcessFetcher.0.I5ci1K_TwCwqo1sKvc0siaBbJTw= -rw-r--r-- 1 mac wheel 8.4K Aug 13 01:41 pre_push.log MacPro:org_data mac$ ls -lth /tmp/ total 32 drwx------ 3 root wheel 102B Aug 13 06:05 KSOutOfProcessFetcher.0.I5ci1K_TwCwqo1sKvc0siaBbJTw= -rw-r--r-- 1 mac wheel 8.4K Aug 13 01:41 pre_push.log -rw-r--r-- 1 mac wheel 1.5K Aug 9 09:49 test_docker_reboot.sh drwx------ 3 root wheel 102B Aug 7 19:07 launchd-412.p65VLV drwx------ 3 mac wheel 102B Aug 7 19:07 launch-fHcqj0 drwx------ 3 mac wheel 102B Aug 7 19:07 launch-v7pPEn drwx------ 3 mac wheel 102B Aug 7 19:05 launchd-268.WPmyrE drwx------ 3 _spotlight wheel 102B Aug 7 19:03 launchd-261.pK6Cby MacPro:org_data mac$ #+END_EXAMPLE ** DONE copy a list of files preseving the directory path CLOSED: [2015-08-13 Thu 08:25] http://serverfault.com/questions/180853/how-to-copy-file-preserving-directory-path-in-linux #+BEGIN_EXAMPLE rm -rf /tmp/backup/* cp -r --parents /var/lib/jenkins/jobs/BuildRepoCode /tmp/backup ls -lth /tmp/backup #+END_EXAMPLE ** DONE bash string to list delimiter separator: IFS=$'\n'; echo "${System[*]}" CLOSED: [2015-10-27 Tue 11:01] http://superuser.com/questions/461981/how-do-i-convert-a-bash-array-variable-to-a-string-delimited-with-newlines

#+BEGIN_EXAMPLE Here's a way that utilizes bash parameter expansion and its IFS special variable.

$ System=('s1' 's2' 's3' 's4 4 4') $ ( IFS=$'\n'; echo "${System[]}" ) #+END_EXAMPLE ** DONE base dirname: echo ${PIDFILE%/} CLOSED: [2015-11-24 Tue 20:33] #+BEGIN_EXAMPLE Denny-mac:~ mac$ PIDFILE="/etc/hosts/abc/123/tmp.pid" You have new mail in /var/mail/mac Denny-mac:~ mac$ echo ${PIDFILE%/*} /etc/hosts/abc/123 #+END_EXAMPLE ** DONE bash check variable is number CLOSED: [2016-05-02 Mon 18:41] http://stackoverflow.com/questions/806906/how-do-i-test-if-a-variable-is-a-number-in-bash

#+BEGIN_SRC sh re='^[0-9]+$' if ! [[ $yournumber =~ $re ]] ; then echo "error: Not a number" >&2; exit 1 fi #+END_SRC ** DONE bash count matched string: echo "echo 1234 echo" | grep -o echo CLOSED: [2016-05-23 Mon 07:47] http://stackoverflow.com/questions/6741967/how-can-i-count-the-occurrences-of-a-string-within-a-file-using-bash ** DONE bash check whethter string is a valid ip CLOSED: [2016-05-30 Mon 17:44] http://stackoverflow.com/questions/13777387/check-for-ip-validity #+BEGIN_SRC sh #!/bin/bash ip=1.2.3.4

if [[ $ip =~ ^[0-9]+.[0-9]+.[0-9]+.[0-9]+$ ]]; then echo "success" else echo "fail" fi #+END_SRC ** DONE Changing a linux password via script: echo "root:NEWPASSWD" | chpasswd :noexport: CLOSED: [2016-06-01 Wed 13:59] http://stackoverflow.com/questions/27837674/changing-a-linux-password-via-script

echo "root:DevOpsChangeMe1" | chpasswd

ssh -i my_id_rsa [email protected] "echo root:NEWPASSWD | chpasswd"

echo -e "newpasswd123\nnnewpasswd123" | passwd user ** DONE bash to check whether ssh server reachable CLOSED: [2016-06-10 Fri 10:57] function ip_list_ping_reachable() { # Sample: # ip_list_ping_reachable "true" "172.17.0.2 # 172.17.0.3 # 172.17.0.4" # ip_list_ping_reachable "false" "$ip_list" local exit_if_fail=${1?} local ip_list=${2?} for ip in $ip_list; do # echo "ping ip: ${ip}" if ! ping -c3 "$ip" 2>/dev/null 1>/dev/null; then if [ "$exit_if_fail" = "true" ]; then echo "ERROR: Current machine can't ping $ip. Please check input parameters." exit 1 else echo "Warning: Current machine can't ping $ip. Please check input parameters." fi fi done } ** DONE bash get file acl mode: stat -c "%a %n" /etc/hosts CLOSED: [2016-06-14 Tue 16:29] http://askubuntu.com/questions/152001/how-can-i-get-octal-file-permissions-from-command-line ** DONE bash break long line to short line CLOSED: [2016-06-17 Fri 10:59] http://unix.stackexchange.com/questions/82182/having-multi-lines-in-bash-command-substitution http://stackoverflow.com/questions/18599711/how-can-i-split-a-bash-command-over-multiple-lines-when-using-an-if-statement #+BEGIN_SRC sh package_list="lsof curl git tar apt wget vim strace
libcurl3 openssh-client " #+END_SRC ** DONE shell break for and continue CLOSED: [2015-05-13 Wed 22:19] http://www.tutorialspoint.com/unix/unix-loop-control.htm

break #+BEGIN_SRC sh #!/bin/sh

for var1 in 1 2 3 do for var2 in 0 5 do if [ $var1 -eq 2 -a $var2 -eq 0 ] then break 2 else echo "$var1 $var2" fi done done #+END_SRC

continue #+BEGIN_SRC sh #!/bin/sh

NUMS="1 2 3 4 5 6 7"

for NUM in $NUMS do Q=expr $NUM % 2 if [ $Q -eq 0 ] then echo "Number is an even number!!" continue fi echo "Found odd number" done #+END_SRC ** DONE Check if a function exists before executing it in shell CLOSED: [2016-01-29 Fri 17:27] http://stackoverflow.com/questions/17972087/check-if-a-function-exists-before-executing-it-in-shell http://www.cyberciti.biz/faq/bash-shell-scripting-find-out-if-function-definedornot/ #+BEGIN_SRC sh if type foo | grep -i function > /dev/null; then

foo is a function

fi #+END_SRC ** TODO set -o pipefail pipefail: the return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status

http://stackoverflow.com/questions/1221833/bash-pipe-output-and-capture-exit-status ** DONE bash remove whitespace and new line CLOSED: [2016-12-08 Thu 17:44] http://stackoverflow.com/questions/13659318/how-to-remove-space-from-string

http://www.regular-expressions.info/posixbrackets.html

jenkins_jobs="BuildMDMRepo ,UpdateSandboxMDM,UpdateJenkinsItself,TailLogfile,CollectFiles" echo "${jenkins_jobs//[[:blank:]]/}"

echo "$jenkins_jobs" | tr -s " "

#!/bin/bash -e jenkins_jobs="BuildMDMRepo , UpdateSandboxMDM,UpdateJenkinsItself,TailLogfile,CollectFiles" echo "${jenkins_jobs//[[:space:]]/}" ** # --8<-------------------------- separator ------------------------>8-- ** DONE remove old backup and keep laest 2 backup CLOSED: [2016-12-26 Mon 10:45] ssh -p 2702 [email protected]

/tmp/test.sh && vim /tmp/test.sh

bash /tmp/test.sh

#+BEGIN_SRC sh #!/bin/bash function remove_old_folder() { folder_to_check=${1?} keep_latest_count=${2:-1} i=1 cd "$folder_to_check" for d in $(ls -t .); do echo "d: $d, i: $i, keep_latest_count: $keep_latest_count" if [ $i -gt $keep_latest_count ]; then echo "rm $folder_to_check/$d" #rm -rf "$folder_to_check/$d" fi i=$((i+1)) done }

remove_old_folder "/mnt/cb-backup-01/backup/mdm-staging/" 1 #+END_SRC *** output #+BEGIN_EXAMPLE root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# dir_to_check="/mnt/cb-backup-01/backup/mdm-staging/" root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# for d in $(find "$dir_to_check" -type d -ctime +7); do

if [ -d "$d" ]; then echo "rm $d" rm -rf $d fi done root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# ls -lth total 12K drwxr-xr-x 3 root root 4.0K Dec 25 01:47 2016-12-25T064711Z root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# date Sun Dec 25 05:52:57 EST 2016

find . -maxdepth 1 -type d -ctime +7

root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# find . -maxdepth 1 -type d -ctime +7 root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# ls 2016-12-11T053231Z 2016-12-18T060028Z 2016-12-25T064711Z root@prod-cb-backup:/mnt/cb-backup-01/backup/mdm-staging# stat 2016-12-11T053231Z Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 810h/2064d Inode: 16252977 Links: 7 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2016-12-25 04:57:02.775279051 -0500 Modify: 2016-12-24 00:22:22.999279051 -0500 Change: 2016-12-24 00:22:22.999279051 -0500 Birth: - #+END_EXAMPLE ** DONE shell get current file name: basename "$0" CLOSED: [2017-05-12 Fri 22:53] ** DONE Turning multiple lines into one line with comma separated: sed -e 's/ /,/g' CLOSED: [2017-09-08 Fri 17:34] https://stackoverflow.com/questions/15758814/turning-multiple-lines-into-one-line-with-comma-separated-perl-sed-awk ** DONE bash number calculation CLOSED: [2017-10-17 Tue 22:17] #+BEGIN_EXAMPLE #!/usr/bin/env bash ##-------------------------------------------------------------------

@copyright 2017 DennyZhang.com

Licensed under MIT

https://www.dennyzhang.com/wp-content/mit_license.txt

File: test.sh

Author : Denny [email protected]

Description :

https://leetcode.com/problems/cheatsheet-shell-A4/description/][leetcode.com]]

--

Created : <2017-10-17>

Updated: Time-stamp: <2017-10-17 22:13:31>

##------------------------------------------------------------------- set -e

one_line=($(head -n1 ./file.txt)) column_count=${#one_line[@]} line_count=$(wc -l ./file.txt | awk -F' ' '{print $1}')

echo "column_count: $column_count, line_count: $line_count"

output=($(cat ./file.txt))

echo "output: $output"

for((i=0; i<$column_count; i++)); do for((j=0; j<$line_count; j++)); do index=$((i+j*column_count)) if [ $((j+1)) -eq $line_count ]; then echo -n "${output[$index]}" else echo -n "${output[$index]} " fi # echo "${output[$index} " done echo "" done #+END_EXAMPLE ** DONE shell replace string: get master-index-023ea45 from master-index-023ea45: echo "${var//-index/}" CLOSED: [2018-01-29 Mon 16:27]

Useful shell code snippet :noexport:Coding: :PROPERTIES: :type: Language_Linux :END:

#+begin_example Please note: WORDPRESS differentiates among the following: | ｜, ' ', -- －, "" #+end_example

process | Question | Solution | Use case | |-----------------------------------+--------------------------------------------------------------------------------------------------------------------+-----------------------------------| | 列出某个进程打开了哪些文件 | lsof +p 1971 | | | 显示内存占用量最高的前10个process | ps aux ｜ sort -nk +4 ｜ tail | | | 查看某类进程启动的时间 | ps -ef ｜ awk '/[p]hp-fpm/{print $2,$5,$7,$8}' | 查找超时进程 | | 列出运行超过一天的php进程 | ps -o etime,ppid,uid,pid,command -e ｜ grep '/usr/bin/php' ｜ awk -F' ' '{print $1" "$2" "$5}' ｜ grep '^[0-9]+-' | 查找超时进程 | | 列出所有父进程为init1的进程 | ps -f --ppid 1 | 查找孤儿进程,判断是否进行垃圾回收 | | 如何查出某个pid运行在哪个cpu上 | ps -efP ｜　grep 21587 | | | get process detail start time | ps -o uid,pid,etime -p 11880 | | | 查看emacs的资源占用情况 | top -p $(pidof emacs) | |
Common used | Question | Solution | Use case | |--------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------| | 按更新时间, 查看某个文件夹 | ls -alth | | | 循环运行某个命令 | for ((b=1; b<=100; b++)); do echo hello; done; | | | if...else的条件判断 | if [ ! -d /tmp/tt ]; then mkdir /tmp/tt; else echo exists; fi; | | | 模拟并发请求 | 在多个terminal下,同时执行: for ((b=1; b<=100; b++)); do thrift_client -c sanityCheck -s 192.168.75.108 -p 9401; done; | | | 对一组远端服务器, 同时执行某个命令 | command="date"; for puppet_client_ip in "192.168.75.108" "192.168.75.109" "192.168.75.110" "192.168.75.111"; do ssh -o StrictHostKeyChecking=no root@$puppet_client_ip "$command";done; | | | 向文件中追加一个key=value的键值对, 如果存在则修改它 | (grep "key1 +=" ./test.cfg && sudo sed -i 's/key1 +=.*/key1 = value1/' ./test.cfg) ｜｜ ((cat ./test.cfg; echo "key1 = value1") ｜ sudo tee ./test.cfg) | | | 查看svn的某个revision的修改 | svn diff -r 135:136 http://test.server.cn/project/trunk | | | 临时进入某个目录, 并运行某个命令, 同时不改变当前目录的位置 | (cd /tmp && ls) | | | 当命令A执行完后, 需要执行命令B, 但不修改$?　 | A && B ｜｜ true | | | 查看某个文件夹下, 今天被修改文件的最后若干行 | | | | 在指定时间运行某个命令 | echo "ls -l" ｜ at midnight | | | 重新执行上一个命令 | (sudo !!) | | | 按字符串搜索查找某个文件夹 | grep -lir "some text" * | | | 监控某个文件是否被修改了 | yum install inotify-tools; while inotifywait -e modify /var/log/message; do echo "changed"; done | |
file management | Question | Solution | Use case | |-------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------+------------------| | 查看某个文件的第5到第8行 | sed -n 5,8p /etc/hosts | | | 删除某个文件指定的几行 | sed -i 2,3d /etc/hosts | | | 列出某个文件中太长的行 | awk 'length>50' /etc/hosts | | | 列出某个文件夹下, 最近60分钟内被修改的文件及文件夹 | sudo find /var/log -mmin 60 -type f | | | 列出/var/log/下, 文件大小介于50k到100k之间的文件 | sudo find /var/log -type f -size +50k -size -100k | | | 找出满足条件的文件, 并将它们删除 | find ./ -name Thumbs.db -delete | | | Graphical tree of sub-directories | ls -R ｜ grep ":$" ｜ sed -e 's/:$//' -e 's/[^-][^/]//--/g' -e 's/^/ /' -e 's/-/｜/' | | | quickly backup or copy a file with bash | cp filename{,.bak} | | | Delete all files in a folder that don't match a certain file extension | rm !(.foo｜.bar｜.baz) | | | List the size (in human readable form) of all sub folders from the current location | du -h --max-depth=1 | | | make directory tree | mkdir -p work/{d1,d2}/{src,bin,bak} | | | 递归地删除所有空文件夹 | find . -type d -empty -delete | | | 比较两个目录树的差异 | diff <(cd dir1 && find ｜ sort) <(cd dir2 && find ｜ sort) | | | Get the 10 biggest files/folders for the current direcotry | du -s * ｜ sort -n ｜ tail | | | Remove all but one specific file | rm -f !(survivior.txt) | | | Find files that were modified by a given command | touch /tmp/file ; $EXECUTECOMMAND ; find /path -newer /tmp/file | | | Copy a file structure without files | find * -type d -exec mkdir /where/you/wantem/{} ; | | | 找出一个文件夹下, md5相同的文件 | fdupes -r . 或者 find . -type f -exec md5sum '{}' ';' ｜ sort ｜ uniq --all-repeated=separate -w 33 ｜ cut -c 35- | | | 对于一个文件, 指定两个开始和结束的regexp, 只显示满足条件的区间内容 | awk '/127.0.0.1/,/The following/' /etc/hosts | | | 删除修改时间在一年以上的文件 | find -mtime +365 -and -not -type d -delete | 删除不需要的文件 | | 创建一个指定大小的文件 | truncate -s 1M file 或者 dd if=/dev/zero of=foo.txt bs=1M count=1 | 测试文件系统性能 | | 找出最新三天修改的某类文件 | find . -name "*.org" -a ( -mtime 0 -o -mtime 1 -o -mtime 2 ) | | | 创建文件时,　指定acl | umask 226 && echo "stack ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/50_stack_sh | | | 打印一个文件夹的所有内容 | for f in ls . ; do { echo -e "\n======== $f ======"; tail $f;}; done | |
network | Question | Solution | Use case | |------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------| | 查看某个tcp端口被哪个pid占用 | lsof -i tcp:80 | | | 显示本机与所有remote host已经建立的网络链接 | netstat -an ｜ grep ESTABLISHED ｜ awk '{print $5}' ｜ awk -F: '{print $1}' ｜ sort ｜ uniq -c ｜ awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("")}; print "" }' | | | 显示本机的外网ip | curl ifconfig.me | | | Show apps that use internet connection at the moment | ss -p | | | List alive hosts in specific subnet | nmap -sP 192.168.1.0/24 | | | 查看当前网段, 有哪些机器开了某个端口 | nmap -sT -p 80 -oG - 192.168.1. | | | check open ports | lsof -Pni4 ｜ grep LISTEN | | | 查看各网卡的流入与流出带宽 | ifstat -nt | | | 诊断网络状况 | mtr --report-cycles=5 --interval=2 --split | |
remote host management | Question | Solution | Use case | |----------------------------------+----------------------------------------------------------------------------------+----------| | 对两个不同机器上的文件进行diff | diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list) | | | 在远端的两个host之间进行文件传输 | scp user@hostb:file user@hostc: | |
view system information | Question | Solution | Use case | |--------------------------------------------+-----------------------------------------------------------+----------| | 将当前机器硬件生成一个全面,美观的html报表 | lshw -html > hardware.html | | | 查看OS是64位,　还是32位 | file /usr/bin/file; uname -m; getconf -a ｜ grep LONG_BIT | | | 查看CPU信息 | cat /proc/cpuinfo | | | 如何查看linux系统是哪个版本的 | lsb_release -a | | | 查看系统内核 | uname -a | |
Memory | Question | Solution | Use case | |--------------------------+-------------------------------------+----------| | 按内存使用量对进程排序 | ps -eo pid,pmem,rss,comm --sort rss | | | 查看某个进程的内存使用量 | ps -p 2977 -o vsz= | | | 查看某个进程的内存使用量 | pmap 2977 | | | 查看某个进程的内存使用量 | top -n 1 -p 2977 | | | 显示系统内存使用情况 | free -m -t | | | 显示系统内存使用情况 | cat /proc/meminfo | |
Misc | Question | Solution | Use case | |-------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+----------| | 在指定时间点运行某个脚本 | at -f /tmp/test.sh -v 18:07 | | | Convert seconds to human-readable format | date -d@1234567890 | | | Get utc seconds | date +%s | | | get yesterday | date "+%a %d/%m/%Y" --date="1 days ago" | | | Quick access to the ascii table. | man ascii | | | Start COMMAND, and kill it if still running after 5 seconds | timeout 5s COMMAND | | | 产生一串随机数 | echo $RANDOM; echo {0..1}{0..1}{0..1}{0..1} | | | 限制某个进程对cpu的使用率 | cpulimit -p 1234 -l 50 | | | 列出当前安装软件的包大小 | dpkg-query -Wf '${Installed-Size}\t${Package}\n' ｜ sort -n | | | 按固定大小将一个大文件切分成几个小文件 | split -b 500k /var/log/messages message_part_ | | | Display GCC Predefined Macros | gcc -dM -E - < /dev/null | | | svn up某个文件夹下的所有子文件夹 | find . -maxdepth 1 -type d ｜ xargs svn up | | | 批量缩小图片 | rm -rf ../new && mkdir ../new && for img in find . -iname '*.jpg' -a -size +1M; do convert -resize 50%x50% "$img" ../new/"$img"; done | | | 计算sin(5)的值 | echo 's(5)' ｜ bc -l | | | 找出满足条件的文件夹,进行删除 | ls -t /var/lib/hudson/jobs ｜ xargs -I '{}' find /var/lib/hudson/jobs/'{}'/builds/ -name "*" -mtime +15 -and -not -type d -delete | | | 算md5值 | echo -n "password" ｜ md5sum | | | 列出所有本机IPv4的网络连接 | lsof -Pnl +M -i4 | | | CPU architecture information helper | cat /proc/cpuinfo; lscpu | | | 下载递归某个站点 | wget -r -p -np -k http://xxx.com/abc/ | | | 直接在控制台可以运行的, shell for循环 | for((i=0; i< 3; i++)); do { echo $i; }; done | | | 将patch.cfg中的1.3.x字符串替换成1.3.x, 1.4.x | sed -i 's/1.3.x/1.3.x, 1.4.x/g' patch.cfg | | | 将十进制转成十六进制 | bc <<< "obase=16; 11" | | | 查看机器是否为虚拟机 | dmidecode -s system-manufacturer | | | hexdump以二进制看文件 | hexdump -C -n 100 caifv-images.s3.b2c.ekaidian.com.elmar0516 | | | 查看某个文件第10字节后的20个字节 | hexdump -C -n 20 -s 10 /etc/hosts | | | grep只显示matched的string | grep --only-matching '[^ ]*fxw.vmod.cn' /usr/local/nginx/conf/vhosts/default.conf | | ** DONE linux at命令: 定义一个定时任务,在指定时间内执行 :noexport: CLOSED: [2012-05-28 Mon 18:09] #+begin_example ///6f667c06190b783231f9e708249a1d5d#$cat /tmp/test.sh echo date >>/tmp/test.log ///6f667c06190b783231f9e708249a1d5d#$/etc/init.d/atd status atd (pid 4060) is running... ///6f667c06190b783231f9e708249a1d5d#$date Mon May 28 18:05:30 CST 2012 ///6f667c06190b783231f9e708249a1d5d#$sudo at -f /tmp/test.sh -v 18:06 Mon May 28 18:06:00 2012

job 1 at 2012-05-28 18:06 ///6f667c06190b783231f9e708249a1d5d#$sudo at -f /tmp/test.sh -v 18:07 Mon May 28 18:07:00 2012

job 2 at 2012-05-28 18:07 ///6f667c06190b783231f9e708249a1d5d#$date Mon May 28 18:07:01 CST 2012 ///6f667c06190b783231f9e708249a1d5d#$cat /tmp/test.log Mon May 28 18:06:00 CST 2012 Mon May 28 18:07:00 CST 2012 #+end_example ** DONE 运行某个shell命令, 但如果$?是失败的,仍然返回失败: 通过中间变量来实现 CLOSED: [2012-09-10 一 00:50] echo http://192.168.75.236:8999/health_check/report.html && ssh [email protected] "ecae-health-check --report-dir /usr/local/ecae-health-check/report --only-simpletest"; scp [email protected]://usr/local/ecae-health-check/report/report.html /var/lib/hudson/tsung_log/health_check/report.html ** sample usage :noexport: #+begin_src python #!/usr/bin/python

CR

Copyright (c) 2008-2010 EMC Corporation

All Rights Reserved

This software contains the intellectual property of EMC Corporation

or is licensed to EMC Corporation from third parties. Use of this

software and the intellectual property contained therein is expressly

limited to the terms and conditions of the License Agreement under which

it is provided by or on behalf of EMC.

CR

author: Denny Zhang

date: 06/07/10

import sys import os import getopt import xml.dom.minidom

ELEMENT_NAME = "entry" ATTRIBUTE_KEY = "key" ATTRIBUTE_VALUE = "value"

#default indentation when adding new entry INDENTATION = "\n "

FAILURE = 1 SUCCESS = 0

""" Update xml configuration files, for the given key. """

class AtmosXMLCfg: """ AtmosXMLCfg manage maui xml configuration. Notice: The format of each entry shall be " """

####################### Public Begin #######################################
def update_entry(self, key_name, key_value):
    """
    Find first entry of <entry key=key_name value="..."/>, update the value with key_value.

    Raise exception for unexpected content of xml.
    """
    count = self.__count_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
    if count != 1:
        msg = "Find %d entries, rather than exactly 1, with the value of <%s %s=\"%s\" ...>!" % \
            (count, ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
        raise Exception(msg)

    element = self.__get_first_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, key_name)

    if element.hasAttribute(ATTRIBUTE_VALUE) == False:
        msg = "Can't find entry <%s %s=\"\" %s=\"\">!" % \
            (ELEMENT_NAME, ATTRIBUTE_KEY, ATTRIBUTE_VALUE)
        raise Exception(msg)

    element.setAttribute(ATTRIBUTE_VALUE, key_value)

def remove_entry(self, key_name):
    """
    Remove entry with the attribute of key_name.

    Raise exception for unexpected content of xml.
    """
    count = self.__count_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
    # The qualified element shall be exactly one.
    if count != 1:
        msg = "Find %d entries, rather than exactly 1, with the value of <%s %s=\"%s\" ...>!" % \
            (count, ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
        raise Exception(msg)

    element = self.__get_first_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, key_name)

    parent_element = element.parentNode
    parent_element.removeChild(element)

def add_entry(self, last_key_name, key_name, key_value, indentation):
    """
    Add an entry after the first entry which is <entry key=last_key_name value=...>

    If we fail to find the entry to append the new entry, append the entry as the last child.

    Raise exception for unexpected content of xml.
    """
    count = self.__count_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
    # When adding new entry, it shall not exists before
    if count != 0:
        msg = "Already have %d entries, with the value of <%s %s=\"%s\" ...>!" % \
            (count, ELEMENT_NAME, ATTRIBUTE_KEY, key_name)
        raise Exception(msg)

    if last_key_name == "":
        parent_element = self.get_last_entry()
        ref_child = None
    else:
        element = self.__get_first_element_by_attribute(ELEMENT_NAME, ATTRIBUTE_KEY, last_key_name)
        if element == None:
            msg = "Add entry fail: %s not found, can't add new entry." % (last_key_name)
            raise Exception(msg)
        # Get parent entry
        parent_element = element.parentNode
        ref_child = element.nextSibling

    # Construct a new entry
    new_entry = self.xml_doc.createElement(ELEMENT_NAME)
    key_attribute_node = self.xml_doc.createAttribute(ATTRIBUTE_KEY)
    value_attribute_node = self.xml_doc.createAttribute(ATTRIBUTE_VALUE)

    new_entry.setAttributeNode(key_attribute_node)
    new_entry.setAttribute(ATTRIBUTE_KEY, key_name)

    new_entry.setAttributeNode(value_attribute_node)
    new_entry.setAttribute(ATTRIBUTE_VALUE, key_value)

    # insert the new entry to the right location
    if ref_child == None:
        parent_element.appendChild(new_entry)
    else:
        parent_element.insertBefore(new_entry, ref_child)

    # append a newline and prefix of indentation to the head of the new entry
    empty_text_node = self.xml_doc.createTextNode(indentation)
    parent_element.insertBefore(empty_text_node, new_entry)

def write_xml_to_file(self, fname):
    """
    Flush xml dom object to a file.
    """
    fw = open(fname, 'w')
    fw.write(self.xml_doc.toxml())
    fw.close()

def get_last_entry(self):
    last_entry = None
    if self.xml_doc.childNodes != None:
        length = len(self.xml_doc.childNodes)
        last_entry = self.xml_doc.childNodes[length -1 ]
    return last_entry

def xml_check(self):
    """
    Check whether the xml is in the right format.
    Raise exception, if not.
    The xml shall conform the following format:
    - The xml is composed of three nodes: xml version, license, and maui properties.
    - Each child node of maui properties shall be either comment,
                 text node or element node with the format of <entry key="..." value="..."/>
    - Maui properties's children shall have no duplicate key.
    """
    # TODO, denny
    return

####################### Public End #######################################

####################### Private Begin ######################################
def __init__(self, fname):
    self.fname = fname
    self.xml_doc = xml.dom.minidom.parse(self.fname)
    self.xml_check()

def __count_element_by_attribute(self, tag_name, attribute_name, \
                                       attribute_value):
    """
    Return how many qualified elements which has attribute like the following:
           <tag_name attribute_name=attribute_value...>
    """
    count = 0
    for element in self.xml_doc.getElementsByTagName(tag_name):
        if element.hasAttribute(attribute_name) and \
                element.getAttribute(attribute_name) == attribute_value:
            count = count + 1
    return count

def __get_first_element_by_attribute(self, tag_name, attribute_name, \
                                           attribute_value):
    """
    Return the first element which has attribute like the following:
           <tag_name attribute_name=attribute_value...>
    """
    ret = None
    for element in self.xml_doc.getElementsByTagName(tag_name):
        if element.hasAttribute(attribute_name) and \
                element.getAttribute(attribute_name) == attribute_value:
            ret = element
    return ret
####################### Private End #######################################

def pre_check(fname, action, after_key, key, value, indentation): """ Input parameter check """

if fname == "":
    msg = "ERROR: Mandatory setting of file are not given"
    print msg
    return False

if action not in ("add", "remove", "update"):
    if action == "":
        msg = "ERROR: Mandatory setting of action are not given"
    else:
        msg = "Action of %s is not supported." % (action)
    print msg
    return False

if key == "":
    msg = "ERROR: Mandatory setting of key are not given"
    print msg
    return False

if action in ("update", "add"):
    if value == "":
        msg = "ERROR: For operation of %s, mandatory setting of value are not given" % (action)
        print msg
        return False

return True

def do_operation(fname, action, after_key, keys, values, indentation=INDENTATION): """ Perform the operation on maui xml configuration. """ xml_cfg = AtmosXMLCfg(fname)

if action == "update":
    for k, v in zip(keys, values):
        xml_cfg.update_entry(k, v)

if action == "remove":
    for k in keys:
        xml_cfg.remove_entry(k)

if action == "add":
    for k, v in zip(keys, values):
        xml_cfg.add_entry(after_key, k, v, indentation)

xml_cfg.write_xml_to_file(fname)

def usage(): _usage = """Usage: %s <OPTION> [SETTING] Update xml configuration files, for the given key.

OPTION: -h --help help info -a --action Action what will be performed. Supported options are add, remove, update -f --file The file location of xml -k --key The name of key -v --value The value of key --after-key It indicates after which entry the new entry is appended. If option is not given, the new entry will be appened at the end of maui section. -i --indentation Append indentation in the new entry, when adding entry. Default value is "\n ". Example:

Add certain key, after certain key.

%s --action='add' --file='/etc/maui/cm_cfg.xml' --after-key='key' --key='key1' --value='value2'
Add certain key, after certain key, with the prefix of "\n ".

%s --action='add' --file='/etc/maui/cm_cfg.xml' --after-key='key' --key='key1' --value='value2' --indentation=$'\n '
Update certain key

%s --action='update' --file='/etc/maui/cm_cfg.xml' --key='key1' --value='value2'
Remove certain key

%s --action='remove' --file='/etc/maui/cm_cfg.xml' --key='key1'

Report bugs to Maui System Management Team. """ _name_filler = (os.path.basename(sys.argv[0]),) * 5

print _usage %_name_filler

if name=='main': # Get options try: opts, args = getopt.getopt(sys.argv[1:], "ha:k:v:f:i:", ["help", "action=", "file=", "key=", "value=", "after-key=", "indentation=" ]) except getopt.GetoptError, err: print str(err) usage() sys.exit(1)

#set default value
action = ""
fname = ""
after_key = ""
key = ""
value = ""
indentation = INDENTATION

# Parse options
for o, a in opts:
    if o in ("-h", "--help"):
        usage()
        sys.exit(0)
    elif o in ("-a", "--action"):
        action = a
    elif o in ("--after-key"):
        after_key = a
    elif o in ("-f", "--file"):
        fname = a
    elif o in ("-k", "--key"):
        key = a
    elif o in ("-v", "--value"):
        value = a
    elif o in ("-i", "--indentation"):
        indentation = a
    else:
        print("Invalid option: " + o)
        usage()
        sys.exit(FAILURE)

# Input parameter check
if pre_check(fname, action, after_key, [key], [value], indentation) == False:
    usage()
    sys.exit(FAILURE)

try:
    do_operation(fname, action, after_key, key, value, indentation)
    print "Operation succeeds."
except Exception, msg:
    print "Operation fails."
    print msg
    sys.exit(FAILURE)

sys.exit(SUCCESS)

#+end_src

TODO shell: create normal users with root priveledge :noexport: :PROPERTIES: :ID: 1B974171-7A28-419E-8113-36D6B5627B4D :END: /home/denny/devstack/stack.sh #+begin_src sh

since this script runs as a normal user, we need to give that user

ability to run sudo

dpkg -l sudo || apt_get update && apt_get install sudo

if ! getent passwd stack >/dev/null; then echo "Creating a user called stack" useradd -U -G sudo -s /bin/bash -d $DEST -m stack fi

echo "Giving stack user passwordless sudo priviledges"

some uec images sudoers does not have a '#includedir'. add one.

grep -q "^#includedir.*/etc/sudoers.d" /etc/sudoers || echo "#includedir /etc/sudoers.d" >> /etc/sudoers ( umask 226 && echo "stack ALL=(ALL) NOPASSWD:ALL"
> /etc/sudoers.d/50_stack_sh ) #+end_src
TODO shell: special matching syntax :noexport: :PROPERTIES: :ID: 89C8040D-8568-4B78-BD1E-5938522A4D64 :END: #+begin_src sh function is_service_enabled() { services=$@ for service in ${services}; do [[ ,${ENABLED_SERVICES}, =~ ,${service}, ]] && return 0 [[ ${service} == "nova" && ${ENABLED_SERVICES} =~ "n-" ]] && return 0 [[ ${service} == "glance" && ${ENABLED_SERVICES} =~ "g-" ]] && return 0 [[ ${service} == "quantum" && ${ENABLED_SERVICES} =~ "q-" ]] && return 0 done return 1 } #+end_src
--8<-------------------------- separator ------------------------>8-- :noexport:
echo :noexport: | Question | Solution | |------------------+---------------------------------------------------------| | 使用红色输出内容 | echo -en hello, '\E[47;31m'"\033[1mdenny\033[0m" | | 输出多行 | echo -e "hello,\ndenny" | | 加粗显示 | echo -e hello, "\033[1mThis is bold text.\033[0m" | | 下划线 | echo -e hello, "\033[4mThis is underlined text.\033[0m" |
grep :noexport: | Question | Solution | Use case | |-----------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+----------| | 向文件中追加一个key=value的键值对, 如果存在则修改它 | (grep "key1 +=" ./test.cfg && sudo sed -i 's/key1 +=.*/key1 = value1/' ./test.cfg) ｜｜ ((cat ./test.cfg; echo "key1 = value1") ｜ sudo tee ./test.cfg) | | | 按字符串搜索查找某个文件夹 | grep -lir "some text" * | | | show line number | grep -in null myfile.txt | | ** DONE grep a directory: grep -R "dennyzhang" . CLOSED: [2016-04-11 Mon 20:32] http://stackoverflow.com/questions/4121803/how-can-i-use-grep-to-find-a-word-inside-a-folder http://www.cyberciti.biz/faq/grep-subdirectory-for-files-on-linux-bsd-osx-unix-oses/ ** DONE grep two words with or: grep -iE "word1|word2|word3" a.txt CLOSED: [2015-02-09 Mon 19:17] ** TODO why grep is so fast :PROPERTIES: :ID: 41F3231C-F7D5-4E4D-AE50-E18DCBC46AEE :END: http://xboss.diandian.com/post/2011-08-19/4094774 http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://news.ycombinator.com/item?id=2393587 #+begin_example denny@denny-Vostro-1014:~/backup/pirate$ ls -lth ./xp.iso -rwxrwxr-x. 1 denny denny 687M Sep 25 09:53 ./xp.iso denny@denny-Vostro-1014:~/backup/pirate$ time grep "test" xp.iso Binary file xp.iso matches

real 0m0.197s user 0m0.012s sys 0m0.020s denny@denny-Vostro-1014:~/backup/pirate$ time grep "hello" xp.iso

real 0m10.211s user 0m0.752s sys 0m0.748s denny@denny-Vostro-1014:~/backup/pirate$ time grep "test" xp.iso Binary file xp.iso matches

real 0m0.046s user 0m0.016s sys 0m0.000s denny@denny-Vostro-1014:~/backup/pirate$ time grep "world" xp.iso Binary file xp.iso matches

real 0m0.023s user 0m0.012s sys 0m0.008s denny@denny-Vostro-1014:~/backup/pirate$ time grep "microsoft" xp.iso Binary file xp.iso matches

real 0m0.008s user 0m0.004s sys 0m0.000s #+end_example ** DONE grep a text file which contains some binary data: grep -a(--text): CLOSED: [2016-10-08 Sat 11:37] man grep

File and Directory Selection -a, --text Process a binary file as if it were text; this is equivalent to the --binary-files=text option.

http://stackoverflow.com/questions/9988379/how-to-grep-a-text-file-which-contains-some-binary-data root@wilson-sandbox-qa:~# tail -n 200000 /root/test.log | grep -C 3 password tail -n 200000 /root/test.log | grep -C 3 password Binary file (standard input) matches

ps :noexport: ** DONE Get process start time: ps -o lstart= -p 26351 CLOSED: [2014-11-28 Fri 15:40] root@fluig-id-cust-01:/data/fluigidentity-logs# ps -o lstart= -p 26351 ps -o lstart= -p 26351 Wed Nov 26 06:18:22 2014 ** DONE linux sort process by memory: ps aux --sort -rss CLOSED: [2014-07-12 Sat 11:02] ** DONE sort process: ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | sort -nrk5 CLOSED: [2016-01-28 Thu 16:08] ** DONE [#B] linux get memory landscape quickly: sortmem: ps aux | sort -nk +4 | tail CLOSED: [2014-08-21 Thu 15:19]
xargs :noexport: | Question | Solution | Use case | |-----------------------------------+--------------------------------------------------------------------+----------| | 对满足条件的文件, 进行按内容查找 | find /etc/yum.repos.d/ -name ".repo" -type f ｜ xargs grep "http" | | | 对满足条件的文件, 进行匹配重命名 | find -iname ".pdf" ｜ xargs -I '{}' mv '{}.tmp' '{}.tmp2' | |
awk :noexport:

awk syntax: awk 'pattern{action}' file | Question | Solution | Use case | |---------------------------------------------------------------------+-------------------------------------------------------------------+----------| | 将文件按行反序 | awk '{ a[NR]=$0 } END { for(i=NR; i; --i) print a[i] } ' datafile | | | 查找/etc/hosts中, 以"# The"开头的内容, 并将该行及之后的内容打印出来 | awk "/^# The/{a=1} {if(a) print $1}" /etc/hosts | | | 显示文件的第一列 | awk '{print $1}' | | | 反序显示文件的前两列 | awk '{print $2,"\t",$1}' | | | 输出前两列的总和 | awk '{print $1 + $2}' | | | 查找所有包括"money" 行并输出最后一列 | awk '/money/ {print $NF}' | | | 查找第二列中包含 "money" | awk '$2 ~ /money/ {print $0}' | | | 查找第三列中不包括"A" | awk '$3 !~ /A$/ {print $0}' | | | 从文件找出太长的行,打印出来,并给出行号 | awk -F' ' 'length>100{print NR $0}' ./filebat.txt | | ** basic use #+begin_example awk的处理文本和数据的方式是这样的,它逐行扫描文件,从第一行到最后一行, 寻找匹配的特定模式的行,并在这些行上进行你想要的操作.如果没有指定处理动作,则把匹配的行显示到标准输出(屏幕),如果没有指定模式,则所有被操作所指定的行都被处理.

awk的语法有两种形式:

awk [options] 'script' var=value file(s)
awk [options] -f scriptfile var=value file(s) #+end_example ** useful link http://hi.baidu.com/beibeiboo/blog/item/48906fce7cb5cb2cb600c892.html\ AWK详细参考(转载整理) http://en.wikipedia.org/wiki/AWK\ awk ** DONE awk filter row and manage column CLOSED: [2013-12-30 Mon 19:04] https://review.openstack.org/#/c/61904/3/bin/heat-keystone-setup keystone user-list | awk -F'|' "NR == 2 && /^|/ { for (i=2; i<NF; i++) if ($i ~ "^ *name *$") print (i - 1) }" ** DONE awk show rest of columns CLOSED: [2016-06-17 Fri 11:13] http://stackoverflow.com/questions/2961635/using-awk-to-print-all-columns-from-the-nth-to-the-last

awk -F' ' '{$1=$2=""; print $0;}' /etc/hosts

awk '{out=""; for(i=3;i<=NF;i++){out=out" "$i}; print out}' /etc/hosts awk '{for(i=3;i<=NF;++i) print $i}' /etc/hosts ** DONE 乱序排列一个文件: awk 'BEGIN{srand()}{b[rand()NR]=$0}END{for(x in b)print b[x]}' ./data.sql CLOSED: [2014-03-22 Sat 17:41] ** DONE linux shell remove tailing \n: awk 1 ORS='' file CLOSED: [2014-12-10 Wed 22:22] ** DONE bash: extract entire block that starts from "begin" till "end": awk '/begin/,/end/' text. CLOSED: [2015-10-13 Tue 10:29] http://askubuntu.com/questions/551338/how-do-i-grep-for-multiple-patterns-on-multiple-lines ** DONE awk: group entries by fields and do the sum by group CLOSED: [2015-06-26 Fri 21:31] cat test.txt | grep '^[1-9]' | awk '{a[substr($4,2,17)] += $10;}END{for(x in a)print x" "a[x];}'

sort :noexport: ** DONE sort by second column: sort -k 2 file.txt CLOSED: [2017-10-17 Tue 22:39] https://stackoverflow.com/questions/4262650/linux-shell-sort-file-according-to-the-second-column
role model: https://github.com/alexanderepstein/Bash-Snippets :noexport:
how to use pushd :noexport:
cron :noexport: ** DONE manually start cron: cron CLOSED: [2016-10-24 Mon 11:42] ** DONE linux list all crontab CLOSED: [2016-10-23 Sun 16:58] 1.for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

/etc/crontab*

http://stackoverflow.com/questions/134906/how-do-i-list-all-cron-jobs-for-all-users http://unix.stackexchange.com/questions/7053/how-can-get-a-list-of-all-scheduled-cron-jobs-on-my-machine http://www.cyberciti.biz/faq/linux-show-what-cron-jobs-are-setup/

#+BEGIN_EXAMPLE Depending on how your linux system is set up, you can look in:

/var/spool/cron/* (user crontabs) /etc/crontab (system-wide crontab) also, many distros have:

/etc/cron.d/* These configurations have the same syntax as /etc/crontab /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly These are simply directories that contain executables that are executed hourly, daily, weekly or monthly, per their directory name.

On top of that, you can have at jobs (check /var/spool/at/), anacron (/etc/anacrontab and /var/spool/anacron/) and probably others I'm forgetting. #+END_EXAMPLE ** DONE Linux create crontab, without editor CLOSED: [2016-10-23 Sun 17:10] cat > /etc/cron.d/crontab_test << EOF PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

Activity reports every 10 minutes everyday

*/5 * * * * root echo "hello, world" >> /tmp/test.log

Additional run at 23:59 to rotate the statistics file

EOF

"crontab -l" won't find it

watch "tail /tmp/test.log" ** DONE crontab format: (Min, Hour, DayOfMonth, Month, DayOfWeek, Command) CLOSED: [2017-01-10 Tue 10:41] http://www.nncron.ru/help/EN/working/cron-format.htm

https://www.pantz.org/software/cron/croninfo.html #+BEGIN_EXAMPLE

Minute Hour Day of Month Month Day of Week Command

(0-59) (0-23) (1-31) (1-12 or Jan-Dec) (0-6 or Sun-Sat)

0        2          12             *                *            /usr/bin/find

#+END_EXAMPLE

String :noexport: | Name | Comment | |-----------------+---------| | ${var%pattern} | | | ${var%%pattern} | | | ${var#pattern} | | | ${var##pattern} | | | ${#var} | |
--8<-------------------------- separator ------------------------>8-- :noexport:
HALF add shell example :noexport: #+BEGIN_SRC sh #!/usr/bin/env bash

@copyright 2018 DennyZhang.com

Licensed under MIT

https://www.dennyzhang.com/wp-content/mit_license.txt

File: hello

Author : Denny https://www.dennyzhang.com/contact

Description :

--

Created : <2018-09-20>

Updated: Time-stamp: <2018-09-20 10:41:04>

##------------------------------------------------------------------- set -e

function ensure_variable_isset { var=${1?} message=${2:-"parameter name should be given"} # TODO support sudo, without source if [ -z "$var" ]; then echo "Error: Certain variable($message) is not set" exit 1 fi }

File: hello ends

#+END_SRC

HALF shell scripts :noexport:

To implement a for loop:

for file in *; do echo $file found; done

To implement a case command:

case "$1" in 0) echo "zero found";; 1) echo "one found";; 2) echo "two found";; 3*) echo "something beginning with 3 found";; esac

Turn on debugging:

set -x

Turn off debugging:

set +x

Retrieve N-th piped command exit status

printf 'foo' | fgrep 'foo' | sed 's/foo/bar/' echo ${PIPESTATUS[0]} # replace 0 with N

Lock file:

( set -o noclobber; echo > my.lock ) || echo 'Failed to create lock file'

HALF make shell function smaller :noexport: if [ -z "$GET_CREDENTIALS_HOOK" ]; then echo "GET_CREDENTIALS_HOOK is required" exit 1 fi
more content :noexport: ** ab

send 100 requests with a concurency of 50 requests to an URL

ab -n 100 -c 50 http://www.example.com/

send requests during 30 seconds with a concurency of 50 requests to an URL

ab -t 30 -c 50 URL http://www.example.com/ ** sort

To sort a file:

sort file

To sort a file by keeping only unique:

sort -u file

To sort a file and reverse the result:

sort -r file

To sort a file randomly:

sort -R file ** split

To split a large text file into smaller files of 1000 lines each:

split file.txt -l 1000

To split a large binary file into smaller files of 10M each:

split file.txt -b 10M

To consolidate split files into a single file:

cat x* > file.txt ** piplines

To redirect stderr to stdout:

some-command 2>&1

To redirect stderr to a file

some-command 2> errors.txt ** cut

To cut out the third field of text or stdoutput that is delimited by a #:

cut -d# -f3 ** date

Print date in format suitable for affixing to file names

date +"%Y%m%d_%H%M%S"

Convert Unix timestamp to Date(Linux)

date -d @1440359821

Convert Unix timestamp to Date(Mac)

date -r 1440359821 ** tree

To display a recursive directory tree

tree

To make tree output contents from path `/foo/bar`

tree /foo/bar

To make tree omit any empty directories from the output

tree --prune

To list directories only (`-d`), and at a max depth of two levels (`-L`)

tree -d -L 2 ** truncate

To clear the contents from a file:

truncate -s 0 file.txt

To truncate a file to 100 bytes:

truncate -s 100 file.txt

To truncate a file to 100 KB:

truncate -s 100K file.txt

(M, G, T, P, E, Z, and Y may be used in place of "K" as required.)

** uname

Print all system information

uname -a

Linux system-hostname 3.2.0-4-amd64 #1 SMP Debian 3.2.32-1 x86_64 GNU/Linux

Print the hostname

uname -n

system-hostname

Print the kernel release

uname -r

3.2.0-4-amd64

Print the kernel version, with more specific information

uname -v

#1 SMP Debian 3.2.32-1

Print the hardware instruction set

uname -m

x86_64

Print the kernel name

uname -s

Linux

Print the operating system

uname -o

GNU/Linux

** uniq

show all lines without duplication

`sort -u` and `uniq` is the same effect.

sort file | uniq

show not duplicated lines

sort file | uniq -u

show duplicated lines only

sort file | uniq -d

count all lines

sort file | uniq -c

count not duplicated lines

sort file | uniq -uc

count only duplicated lines

sort file | uniq -dc ** top

Update every samples:

top -i

Set the delay between updates to seconds:

top -s

Set event counting to accumulative mode:

top -a

Set event counting to delta mode:

top -d

Set event counting to absolute mode:

top -e

Do not calculate statistics on shared libraries, also known as frameworks:

top -F

Calculate statistics on shared libraries, also known as frameworks (default):

top -f

Print command line usage information and exit:

top -h

Order the display by sorting on in descending order

top -o ** tr #replace : with new line echo $PATH|tr ":" "\n" #equivalent with: echo $PATH|tr -t ":" \n

#remove all occurance of "ab" echo aabbcc |tr -d "ab" #ouput: cc

#complement "aa" echo aabbccd |tr -c "aa" 1 #output: aa11111 without new line #tip: Complement meaning keep aa,all others are replaced with 1

#complement "ab\n" echo aabbccd |tr -c "ab\n" 1 #output: aabb111 with new line

#Preserve all alpha(-c). ":-[:digit:] etc" will be translated to "\n". sequeeze mode. echo $PATH|tr -cs "[:alpha:]" "\n"

#ordered list to unordered list echo "1. /usr/bin\n2. /bin" |tr -cs " /[:alpha:]\n" "+" ** rm

Remove files and subdirs

rm -rf path/to/the/target/

Ignore non existent files

rm -f path/to/the/target

Remove a file with his inode

find /tmp/ -inum 6666 -exec rm -i '{}' ; ** rename

Lowercase all files and folders in current directory

rename 'y/A-Z/a-z/' * ** csplit

Split a file based on pattern

csplit input.file '/PATTERN/'

Use prefix/suffix to improve resulting file names

csplit -f 'prefix-' -b '%d.extension' input.file '/PATTERN/' '{*}' ** bash

To implement a for loop:

for file in *; do echo $file found; done

To implement a case command:

case "$1" in 0) echo "zero found";; 1) echo "one found";; 2) echo "two found";; 3*) echo "something beginning with 3 found";; esac

Turn on debugging:

set -x

Turn off debugging:

set +x

Retrieve N-th piped command exit status

printf 'foo' | fgrep 'foo' | sed 's/foo/bar/' echo ${PIPESTATUS[0]} # replace 0 with N

Lock file:

( set -o noclobber; echo > my.lock ) || echo 'Failed to create lock file' ** grep

Search a file for a pattern

grep pattern file

Case insensitive search (with line numbers)

grep -in pattern file

Recursively grep for string in folder:

grep -R pattern folder

Read search patterns from a file (one per line)

grep -f pattern_file file

Find lines NOT containing pattern

grep -v pattern file

You can grep with regular expressions

grep "^00" file #Match lines starting with 00 grep -E "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" file #Find IP add

Find all files which match {pattern} in {directory}

This will show: "file:line my research"

grep -rnw 'directory' -e "pattern"

Exclude grep from your grepped output of ps.

Add [] to the first letter. Ex: sshd -> [s]shd

ps aux | grep '[h]ttpd'

Colour in red {bash} and keep all other lines

ps aux | grep -E --color 'bash|$' ** hardware

Display all hardware details

sudo lshw

List currently loaded kernel modules

lsmod

List all modules available to the system

find /lib/modules/$(uname -r) -type f -iname "*.ko"

Load a module into kernel

modprobe modulename

Remove a module from kernel

modprobe -r modulename

List devices connected via pci bus

lspci

Debug output for pci devices (hex)

lspci -vvxxx

Display cpu hardware stats

cat /proc/cpuinfo

Display memory hardware stats

cat /proc/meminfo

Output the kernel ring buffer

dmesg

Ouput kernel messages

dmesg --kernel ** indent

format C/C++ source according to the style of Kernighan and Ritchie (K&R), no tabs, 3 spaces per indent, wrap lines at 120 characters.

indent -i3 -kr -nut -l120 ** iconv

To convert file (iconv.src) from iso-8859-1 to utf-8 and save to

/tmp/iconv.out

iconv -f iso-8859-1 -t utf-8 iconv.src -o /tmp/iconv.out ** history

To see most used top 10 commands:

history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl | head -n10 ** readline Moving around Ctrl-b Move the cursor one character ⇦ to the left Ctrl-f Move the cursor one character ⇨ to the right Alt-b Move the cursor one word ⇦ to the left Alt-f Move the cursor one word ⇨ to the right Ctrl-a Move the cursor ⇤ to the start of the line Ctrl-e Move the cursor ⇥ to the end of the line Ctrl-x-x Move the cursor ⇤⇥ to the start, and to the end again

Cut, copy and paste

Backspace Delete the character ⇦ to the left of the cursor DEL Ctrl-d Delete the character underneath the cursor Ctrl-u Delete everything ⇤ from the cursor back to the line start Ctrl-k Delete everything ⇥ from the cursor to the end of the line Alt-d Delete word ⇨ until before the next word boundary Ctrl-w Delete word ⇦ until after the previous word boundary Ctrl-y Yank/Paste prev. killed text at the cursor position Alt-y Yank/Paste prev. prev. killed text at the cursor position

History

Ctrl-p Move in history one line ⇧ before this line Ctrl-n Move in history one line ⇩ after this line Alt-> Move in history all the lines ⇩ to the line currently being entered Ctrl-r Incrementally search the line history ⇧ backwardly Ctrl-s Incrementally search the line history ⇩ forwardly Ctrl-J End an incremental search Ctrl-G Abort an incremental search and restore the original line Alt-Ctrl-y Yank/Paste arg. 1 of prev. cmnd at the cursor position Alt-. Alt-_ Yank/Paste last arg of prev. cmnd at the cursor position

Undo

Ctrl-_ Ctrl-x Ctrl-u Undo the last editing command; you can undo all the way back to an empty line Alt-r Undo all changes made to this line Ctrl-l Clear the screen, reprinting the current line at the top Ctrl-l Clear the screen, reprinting the current line at the top

Completion

TAB Auto-complete a name Alt-/ Auto-complete a name (without smart completion) Alt-? List the possible completions of the preceeding text Alt-* Insert all possible completions of the preceeding text

Transpose

Ctrl-t Transpose/drag char. before the cursor ↷ over the character at the cursor Alt-t Transpose/drag word before the cursor ↷ over the word at/after the cursor ** tail -n +N file

To show the last lines of file starting with the Nth

tail -n +N file ** pgrep

Get a list of PIDs matching the pattern

pgrep example

Kill all PIDs matching the pattern

pgrep -f example | xargs kill ** ping

ping a host with a total count of 15 packets overall.

ping -c 15 www.example.com

ping a host with a total count of 15 packets overall, one every .5 seconds (faster ping).

ping -c 15 -i .5 www.example.com

test if a packet size of 1500 bytes is supported (to check the MTU for example)

ping -s 1500 -c 10 -M do www.example.com

get all ipv6 neighbors via broadcast ping

ping6 -I eth0 ff02::1 ** pkill

To kill a process using it's full process name

pkill

To kill a process by it's partial name

pkill -f ** popd & pushd

Pushes your current directory to the top of a stack while changing to the specified directory

pushd

To return use popd

popd ** numfmt

Convert bytes to Human readable format

numfmt --to=iec --suffix=B --padding=7 1048576 ** strace

Basic stracing

strace

save the trace to a file

strace -o strace.out

follow only the open() system call

strace -e trace=open

follow all the system calls which open a file

strace -e trace=file

follow all the system calls associated with process

management

strace -e trace=process

follow child processes as they are created

strace -f

count time, calls and errors for each system call

strace -c

trace a running process (multiple PIDs can be specified)

strace -p ** su

Switch to another user account

su USERNAME ** mv

Move a file from one place to another

mv ~/Desktop/foo.txt ~/Documents/foo.txt

Move a file from one place to another and automatically overwrite if the destination file exists

(This will override any previous -i or -n args)

mv -f ~/Desktop/foo.txt ~/Documents/foo.txt

Move a file from one place to another but ask before overwriting an existing file

(This will override any previous -f or -n args)

mv -i ~/Desktop/foo.txt ~/Documents/foo.txt

Move a file from one place to another but never overwrite anything

(This will override any previous -f or -i args)

mv -n ~/Desktop/foo.txt ~/Documents/foo.txt

Move listed files to a directory

mv -t ~/Desktop/ file1 file2 file3 ** lsof

List all IPv4 network files

sudo lsof -i4

List all IPv6 network files

sudo lsof -i6

List all open sockets

lsof -i

List all listening ports

lsof -Pnl +M -i4

Find which program is using the port 80

lsof -i TCP:80

List all connections to a specific host

lsof [email protected]

List all processes accessing a particular file/directory

lsof </path/to/file>

List all files open for a particular user

lsof -u

List all files/network connections a command is using

lsof -c

List all files a process has open

lsof -p

List all files open mounted at /mount/point.

Particularly useful for finding which process(es) are using a

mounted USB stick or CD/DVD.

lsof +f -- </mount/point>

See this primer: http://www.danielmiessler.com/study/lsof/

for a number of other useful lsof tips

** ln

To create a symlink:

ln -s path/to/the/target/directory name-of-symlink

Symlink, while overwriting existing destination files

ln -sf /some/dir/exec /usr/bin/exec ** ls

Displays everything in the target directory

ls path/to/the/target/directory

Displays everything including hidden files

ls -a

Displays all files, along with the size (with unit suffixes) and timestamp

ls -lh

Display files, sorted by size

ls -S

Display directories only

ls -d */

Display directories only, include hidden

ls -d .*/ */ ** wc

Count the number of words (file or STDIN)

wc -w /path/to/foo.txt cat /path/to/foo.txt | wc -w

Count the number of lines (file or STDIN)

wc -l /path/to/foo.txt cat /path/to/foo.txt | wc -l

Count the number of bytes (file or STDIN)

wc -c /path/to/foo.txt cat /path/to/foo.txt | wc -c

Count files and directories at a given location

ls -l | wc -l

If you ever use `wc` in a shell script and need to compare the output with an int you can

clean the output (wc returns extra characters around the integer) by using xargs:

ls -l | wc -l | xargs ** [#A] xargs

find all file name ending with .pdf and remove them

find -name *.pdf | xargs rm -rf

if file name contains spaces you should use this instead

find -name *.pdf | xargs -I{} rm -rf '{}'

Will show every .pdf like:

&toto.pdf=

&titi.pdf=

-n1 => One file by one file. ( -n2 => 2 files by 2 files )

find -name *.pdf | xargs -I{} -n1 echo '&{}='

If find returns no result, do not run rm

This option is a GNU extension.

find -name "*.pdf" | xargs --no-run-if-empty rm ** awk

sum integers from a file or stdin, one integer per line:

printf '1\n2\n3\n' | awk '{ sum += $1} END {print sum}'

using specific character as separator to sum integers from a file or stdin

printf '1:2:3' | awk -F ":" '{print $1+$2+$3}'

print a multiplication table

seq 9 | sed 'H;g' | awk -v RS='' '{for(i=1;i<=NF;i++)printf("%dx%d=%d%s", i, NR, i*NR, i==NR?"\n":"\t")}'

Specify output separator character

printf '1 2 3' | awk 'BEGIN {OFS=":"}; {print $1,$2,$3}'

scripts :noexport: ** DONE linux shell unset all varable :noexport: CLOSED: [2015-01-06 Tue 16:11]

env -i bash command cleared all the environment variables to default on login. This won't work in bash
unset $(env | awk -F= '{print $1}')

unset $(env | awk -F= '{print $1}' | grep '-')

** DONE Difference between Login Shell and Non-Login Shell? CLOSED: [2015-01-11 Sun 12:25] http://unix.stackexchange.com/questions/38175/difference-between-login-shell-and-non-login-shell http://superuser.com/questions/636219/where-should-i-export-an-environment-variable-so-that-all-combinations-of-bash-d

A login shell is the first process that executes under your user ID when you log in for an interactive session. When you log in on a text console, or through SSH, or with su -, you get an interactive login shell.
It's rare to run a non-interactive login shell, but some X settings do that when you log in with a display manager, so as to arrange to read the profile files. Other settings (this depends on the distribution and on the display manager) read /etc/profile and ~/.profile explicitly, or don't read them.
When you start a shell in a terminal in an existing session (screen, X terminal, Emacs terminal buffer, a shell inside another, ...), you get an interactive, non-login shell. That shell might read a shell configuration file (~/.bashrc for bash, /etc/zshrc and ~/.zshrc for zsh, /etc/csh.cshrc and ~/.cshrc for csh, etc.). A shell started in a new terminal in a GUI would be an interactive non-login shell. It would source your .bashrc, but not your .profile, for example.

In summary they says (there is more detail in the man page, you should read it):

| When bash is | it reads | |-----------------------------+----------------------------------------------------------------------------------------------| | login shell | /etc/profile and then the first of ~/.bash_profile, ~/.bash_login or ~/.profile that exists. | | interactive non-login shell | /etc/bash.bashrc then ~/.bashrc | | non-interactive shell | The contents of $BASH_ENV (if it exists) | | interactive (as "sh") | The contents of $ENV (if it exists) |

| When dash is | it reads | |-------------------+----------------------------------------------------------------------------------------------| | login shell | /etc/profile then .profile | | interactive shell | The contents of ENV (it it exists, can be set in .profile as well as in initial environment) |

** DONE linux run command as some user in bash: CLOSED: [2014-12-13 Sat 22:03] sudo su mac <<'EOF' ls -lth | head whoami EOF

** DONE linux check whether directory is empty: [ "$(ls -A /path/to/directory)" ] && echo "Not Empty" || echo "Empty" CLOSED: [2014-10-01 Wed 13:26]

** DONE bash: check if a string has spaces CLOSED: [2014-11-09 Sun 05:39] #+begin_example [[ "127.0.0.1 192.168.1.2" = *\ * ]] #+end_example

** DONE linux向尾部追回一串字符串 CLOSED: [2014-01-22 Wed 11:46] *** 通过EOF #+begin_example cat <<EOF >> outputfile some lines of text EOF #+end_example *** 通过cat #+begin_example echo "line 1 line 2" >> greetings.txt #+end_example *** 测试数据 #+begin_example

Zabbix Application PPA

deb http://ppa.launchpad.net/tbfr/zabbix/ubuntu precise main deb-src http://ppa.launchpad.net/tbfr/zabbix/ubuntu precise main #+end_example ** timeout https://github.com/cloudfoundry-incubator/kubo-release/blob/a5e799be8f34609ea1bd82174d7ada9e69a0c6d5/jobs/kubelet/templates/bin/post-start.erb#L33-L42 ** TODO zsh delete by words, instead of characters

TODO send stdout/stderr to different file :noexport: https://github.com/cloudfoundry-incubator/cfcr-etcd-release/blob/5dda9b05123b83b0ef39fe1ba8b3f4830d1adb51/jobs/acceptance-tests/templates/run.sh.erb#L15-L18
TODO pushd/popd :noexport: #+BEGIN_EXAMPLE pushd $pkgname wget -qO- https://storage.googleapis.com/pks-releases/${f}.tar.gz | tar xzvf - popd

#+END_EXAMPLE

TODO cheatsheet https://jimmysong.io/cheatsheets/bash :noexport:
TODO bash avoid blind wait :noexport: timeout 10 sh -c "ls /tmp/a.txt" timeout 100 "ls /tmp/a.txt"

touch /tmp/a.txt ** mac install timeout https://stackoverflow.com/questions/3504945/timeout-command-on-mac-os-x

TODO find ip from hostname :noexport: 89b2a4c2-18d4-49ea-97a2-dea15b4aaee6.internal
--8<-------------------------- separator ------------------------>8-- :noexport:
TODO consolidate: https://bitbucket.org/devops_sysops/cheatsheetcollection/src/a4b5d9acc0a852254a2eb8719068f9361d99e426/Scripts/List.md?fileviewer=file-view-default :noexport:
TODO consolidate: http://www.geekmind.net/2011/01/shortcuts-to-improve-your-bash-zsh.html :noexport:
--8<-------------------------- separator ------------------------>8-- :noexport:
TODO Difference between the varieties of shell: sh, ksh, bash, zsh :noexport:
TODO Difference between xargs vs tee :noexport:
--8<-------------------------- separator ------------------------>8-- :noexport:
TODO [#A] doc: What happens if press Ctrl+C :noexport: https://stackoverflow.com/questions/45993444/in-detail-what-happens-when-you-press-ctrl-c-in-a-terminal

https://www.1point3acres.com/bbs/forum.php?mod=viewthread&tid=554762&ctid=228784

本来投的sde,然后hr联系就也投了pe,可是因为转专业系统的知识之前基本上没有,临时恶补了一周就去面了问题

以下内容需要积分高于 100 您已经可以浏览

ctrl c 之后发生了什么？
troubleshooting,如果发现系统运行很慢,有可能是io block该怎么排查,用哪些命令

基本上是45分钟两道题,每道题面试官会跟着你的思路随时提各种相关问题.第一题没有准备到不太会,瞎猜了一些,比如parent process（也就是bash,因为感觉ctrl c 是发给bash的指令）会先发一个system signal给kernel之类的..面试官一直在引导,但是因为就是不会..感觉面试官也能看出来,第二题是之前有准备过的,具体命令有vmstat, ps aux,strace等等.

HALF [#A] doc: What happens if you run "ls -l *" in terminal :noexport:

OS kernels get i/o interrupt. Then translate into a shell commmand

shell read the command vig getline() from STDIN
shell do the expansion and alias Expand * to all files under current directory
For non built-in command, shell lookup for "ls" binary from $PATH
-l feeds into ls binary, which list all atributes of the files/directories as a list
shell run fork() to create a child process The chlid process inherits parent process's settings
Child process calls execve() to run the user's command execve() does three things: OS stops the duplicated process, loads up the new program, and starts it
Parent process waits until the child completes the execution via wait()
Since the default stdout/stderr are the terminal, the output displays to the terminal
Once it's done, the child process terminates and control will be returned back to the parent process ** Q: entities OS kernel shell parent process child process ** Q: Does * translatation happen in shell or ls? ** Q: how child process gets created? ** notes #+BEGIN_EXAMPLE what happend if you type "ls -l *" and hit enter in shell? 1.Get user input: when entering shell, you will encounter the prompt. After typing "ls -l" to the command line, shell reads the command using getline() from the standard input file stream, STDIN, and stores the input in a buffer as a string. Then buffer is broken down into tokens and stored in an array: {"ls", "-l", "NULL"}.

Check for expansion and alias: before searching for "ls" command, shell will check for special characters need to be expanded (e.g. , $ etc). For example, if input contains ".c", shell will replace "*.c" with all .c files within current directory. Besides, shell checks for alias. If the command is an alias, shell will replace the command with its corresponding value.
Check built-in: if "ls" is not an alias. Shell will check whether the command is a built-in command. A built-int command is a command that is built into and executed in shell itself. 4.Check PATH: if "ls" is not a built-in command, shell will look for a program file called "ls" in environment variable, PATH. PATH is a list of directories the shell searches each time a command is entered. The PATH variable is a key value pair delimited by "=". A copy of PATH will be tokenized by colon ":". Then "/ls" is appended to the end of each token. After searching through each token, an error will be returned if "ls" command is not existing. 5.Fork and execute program in the child process: after locating the "ls" file in PATH, shell will open and run the file "ls" by calling fork(). fork() creates a new child process by cloning current parent process, in this case, shell process. The child process then calls execve() to run the user's command, "ls". The system call execve() does three things: the operating system stops the duplicated process (of the parent), loads up the new program (in this case: ls) and starts. execve() replaces defining parts of the current process's memory stack with the new stuff loaded from the "ls" executable file. Parent process waits until the child completes the execution via wait().
Upon completion, the child process will terminate and control will be returned to the parent process. #+END_EXAMPLE

HALF export PAGER=cat :noexport:
--8<-------------------------- separator ------------------------>8-- :noexport:
TODO cd ~/mydata/test.conf/.. :noexport:
DONE [#A] ls only list folder: ls -ld */ :noexport: CLOSED: [2020-06-11 Thu 22:23]
TODO [#A] linux: how to view huge logfiles by line offset :noexport:IMPORTANT:
TODO [#A] How to make sure your Linux system users won't hide or delete their .bash_history / Securing .bash_history file :noexport: http://www.pc-freak.net/blog/how-to-make-sure-your-linux-system-users-wont-hide-or-delete-their-bash_history-securing-bash_history-file/

cheatsheet-shell-A4 cheatsheet-shell-A4 copied to clipboard

Metadata

Run grep for files filtered by find

Loop with pipes

--8<-------------------------- separator ------------------------>8-- :noexport:

For performance issue, better use internal commands, compared to external commands

Say to add 2 numbers say x & y:

for a given list, return a new list with items re-arranged by random algorithm

An IP range is like "10.32.182.181-184" or just a single IP address

--8<-------------------------- §separator§ ------------------------>8--

Trigger cc recovery task in all SS disk of current node

Iterate each ss disk

Scriptname: speller

Purpose: Check and fix spelling errors in a file

@copyright 2013

File : xzb_update_user_html.sh

Author : filebat [email protected]

Description : Update posts info to mysql

--

Created : <2013-01-31>

Updated: Time-stamp: <2013-01-31 20:08:58>

File : xzb_update_user_html.sh ends

Prints backtrace info

filename:lineno:function

Prints line number and "message" in error format

err $LINENO "message"

Set an option in an INI file

iniset config-file section option value

Normalize config values to True or False

Accepts as False: 0 no No NO false False FALSE

Accepts as True: 1 yes Yes YES true True TRUE

VAR=$(trueorfalse default-value test-value)

Prints backtrace info

filename:lineno:function

Copyright 2012 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License"); you may

not use this file except in compliance with the License. You may obtain

a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS, WITHOUT

WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the

License for the specific language governing permissions and limitations

under the License.

Find out if jenkins has triggered the out-of-memory killer by checking

the output of dmesg before and after a test run.

File : test.sh

Author : filebat [email protected]

Description :

--

Created : <2013-12-29>

Updated: Time-stamp: <2013-12-31 23:58:24>

File : test.sh ends

Check for root privileges

Check for root privileges

File : run_command_in_all_nodes.sh

Author : Denny [email protected]

Description : Run command in a list of servers.

Here we assume ssh key is properly uploaded, thus we

don't need to input ssh password

--

Created : <2014-07-10>

Updated: Time-stamp: <2014-07-31 14:10:04>

Example1: Run a command in all nodes of prod env.

sh ./run_command_in_all_nodes.sh "ssh root@HOSTIP date"

Note: We assume ssh public key is uploaded properly

In above, HOSTIP will be replaced by server ip.

Example2: Run a command in some nodes

sh ./run_command_in_all_nodes.sh "ssh root@HOSTIP date" "172.20.16.14 172.20.16.12"

File : run_command_in_all_nodes.sh ends

foo is a function

@copyright 2017 DennyZhang.com

Licensed under MIT

https://www.dennyzhang.com/wp-content/mit_license.txt

File: test.sh

Author : Denny [email protected]

Description :

https://leetcode.com/problems/cheatsheet-shell-A4/description/][leetcode.com]]

--

Created : <2017-10-17>

cheatsheet-shell-A4
cheatsheet-shell-A4 copied to clipboard