===== Moviendo ficheros entre máquinas =====
rsync
Cuando necesites tener un directorio en una máquina, como //mirror//* de otra máquina, utiliza rsync. Este comando compara todos los ficheros del árbol de sub-directorios y copia cualquiera que haya cambiado, al directorio //mirror// en la otra máquina. Por ejemplo, a continuación cómo podrías “traer” todos los ficheros de registro de livebox.jguru.com al lugar desde donde estas ejecutando el comando rsync:
//*[NdeT.:mirror: maquina con el contenido exacto de otra y que se actualiza periodicamente para mantener esta condición]//
##$ hostname jazz.jguru.com $ rsync -rabz -e ssh -v ‘parrt@livebox.jguru.com:/var/log/jguru/*’ \ /backup/web/logs##
rsync suprimirá o truncará los ficheros para asegurarse de que permanecen igual. Esto es malo si borras un fichero por error –eliminará tus archivos de respaldo. Agrega el argumento –suffix para decirle a rsync que haga una copia de cualquier archivo existente antes de que la sobreescriba:
##$ hostname jazz.jguru.com $ rsync -rabz -e ssh -v –suffix .rsync_`date ‘+%Y%m%d’` \ ‘parrt@livebox.jguru.com:/var/log/jguru/*’ /backup/web/logs##
donde `date ‘+%Y%m%d’` (en comillas inversas simples) significa “ejecute este comando de fecha”.
Para eliminar ciertos patrones de rsync, usa –exclude:
##$ rsync -rabz –exclude=entitymanager/ –suffix .rsync_`date ‘+%Y%m%d’` \ -e ssh -v ‘parrt@livebox.jguru.com:/var/log/jguru/*’ /backup/web/logs##
scp
Para copiar un fichero o directorio manualmente, usa scp:
##$ scp lecture.html parrt@nexus.cs.usfca.edu:~parrt/lectures##
Al igual que cp, usa -r para copiar el directorio e forma recursiva.
===== Miscellaneous =====
find
Most GUIs for Linux or PCs have a search facility, but from the command-line you can use find. To find all files named .p4 starting in directory ~/antlr/depot/projects, use:
$ find ~/antlr/depot/projects -name ‘.p4’
The default “action” is to -print.
You can specify a regular expression to match. For example, to look under your home directory for any xml files, use:
$ find ~ -name ‘*.xml’ -print
Note the use of the single quotes to prevent command-line expansion–you want the ‘*’ to go to the find command.
You can execute a command for every file or directory found that matches a name. For example, do delete all xml files, do this:
$ find ~ -name ‘*.xml’ -exec rm {} \;
where “{}” stands for “current file that matches”. The end of the command must be terminated with ‘;’ but because of the command-line expansion, you’ll need to escape the ‘;’.
You can also specify time information in your query. Here is a shell script that uses find to delete all files older than 14 days.
#!/bin/sh
BACKUP_DIR=/var/data/backup
# number of days to keep backups AGE=14 # days AGE_MINS=$[ $AGE * 60 * 24 ]
# delete dirs/files find $BACKUP_DIR/* -cmin +$AGE_MINS -type d -exec rm -rf {} \;
fuser
If you want to know who is using a port such as HTTP (80), use fuser. You must be root to use this:
$ sudo /sbin/fuser -n tcp 80 80/tcp: 13476 13477 13478 13479 13480 13481 13482 13483 13484 13486 13487 13489 13490 13491 13492 13493 13495 13496 13497 13498 13499 13500 13501 13608
The output indicates the list of processes associated with that port. whereis
Sometimes you want to use a command but it’s not in your PATH and you can’t remember where it is. Use whereis to look in standard unix locations for the command.
$ whereis fuser fuser: /sbin/fuser /usr/man/man1/fuser.1 /usr/man/man1/fuser.1.gz $ whereis ls ls: /bin/ls /usr/man/man1/ls.1 /usr/man/man1/ls.1.gz
whereis also shows man pages. which
Sometimes you might be executing the wrong version of a command and you want to know which version of the command your PATH indicates should be run. Use which to ask:
$ which ls alias ls=’ls –color=tty’
/bin/ls
$ which java /usr/local/java/bin/java
If nothing is found in your path, you’ll see:
$ which fuser /usr/bin/which: no fuser in (/usr/local/bin:/usr/local/java/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/X11R6/bin:/home/parrt/bin)
kill
To send a signal to a process, use kill. Typically you’ll want to just say kill pid where pid can be found from ps or top (see below).
Use kill -9 pid when you can’t get the process to die; this means kill it with “extreme prejudice”. traceroute
If you are having trouble getting to a site, use traceroute to watch the sequence of hops used to get to a site:
$ /usr/sbin/traceroute http://www.cnn.com/ www.cnn.com 1 65.219.20.145 (65.219.20.145) 2.348 ms 1.87 ms 1.814 ms 2 loopback0.gw5.sfo4.alter.net (137.39.11.23) 3.667 ms 3.741 ms 3.695 ms 3 160.atm3-0.xr1.sfo4.alter.net (152.63.51.190) 3.855 ms 3.825 ms 3.993 ms ...
What is my IP address?
$ /sbin/ifconfig
Under the eth0 interface, you’ll see the inet addr:
eth0 Link encap:Ethernet HWaddr 00:10:DC:58:B1:F0
inet addr:138.202.170.4 Bcast:138.202.170.255 Mask:255.255.255.0 ...
pushd, popd
Instead of cd you can use pushd to save the current dir and then automatically cd to the specified directory. For example,
$ pwd /Users/parrt $ pushd /tmp /tmp ~ $ pwd /tmp $ popd ~ $ pwd /Users/parrt
top
To watch a dynamic display of the processes on your box in action, use top. ps
To print out (wide display) all processes running on a box, use ps auxwww. .bash_profile
Your bash shell initialization file is ~username/.bash_profile and has set up code like the following:
PATH=$PATH:$HOME/bin
Typically, you will go in and set your CLASSPATH so that you don’t have to set it all the time.
export CLASSPATH=”.:/home/public/cs601/junit.jar”
The export means that the assignment to CLASSPATH is visible to all child processes (that is, visible to all programs you run from the shell).
===== Useful combinations =====
How to kill a set of processes
If you want to kill all java processes running for parrt, you can either run killall java if you are parrt or generate a “kill” script via:
$ ps auxwww|grep java|grep parrt|awk ‘{print “kill -9 “,$2;}’ > /tmp/killparrt $ bash /tmp/killparrt # run resulting script
The /tmp/killparrt file would look something like:
kill -9 1021 kill -9 1023 kill -9 1024
Note: you can also do this common task with:
$ killall java
Please be aware that this is linux specific; i’m told that it will kill all processing on UNIXen like Solaris! How to make a histogram
A histogram is set of count, value pairs indicating how often the value occurs. The basic operation will be to sort, then count how many values occur in a row and then reverse sort so that the value with the highest count is at the top of the report.
$ ... | sort |uniq -c|sort -r -n
Note that sort sorts on the whole line, but the first column is obviously significant just as the first letter in someone’s last name significantly positions their name in a sorted list.
uniq -c collapses all repeated sequences of values but prints the number of occurrences in front of the value. Recall the previous sorting:
$ awk ‘{print $3;}’ < /home/public/cs601/unix/pageview-20021022.log | \
sort | \uniq
/article/index.jsp /article/index.jsp?page=1 /article/index.jsp?page=10 /article/index.jsp?page=2 ...
Now add -c to uniq:
$ awk ‘{print $3;}’ < /home/public/cs601/unix/pageview-20021022.log | \
sort | \uniq -c
623 /article/index.jsp
6 /article/index.jsp?page=110 /article/index.jsp?page=10
109 /article/index.jsp?page=2 ...
Now all you have to do is reverse sort the lines according to the first column numerically.
$ awk ‘{print $3;}’ < /home/public/cs601/unix/pageview-20021022.log | \
sort | \uniq -c | \sort -r -n
6170 /index.jsp 2916 /search/results.jsp 1397 /faq/index.jsp 1018 /forums/index.jsp 884 /faq/home.jsp?topic=Tomcat ...
In practice, you might want to get a histogram that has been “despidered” and only has faq related views. You can filter out all page view lines associated with spider IPs and filter in only faq lines:
$ grep -v -f /tmp/spider.IPs /home/public/cs601/unix/pageview-20021022.log | \
awk '{print $3;}'| \grep '/faq' | \sort | \uniq -c | \sort -r -n
1397 /faq/index.jsp 884 /faq/home.jsp?topic=Tomcat 525 /faq/home.jsp?topic=Struts 501 /faq/home.jsp?topic=JSP 423 /faq/home.jsp?topic=EJB ...
If you want to only see despidered faq pages that were referenced more than 500 times, add an awk command to the end.
$ grep -v -f /tmp/spider.IPs /home/public/cs601/unix/pageview-20021022.log | \
awk '{print $3;}'| \grep '/faq' | \sort | \uniq -c | \sort -r -n | \awk '{if ($1>500) print $0;}'
1397 /faq/index.jsp 884 /faq/home.jsp?topic=Tomcat 525 /faq/home.jsp?topic=Struts 501 /faq/home.jsp?topic=JSP
Generating scripts and programs
I like to automate as much as possible. Sometimes that means writing a program that generates another program or script. Processing mail files
I wanted to get a sequence of SQL commands that would update our database whenever someone’s email bounced. Processing the mail file is pretty easy since you can look for the error code followed by the email address. A bounced email looks like:
From MAILER-DAEMON@localhost.localdomain Wed Jan 9 17:32:33 2002 Return-Path: <> Received: from web.jguru.com (web.jguru.com [64.49.216.133])
by localhost.localdomain (8.9.3/8.9.3) with ESMTP id RAA18767 for <notifications@jguru.com>; Wed, 9 Jan 2002 17:32:32 -0800
Received: from localhost (localhost)
by web.jguru.com (8.11.6/8.11.6) id g0A1W2o02285; Wed, 9 Jan 2002 17:32:02 -0800
Date: Wed, 9 Jan 2002 17:32:02 -0800 From: Mail Delivery Subsystem mailto:%26%23x4d%3B%26%23x41%3B%26%23x49%3B%26%23x4c%3B%26%23x45%3B%26%23x52%3B%26%23x2d%3B%26%23x44%3B%26%23x41%3B%26%23x45%3B%26%23x4d%3B%26%23x4f%3B%26%23x4e%3B%26%23x40%3B%26%23x77%3B%26%23x65%3B%26%23x62%3B%26%23x2e%3B%26%23x6a%3B%26%23x67%3B%26%23x75%3B%26%23x72%3B%26%23x75%3B%26%23x2e%3B%26%23x63%3B%26%23x6f%3B%26%23x6d%3B MAILER-DAEMON@web.jguru.com Message-Id: mailto:%26%23x32%3B%26%23x30%3B%26%23x30%3B%26%23x32%3B%26%23x30%3B%26%23x31%3B%26%23x31%3B%26%23x30%3B%26%23x30%3B%26%23x31%3B%26%23x33%3B%26%23x32%3B%26%23x2e%3B%26%23x67%3B%26%23x30%3B%26%23x41%3B%26%23x31%3B%26%23x57%3B%26%23x32%3B%26%23x6f%3B%26%23x30%3B%26%23x32%3B%26%23x32%3B%26%23x38%3B%26%23x35%3B%26%23x40%3B%26%23x77%3B%26%23x65%3B%26%23x62%3B%26%23x2e%3B%26%23x6a%3B%26%23x67%3B%26%23x75%3B%26%23x72%3B%26%23x75%3B%26%23x2e%3B%26%23x63%3B%26%23x6f%3B%26%23x6d%3B 200201100132.g0A1W2o02285@web.jguru.com To: mailto:%26%23x6e%3B%26%23x6f%3B%26%23x74%3B%26%23x69%3B%26%23x66%3B%26%23x69%3B%26%23x63%3B%26%23x61%3B%26%23x74%3B%26%23x69%3B%26%23x6f%3B%26%23x6e%3B%26%23x73%3B%26%23x40%3B%26%23x6a%3B%26%23x67%3B%26%23x75%3B%26%23x72%3B%26%23x75%3B%26%23x2e%3B%26%23x63%3B%26%23x6f%3B%26%23x6d%3B notifications@jguru.com MIME-Version: 1.0 Content-Type: multipart/report; report-type=delivery-status;
boundary="g0A1W2o02285.1010626322/web.jguru.com"
Subject: Returned mail: see transcript for details Auto-Submitted: auto-generated (failure)
This is a MIME-encapsulated message
–g0A1W2o02285.1010626322/web.jguru.com
The original message was received at Wed, 9 Jan 2002 17:32:02 -0800 from localhost [127.0.0.1]
~1) —- The following addresses had permanent fatal errors —–
mailto:%26%23x70%3B%26%23x61%3B%26%23x69%3B%26%23x6e%3B%26%23x40%3B%26%23x69%3B%26%23x6e%3B%26%23x74%3B%26%23x68%3B%26%23x65%3B%26%23x6e%3B%26%23x65%3B%26%23x63%3B%26%23x6b%3B%26%23x2e%3B%26%23x63%3B%26%23x6f%3B%26%23x6d%3B pain@intheneck.com
(reason: 550 Host unknown)
~1) —- Transcript of session follows —–
550 5.1.2 mailto:%26%23x70%3B%26%23x61%3B%26%23x69%3B%26%23x6e%3B%26%23x40%3B%26%23x69%3B%26%23x6e%3B%26%23x74%3B%26%23x68%3B%26%23x65%3B%26%23x6e%3B%26%23x65%3B%26%23x63%3B%26%23x6b%3B%26%23x2e%3B%26%23x63%3B%26%23x6f%3B%26%23x6d%3B pain@intheneck.com... Host unknown (Name server: intheneck.com: host not found) ...
Notice the SMTP 550 error message. Look for that at the start of a line then kill the angle brackets, remove the ... and use awk to print out the SQL:
# This script works on one email or a file full of other emails # since it just looks for the SMTP 550 or 554 results and then # converts them to SQL commands. grep -E ‘^(550|554)’ | \
sed 's/[<>]//g' | \ sed 's/\.\.\.//' | \ awk "{printf(\"UPDATE PERSON SET bounce=1 WHERE email='%s';\n\",\$3);}" >> bounces.sql
I have to escape the $3 because it means something to the surround bash shell script and I want awk to see the dollar sign. Generating getter/setters
#!/bin/bash # From a type and name (plus firstlettercap version), # generate a Java getter and setter # # Example: getter.setter String name Name #
TYPE=$1 NAME=$2 UPPER_NAME=$3
echo “public $TYPE get$UPPER_NAME() {” echo " return $NAME;” echo “}” echo echo “void set$UPPER_NAME($TYPE $NAME) {” echo " this.$NAME = $NAME;” echo “}” echo
===== Have I been hacked? =====
Failed logins: /var/log/messages
last, w, uptime
/etc/passwd changed?
fuser for ports
portscans in server report
weird processing hogging CPU?
