Hawk's Babble-Log

A friend of mine asked me to help him make a copy of two drives. These drives work together on a RAID 0. With a RAID 0, you have approximately twice the speed, but also twice the chance of failure.

So, understandably, they would like to have a backup of the system. But they don’t want something that has to be installed, and then configured, and all that stuff. They are not “linux people”. They want something where they can yank out the old drives and insert two new drives and have it “just work”.

Here’s the overall procedure:

• Make a bootable Knoppix USB drive.
• Connect a new drive to the system. This new drive will store the two source disk images.
• Boot Knoppix and Ctrl-Alt-F1 into command-line mode.
• Make a file from the entire 1st disk and put it on the new disk.
• If the other drive is not connected, shut the system down and connect the other drive.
• Make a file from the entire 2nd disk and put it on the new disk.
• Disconnect the new drive & the USB drive, and reconnect the originals.
• Boot it up again and watch it to make sure all is well.

For the linux guru, here are the commands I used:


fdisk -l


fdisk is usually the tool you use to set up partitions from the command-line. Using it like this with the -l switch gives you a list of all the partitions on all connected drives, whether they are mounted or not. Very useful.


dd if=/dev/sdc | gzip > drive1_250g_2011-08-19.img.gz


This is the magic linux command used to actually make the file from one whole disk. dd is a command which simply routes data. Every device in linux is a file, so it’s easy to read an entire disk. In the above case, dd is getting it’s data (if = In File) from the device which represents the entire disk: /dev/sdc. sdc means the third SCSI drive detected. On your system, it may be sda or sdd.

By default, dd outputs data to STDOUT, so here I’m piping it into the gzip command, which compresses data. If you don’t compress data, then you might end up with a file which is bigger than you expect when reading raw filesystems, due to the way filesystems lay down data with sector markers and whatnot.

Finally, I’m redirecting the output from gzip into a file, and I’m making sure that it has an extension which describes what kind of file it is. But you can call it whatever you want.

The first time I tried this command, I used bzip2 because it has better data compression routines and therefore makes smaller files. But better compression also means more CPU computations, and that alone slowed down performance to an unacceptable level. Why an unacceptable level? This server that I was working on had to be fully online by 7am. I took it offline and started work around midnight. This gave me about 6 hours to copy the drives. I gave myself an extra hour to debug and get it put back together and installed where it was in the rack, which should be plenty of time.

Using the above dd command with bzip2, I was able to copy about 9-11 MB / sec. This put the total time for one drive at about 6-8 hours. I didn’t have that much time.

Using the above dd command with gzip, I was able to copy about 40-45 MB / sec. This put the total time for one drive at about 90-105 minutes. Now I had enough time in one night to copy the drives. Additionally, the files were very small with gzip, so bzip2 would not have made a significant difference but would have taken much, much longer.

How did I know the rate of copy using dd?


kill -USR1 (pid of dd)


First, find out what the Process ID is for the running dd command. To do this, Ctrl-Alt-F2 into the second virtual terminal and run:


ps aux | grep "dd if"


The PID is the first number on the dd line. Once you’ve got that, use the above kill -USR1 command and dd will output several lines of useful information (in the virtual terminal where it’s running), including how much data has been transferred and the rate.

I set it up so that it would do this automatically every 5 minutes like this:


while kill -USR1 (pid of dd) ; do sleep 300 ; done


This while loop runs the stated command until that command returns false. The body of the while loop simply sleeps for 5 minutes. After starting this loop, Ctrl-Alt-F1 back into the first virtual terminal to watch dd‘s progress.

I was fully successful in copying the two drives to files and had the server back in operation before 5am.

Once I got home, I ordered up two new drives exactly like the old ones. Well, mostly exactly like the old ones. Once they arrived, I connected one to my SATA-to-USB drive adapter and ran this:


zcat (filename) | dd of=/dev/sdx


MAKE SURE YOU KNOW WHAT /dev/sdx is! Running this command on the wrong device will wipe out whatever is on that device, including your main linux filesystem!

How to find out the device you just plugged in?


dmesg | tail


From the output of the above command, you should be able to figure it out.

Decompressing on the fly and writing to a drive like this via a SATA-to-USB interface will be very slow. The max rate I achieved was 4M / sec. More commonly, I achieved 3.8M / sec, which is a little more than 1M every 5 minutes. That puts the total time up around 18 hours per drive. But hey, I was in no hurry. The drive finished up over night and then I did the other one. It was the same command but with a different filename.

Hope this helps.

I’m sitting at my desk when I hear two girls in the office giggling. One of them is getting married, and the other one is, like, sooo excited for her!!!

The one who is getting married shows the other one a picture of a wedding dress she printed out. The other one squeals in delight and exclaims, “Oohhhh my goood!!! It’s, like, soooo amazing!!!”

“Yeah!!! I know, right!?!?” says the other one!!!

“Ohhh my god and I love the back!!!”

“Yeah!!!”

“Ohhhhh!!!” she says excitedly!!! She handed me the photos of the dress. I looked at the picture and it was, like, sooo amazing!! I nodded enthusiastically because it was, like, a very nice dress!!!

The one who is getting married then says, “Ohhhh!!! I saw the most amazing shoes!!!”

“Really?!??” says the other one!!!

“Yeah!!! Like, soooo amazing!!! By this guy named… ummm…” The other girl fills in the name; it’s some unpronounceable name!!!

“Yeah!!! Search for (insert unpronounceable name here)…” the other girl bangs away on her keyboard, then exclaims, “Ohhhh my god!!! Those are, like, soooo amazing!!!”

A this point, they both started giggling and cackling and, like, oh my god!!! Now I totally have to buy those shoes!!!

I’ve had an email account with Ebay for a while. I nearly always register my username as ‘jedihawk’, if I can. Usually, it’s available. So I got ‘jedihawk’ as my Ebay username. That makes sense. I also have registered many email adresses over the years. I always go with jedihawk@whatever.net.

So I’m now trying to update my email address in the Ebay system and it won’t let me change it. Why? Because my Ebay username is the same as the first part of my email address.

This means I’m stuck. I can’t change my Ebay email address to any of my other email addresses. They all (or nearly all) start with ‘jedihawk’!

Ebay’s email policies suck!!! Ebay is trying to take my security out of my hands. They are trying to prevent me from “hurting myself”. Well, I’m a big boy now. I’ve been online for decades. I can spot a spoof email a mile away. I have never been fooled by a spoof email. Ebay doesn’t need to protect me. I can handle it all by my self. I’m all growed up ‘n stuff.

I called Ebay support and talked to a nice lady who tried to help, but was completely unable to. I explained how I was a “net veteran”, and how I was in no danger by having my email address the same as my username (like it already is!). She understood, but still could not update my email with Ebay.

So it looks like I’ll be missing out on Ebay emails soon, as my old email address is going away. Oh well, at least they have an internal email system I can access.

Ebay, if you’re listing: Your email “security” policies SUCK!! Grow a brain and stop trying to protect me. I can handle it myself. Put the responsibility where it belongs: with me! Rather than prevent me from updating my email address, put up a warning, then let them me the change!

I heard an ad for Santrex internet hosting services on di.fm, so I thought I’d give ‘em a try. Basically, here’s what I think:

Santrex sucks ass!

Why? Because they told me that my order was flagged as fraudulent and asked me to scan my credit card (both sides), AND scan my ID card (both sides), plus my billing address (which I already gave them), plus my phone (which I already gave them) and send it to them to prove I am not fraudulent!

Can you believe that? I told ‘em, ummm, like no ‘n stuff; cancel my account and issue me a refund.

They asked my why I was “afraid to submit this information”. I told ‘em, I’m sticking with ServerBeach, where they don’t flag me as fraud.

I was going to start with just one server with Santrex. If it worked out well, and the bandwidth was good, and the power stayed on and so forth, I would have increased the number of servers, plus I would have rented more expensive servers for some of my clients (with more bandwidth, more storage, etc.) But now, Santrex will not be getting any of my business.

I’m not saying Santrex is collecting ID’s and credit cards for fraudulent purposes. I’m not saying that at all. I’m only saying Santrex is not recommended. Go with ServerBeach.com instead.

I took my roomie to the Americana in Glendale and he was surprised by the following:

* The food was expensive!
* The food only came in large containers!
* The food must have contained 99% sugar!

Here are some notable quotes:

* “It costs how much?!”
* “You don’t have a small size?!”
* “You know, I think this is mostly sugar!”

As you can see, an afternoon out with my roomie is a blast!

I need more exclamation marks!

We’re running a webserver on Ubuntu 9.10. For various reasons, we decided to upgrade Apache without upgrading Ubuntu. Docs for Apache I followed:
http://httpd.apache.org/docs/2.2/install.html

And here’s what I did:

1) Download the latest Apache source tarball. In my case, it was Apache 2.2.19.


wget http://apache.mirrors.tds.net//httpd/httpd-2.2.19.tar.bz2


2) Extract.


tar xjvf httpd-2.2.19.tar.bz2
cd httpd-2.2.19


3) Configure the build to be like our existing Apache install. Use your own ./configure command.


./configure --prefix=/usr/local/apache2 --enable-mods-shared="all authn_file authz_default authz_groupfile authz_host authz_user deflate php5 proxy_http proxy rewrite ssl"


4) Build it.


make && echo done with step: make
make install && echo done with step: make install


5) Now comes the hard part. I had to migrate our existing config tree over to this new custom source compile. I started off by copying the entire /etc/apache2 dir to /usr/local/apache2/conf


cp -a /etc/apache2/* /usr/local/apache2/conf/


I tried several configs, but the easiest thing to do was to rename Ubuntu’s old apache2.conf file to httpd.conf as that is the default config file Apache uses when it starts up. I also took out Ubuntu’s Include line which included the file httpd.conf as I don’t want it trying to include itself endlessly.

After that, I just needed to update the config.

I updated the ServerRoot directive:


#ServerRoot "/etc/apache2"
ServerRoot "/usr/local/apache2"


I changed Apache’s default user to Ubuntu’s standard www-data user:


User www-data
Group www-data


Instead of keeping the tree of mods-available and mods-enabled and all those separate files and symlinks, I just put it all into the main httpd.conf file. That works best for us; easier to maintain.

I used vim to edit our new httpd.conf file. From within vim:


:r !cat mods-enabled/*.load
:r !cat mods-enabled/*.conf


Order is significant here; proxy_module first, then proxy_http_module:


LoadModule proxy_module /usr/local/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/local/apache2/modules/mod_proxy_http.so


I updated the paths for each module. A standard Ubuntu module looks like this:


LoadModule log_config_module /usr/lib/apache2/modules/mod_log_config.so


But with my new custom compiled Apache, I needed this:


LoadModule log_config_module /usr/local/apache2/modules/mod_log_config.so


… because my new Apache lives in /usr/local/apache2.

I kept Ubuntu’s PHP5 binary install:


LoadModule php5_module /usr/lib/apache2/modules/libphp5.so


After all that config stuff, it was time to test it out. I temporarily changed the ports from 80 to 180, and from 443 to 1443 (for SSL), and tried it out.


/usr/local/apache2/bin/apachectl start


At this point, I have our original Apache running alongside my new custom-compiled one. Cool. But I’m done testing.


/usr/local/apache2/bin/apachectl stop


Once I’m done testing and I see that it’s working on non-standard ports, it’s time to put the ports back and go live. But there is one more important step here: the startup/control script!

Ubuntu uses /etc/init.d/apache2 as the control script. It has some fancy functions in it, sets some things using environment vars, and other stuff I don’t need. So I used some functions from Ubuntu’s startup script, and some stuff from a generic startup script and came up with this:

#!/bin/sh

# init
PIDFILE=/usr/local/apache2/logs/httpd.pid

# functions

pidof_apache() {
  # if pidof is null for some reasons the script exits automagically
  # classified as good/unknown feature
  PIDS=$(pidof httpd) || true

  [ -e $PIDFILE ] && PIDS2=$(cat $PIDFILE)

  # if there is a pid we need to verify that belongs to apache2
  # for real
  for i in $PIDS; do
    if [ "$i" = "$PIDS2" ]; then
      # in this case the pid stored in the
      # pidfile matches one of the pidof apache
      # so a simple kill will make it
      echo $i
      return 0
    fi
  done
  return 1
}

# start

case "$1" in

start)
echo "Starting Apache ..."
/usr/local/apache2/bin/apachectl start
;;

stop)
echo "Stopping Apache ..."
/usr/local/apache2/bin/apachectl stop
;;

graceful)
echo "Restarting Apache gracefully..."
/usr/local/apache2/bin/apachectl graceful
;;

restart)
echo "Restarting Apache ..."
/usr/local/apache2/bin/apachectl restart
;;

status)
PID=$(pidof_apache)
if [ -n "$PID" ]; then
  echo "Apache is running (pid $PID)."
  exit 0
else
  echo "Apache is not running."
  exit 1
fi
;;

*)
echo "Usage: '$0' {status|start|stop|restart|graceful}" >&2
exit 64
;;

esac

exit 0

I renamed Ubuntu’s default Apache control script…


cd /etc/init.d
cp -a apache2 apache2.orig


… and installed the above into apache2…


(copy script into your clipboard)
cat > apache2
(paste script)
^D (Ctrl-D for Done)


The above procedure will completely overwrite your apache2 script, so be sure you make a copy of it first.

And now for the moment of truth:


./apache2.orig stop && sleep 1 && ./apache start


I hit our website and all was well. We were down for about 2 seconds. I can live with that downtime.

The nice thing about this is that if it doesn’t work, I can always go back to the stock, binary Apache that comes with Ubuntu.

Hope this helps.

I live in the great land of America, in a very badly managed state called Nazifornia. Oops, I mean California.

I have no credit card debt. I played that game and lost. Then I canceled them all and paid them all off. I have zero credit cards. I use my check card (a Mastercard) for most everything.

I do not accept any money from my government. I am not on unemployment, welfare, or any other government hand-out program.

I have to pay in to the Social “Security” program, but I’ll never get anything out of it. Nor would I want to. I have my own retirement plans.

I do not have any kind of medical insurance, dental insurance, medicare, etc. I pay my own medical bills and dental bills.

I do not depend on my government at all. They could shut down tomorrow and I would not be affected (personally, financially, and so on).

http://www.hjsplit.org/

I can recommend HJ-Split.

I have used HJ-Split in Windows, to split up a huge file, and send it to a friend of mine on a Mac. There’s no way he is ever going to install it into his Mac, so I’m not sure if that version works, but HJ-Split works great in Windows.

I also used it to break up a large file and get it home and into my Linux system. I used the software to break up a 31G file into 1.5G pieces, put the pieces on my 32G USB drive, took it home, copied all the pieces into my Linux machine at home, and then pieced it back together again using the ‘cat‘ command. Worked perfectly.