Table of Contents
Tools and tips for managing binary and text data on the Debian system are described.
The security of the data and its controlled sharing have several aspects.
These can be realized by using some combination of tools.
Here is a summary of archive and compression tools available on the Debian system.
Table 10.1. List of archive and compression tools
package | popcon | size | command | extension | comment |
---|---|---|---|---|---|
tar
*
|
V:62, I:99 | 2044 |
tar (1)
|
.tar
|
the standard archiver (de facto standard) |
cpio
*
|
V:36, I:99 | 664 |
cpio (1)
|
.cpio
|
Unix System V style archiver, use with find (1)
|
binutils
*
|
V:50, I:77 | 9964 |
ar (1)
|
.ar
|
archiver for the creation of static libraries |
fastjar
*
|
V:4, I:39 | 220 |
fastjar (1)
|
.jar
|
archiver for Java (zip like) |
pax
*
|
V:1.5, I:5 | 172 |
pax (1)
|
.pax
|
new POSIX standard archiver, compromise between tar and cpio
|
afio
*
|
V:0.3, I:1.6 | 240 |
afio (1)
|
.afio
|
extended cpio with per-file compression etc.
|
gzip
*
|
V:90, I:99 | 292 |
gzip (1), zcat (1), …
|
.gz
|
GNU LZ77 compression utility (de facto standard) |
bzip2
*
|
V:56, I:80 | 132 |
bzip2 (1), bzcat (1), …
|
.bz2
|
Burrows-Wheeler block-sorting compression utility with higher compression ratio than gzip (1) (slower than gzip with similar syntax)
|
lzma
*
|
V:9, I:69 | 172 |
lzma (1)
|
.lzma
|
LZMA compression utility with higher compression ratio than gzip (1) (slower than gzip with similar syntax)
|
p7zip
*
|
V:3, I:25 | 1052 |
7zr (1), p7zip (1)
|
.7z
|
7-Zip file archiver with high compression ratio (LZMA compression) |
p7zip-full
*
|
V:11, I:21 | 3612 |
7z (1), 7za (1)
|
.7z
|
7-Zip file archiver with high compression ratio (LZMA compression and others) |
lzop
*
|
V:0.9, I:7 | 144 |
lzop (1)
|
.lzo
|
LZO compression utility with higher compression and decompression speed than gzip (1) (lower compression ratio than gzip with similar syntax)
|
zip
*
|
V:9, I:59 | 628 |
zip (1)
|
.zip
|
InfoZIP: DOS archive and compression tool |
unzip
*
|
V:19, I:71 | 384 |
unzip (1)
|
.zip
|
InfoZIP: DOS unarchive and decompression tool |
Warning | |
---|---|
Do not set the " |
Note | |
---|---|
The gzipped |
Note | |
---|---|
|
Note | |
---|---|
|
Note | |
---|---|
|
Note | |
---|---|
Internal structure of OpenOffice data files are " |
Here is a summary of simple copy and backup tools available on the Debian system.
Table 10.2. List of copy and synchronization tools
package | popcon | size | tool | function |
---|---|---|---|---|
coreutils
*
|
V:91, I:99 | 12868 | GNU cp | locally copy files and directories ("-a" for recursive) |
openssh-client
*
|
V:53, I:98 | 2076 | scp |
remotely copy files and directories (client, "-r " for recursive)
|
openssh-server
*
|
V:65, I:78 | 808 | sshd | remotely copy files and directories (remote server) |
rsync
*
|
V:16, I:42 | 640 | - | 1-way remote synchronization and backup |
unison
*
|
V:0.8, I:3 | 1644 | - | 2-way remote synchronization and backup |
pdumpfs
*
|
V:0.05, I:0.16 | 148 | - |
daily local backup using hardlinks (similar to Plan9's dumpfs )
|
Tip | |
---|---|
Execution of the |
Tip | |
---|---|
Version control system (VCS) tools in Table 10.16, “List of version control system tools” can function as the multi-way copy and synchronization tools. |
Here are several ways to archive and unarchive the entire content of the directory "/source
" using different tools.
GNU tar
(1):
$ tar cvzf archive.tar.gz /source $ tar xvzf archive.tar.gz
cpio
(1):
$ find /source -xdev -print0 | cpio -ov --null > archive.cpio; gzip archive.cpio $ zcat archive.cpio.gz | cpio -i
afio
(1):
$ find /source -xdev -print0 | afio -ovZ0 archive.afio $ afio -ivZ archive.afio
Here are several ways to make the entire content of local or remote copies using different tools.
/source
" directory → "/dest
" directory
/source
" directory at local host → "/dest
" directory at "user@host.dom
" host
GNU cp
(1) and openSSH scp
(1):
# cp -a /source /dest # scp -pr /source user@host.dom:/dest
GNU tar
(1):
# (cd /source && tar cf - . ) | (cd /dest && tar xvfp - ) # (cd /source && tar cf - . ) | ssh user@host.dom '(cd /dest && tar xvfp - )'
cpio
(1):
# cd /source; find . -print0 | cpio -pvdm --null --sparse /dest
afio
(1):
# cd /source; find . -print0 | afio -pv0a /dest
Tip | |
---|---|
|
# scp -pr user1@host1.dom:/source user2@host2.dom:/dest
find
(1) is used to select files for archive and copy commands (see Section 10.1.3, “Idioms for the archive” and Section 10.1.4, “Idioms for the copy”) or for xargs
(1) (see Section 9.5.9, “Repeating a command looping over files”). This can be enhanced by using its command arguments.
Basic syntax of find
(1) can be summarized as the following.
-o
" between conditionals) has lower precedence than "logical AND" (specified by "-a
" or nothing between conditionals).
!
" before a conditional) has higher precedence than "logical AND".
-prune
" always returns logical TRUE and, if it is a directory, searching of file is stopped beyond this point.
-name
" matches the base of the filename with shell glob (see Section 1.5.6, “Shell glob”) but it also matches its initial ".
" with metacharacters such as "*
" and "?
". (New POSIX feature)
-regex
" matches the full path with emacs style BRE (see Section 1.6.2, “Regular expressions”) as default.
-size
" matches the file based on the file size (value precedented with "+
" for larger, precedented with "-
" for smaller)
-newer
" matches the file newer than the one specified in its argument.
-print0
" always returns logical TRUE and print the full filename (null terminated) on the standard output.
find
(1) is often used with an idiomatic style as the following.
# find /path/to \ -xdev -regextype posix-extended \ -type f -regex ".*\.afio|.*~" -prune -o \ -type d -regex ".*/\.git" -prune -o \ -type f -size +99M -prune -o \ -type f -newer /path/to/timestamp -print0
This means to do following actions.
/path/to
"
.*\.afio
" or ".*~
" from search by stop processing
.*/\.git
" from search by stop processing
/path/to/timestamp
"
Please note the idiomatic use of "-prune -o
" to exclude files in the above example.
Note | |
---|---|
For non-Debian Unix-like system, some options may not be supported by |
We all know that computers fail sometime or human errors cause system and data damages. Backup and recovery operations are the essential part of successful system administration. All possible failure modes hit you some day.
Tip | |
---|---|
Keep your backup system simple and backup your system often. Having backup data is more important than how technically good your backup method is. |
There are 3 key factors which determine actual backup and recovery policy.
Knowing what to backup and recover.
~/
"
/var/
" (except "/var/cache/
", "/var/run/
", and "/var/tmp/
")
/etc/
"
/usr/local/
" or "/opt/
"
Knowing how to backup and recover.
Assessing risks and costs involved.
As for secure storage of data, data should be at least on different disk partitions preferably on different disks and machines to withstand the filesystem corruption. Important data are best stored on a write-once media such as CD/DVD-R to prevent overwrite accidents. (See Section 10.2, “The binary data” for how to write to the storage media from the shell commandline. GNOME desktop GUI environment gives you easy access via menu: "Places→CD/DVD Creator".)
Note | |
---|---|
You may wish to stop some application daemons such as MTA (see Section 6.3, “Mail transport agent (MTA)”) while backing up data. |
Note | |
---|---|
You should pay extra care to the backup and restoration of identity related data files such as " |
Note | |
---|---|
If you run a cron job as a user process, you must restore files in " |
Here is a select list of notable backup utility suites available on the Debian system.
Table 10.3. List of backup suite utilities
package | popcon | size | description |
---|---|---|---|
rdiff-backup
*
|
V:1.2, I:3 | 764 | (remote) incremental backup |
dump
*
|
V:0.4, I:1.6 | 620 |
4.4 BSD dump (8) and restore (8) for ext2/ext3 filesystems
|
xfsdump
*
|
V:0.3, I:2 | 684 |
dump and restore with xfsdump (8) and xfsrestore (8) for XFS filesystem on GNU/Linux and IRIX
|
backupninja
*
|
V:0.4, I:0.5 | 408 | lightweight, extensible meta-backup system |
mondo
*
|
V:0.13, I:0.7 | 1172 | Mondo Rescue: disaster recovery backup suite |
sbackup
*
|
V:0.09, I:0.2 | 488 | simple backup suite for GNOME desktop |
keep
*
|
V:0.2, I:0.5 | 1196 | backup system for KDE |
bacula-common
*
|
V:1.1, I:2 | 1244 | Bacula: network backup, recovery and verification - common support files |
bacula-client
*
|
I:0.9 | 52 | Bacula: network backup, recovery and verification - client meta-package |
bacula-console
*
|
V:0.3, I:1.2 | 116 | Bacula: network backup, recovery and verification - text console |
bacula-server
*
|
I:0.6 | 52 | Bacula: network backup, recovery and verification - server meta-package |
amanda-common
*
|
V:0.4, I:1.0 | 3120 | Amanda: Advanced Maryland Automatic Network Disk Archiver (Libs) |
amanda-client
*
|
V:0.4, I:0.9 | 560 | Amanda: Advanced Maryland Automatic Network Disk Archiver (Client) |
amanda-server
*
|
V:0.14, I:0.3 | 1264 | Amanda: Advanced Maryland Automatic Network Disk Archiver (Server) |
backuppc
*
|
V:0.7, I:0.9 | 2082 | BackupPC is a high-performance, enterprise-grade system for backing up PCs (disk based) |
backup-manager
*
|
V:0.4, I:0.5 | 660 | command-line backup tool |
backup2l
*
|
V:0.2, I:0.3 | 140 | low-maintenance backup/restore tool for mountable media (disk based) |
faubackup
*
|
V:0.19, I:0.2 | 156 | backup system using a filesystem for storage (disk based) |
Backup tools have their specialized focuses.
sbackup
and keep
packages provide easy GUI frontend for desktop users to make regular backups of user data. An equivalent function can be realized by a simple script (Section 10.1.8, “An example script for the system backup”) and cron
(8).
Basic tools described in Section 10.1.1, “Archive and compression tools” and Section 10.1.2, “Copy and synchronization tools” can be used to facilitate system backup via custom scripts. Such script can be enhanced by the following.
rdiff-backup
package enables incremental (remote) backups.
dump
package helps to archive and restore the whole filesystem incrementally and efficiently.
Tip | |
---|---|
See files in " |
For a personal Debian desktop system running unstable
suite, I only need to protect personal and critical data. I reinstall system once a year anyway. Thus I see no reason to backup the whole system or to install a full featured backup utility.
I use a simple script to make a backup archive and burn it into CD/DVD using GUI. Here is an example script for this.
#!/bin/sh -e # Copyright (C) 2007-2008 Osamu Aoki <osamu@debian.org>, Public Domain BUUID=1000; USER=osamu # UID and name of a user who accesses backup files BUDIR="/var/backups" XDIR0=".+/Mail|.+/Desktop" XDIR1=".+/\.thumbnails|.+/\.?Trash|.+/\.?[cC]ache|.+/\.gvfs|.+/sessions" XDIR2=".+/CVS|.+/\.git|.+/\.svn|.+/Downloads|.+/Archive|.+/Checkout|.+/tmp" XSFX=".+\.iso|.+\.tgz|.+\.tar\.gz|.+\.tar\.bz2|.+\.afio|.+\.tmp|.+\.swp|.+~" SIZE="+99M" DATE=$(date --utc +"%Y%m%d-%H%M") [ -d "$BUDIR" ] || mkdir -p "BUDIR" umask 077 dpkg --get-selections \* > /var/lib/dpkg/dpkg-selections.list debconf-get-selections > /var/cache/debconf/debconf-selections { find /etc /usr/local /opt /var/lib/dpkg/dpkg-selections.list \ /var/cache/debconf/debconf-selections -xdev -print0 find /home/$USER /root -xdev -regextype posix-extended \ -type d -regex "$XDIR0|$XDIR1" -prune -o -type f -regex "$XSFX" -prune -o \ -type f -size "$SIZE" -prune -o -print0 find /home/$USER/Mail/Inbox /home/$USER/Mail/Outbox -print0 find /home/$USER/Desktop -xdev -regextype posix-extended \ -type d -regex "$XDIR2" -prune -o -type f -regex "$XSFX" -prune -o \ -type f -size "$SIZE" -prune -o -print0 } | cpio -ov --null -O $BUDIR/BU$DATE.cpio chown $BUUID $BUDIR/BU$DATE.cpio touch $BUDIR/backup.stamp
This is meant to be a script example executed from root.
I expect you to change and execute this as follows.
find … -print0
" with "find … -newer $BUDIR/backup.stamp -print0
" to make a incremental backup.
scp
(1) or rsync
(1) or burn them to CD/DVD for extra data security. (I use GNOME desktop GUI for burning CD/DVD. See Section 12.1.8, “Shell script example with zenity” for extra redundancy.)
Keep it simple!
Tip | |
---|---|
You can recover debconf configuration data with " |
For the set of data under a directory tree, the copy with "cp -a
" provides the normal backup.
For the set of large non-overwritten static data under a directory tree such as the one under the "/var/cache/apt/packages/
" directory, hardlinks with "cp -al
" provide an alternative to the normal backup with efficient use of the disk space.
Here is a copy script, which I named as bkup
, for the data backup. This script copies all (non-VCS) files under the current directory to the dated directory on the parent directory or on a remote host.
#!/bin/sh -e # Copyright (C) 2007-2008 Osamu Aoki <osamu@debian.org>, Public Domain function fdot(){ find . -type d \( -iname ".?*" -o -iname "CVS" \) -prune -o -print0;} function fall(){ find . -print0;} function mkdircd(){ mkdir -p "$1";chmod 700 "$1";cd "$1">/dev/null;} FIND="fdot";OPT="-a";MODE="CPIOP";HOST="localhost";EXTP="$(hostname -f)" BKUP="$(basename $(pwd)).bkup";TIME="$(date +%Y%m%d-%H%M%S)";BU="$BKUP/$TIME" while getopts gcCsStrlLaAxe:h:T f; do case $f in g) MODE="GNUCP";; # cp (GNU) c) MODE="CPIOP";; # cpio -p C) MODE="CPIOI";; # cpio -i s) MODE="CPIOSSH";; # cpio/ssh S) MODE="AFIOSSH";; # afio/ssh t) MODE="TARSSH";; # tar/ssh r) MODE="RSYNCSSH";; # rsync/ssh l) OPT="-alv";; # hardlink (GNU cp) L) OPT="-av";; # copy (GNU cp) a) FIND="fall";; # find all A) FIND="fdot";; # find non CVS/ .???/ x) set -x;; # trace e) EXTP="${OPTARG}";; # hostname -f h) HOST="${OPTARG}";; # user@remotehost.example.com T) MODE="TEST";; # test find mode \?) echo "use -x for trace." esac; done shift $(expr $OPTIND - 1) if [ $# -gt 0 ]; then for x in $@; do cp $OPT $x $x.$TIME; done elif [ $MODE = GNUCP ]; then mkdir -p "../$BU";chmod 700 "../$BU";cp $OPT . "../$BU/" elif [ $MODE = CPIOP ]; then mkdir -p "../$BU";chmod 700 "../$BU" $FIND|cpio --null --sparse -pvd ../$BU elif [ $MODE = CPIOI ]; then $FIND|cpio -ov --null | ( mkdircd "../$BU"&&cpio -i ) elif [ $MODE = CPIOSSH ]; then $FIND|cpio -ov --null|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&&cpio -i )" elif [ $MODE = AFIOSSH ]; then $FIND|afio -ov -0 -|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&&afio -i - )" elif [ $MODE = TARSSH ]; then (tar cvf - . )|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&& tar xvfp - )" elif [ $MODE = RSYNCSSH ]; then rsync -rlpt ./ "${HOST}:${EXTP}-${BKUP}-${TIME}" else echo "Any other idea to backup?" $FIND |xargs -0 -n 1 echo fi
This is meant to be command examples. Please read script and edit it by yourself before using it.
Tip | |
---|---|
I keep this |
Tip | |
---|---|
For making snapshot history of a source file tree or a configuration file tree, it is easier and space efficient to use |
Removable storage devices may be any one of the following.
These removable storage devices can be automatically mounted as a user under modern desktop environment, such as GNOME using gnome-mount
(1).
Mount point under GNOME is chosen as "/media/<disk_label>
" which can be customized by the following.
mlabel
(1) for FAT filesystem
genisoimage
(1) with "-V
" option for ISO9660 filesystem
tune2fs
(1) with "-L
" option for ext2/ext3 filesystem
Note | |
---|---|
Automounting under modern desktop environment happens only when those removable media devices are not listed in " |
Tip | |
---|---|
When providing wrong mount option causes problem, erase its corresponding setting under " |
Table 10.4. List of packages which permit normal users to mount removable devices without a matching "/etc/fstab
" entry
package | popcon | size | description |
---|---|---|---|
gnome-mount
*
|
V:23, I:40 | 968 | wrapper for (un)mounting and ejecting storage devices (used by GNOME) |
pmount
*
|
V:10, I:35 | 868 | mount removable devices as normal user (used by KDE) |
cryptmount
*
|
V:0.12, I:0.5 | 304 | Management and user-mode mounting of encrypted filesystems |
usbmount
*
|
I:1.8 | 104 | automatically mount and unmount USB storage devices |
When sharing data with other system via removable storage device, you should format it with common filesystem supported by both systems. Here is a list of filesystem choices.
Table 10.5. List of filesystem choices for removable storage devices with typical usage scenarios
filesystem | description of typical usage scenario |
---|---|
FAT12 | cross platform sharing of data on the floppy disk (<32MiB) |
FAT16 | cross platform sharing of data on the small hard disk like device (<2GiB) |
FAT32 | cross platform sharing of data on the large hard disk like device (<8TiB, supported by newer than MS Windows95 OSR2) |
NTFS | cross platform sharing of data on the large hard disk like device (supported natively on MS Windows NT and later version, and supported by NTFS-3G via FUSE on Linux) |
ISO9660 | cross platform sharing of static data on CD-R and DVD+/-R |
UDF | incremental data writing on CD-R and DVD+/-R (new) |
MINIX filesystem | space efficient unix file data storage on the floppy disk |
ext2 filesystem | sharing of data on the hard disk like device with older Linux systems |
ext3 filesystem | sharing of data on the hard disk like device with current Linux systems (journaling filesystem) |
Tip | |
---|---|
See Section 9.4.1, “Removable disk encryption with dm-crypt/LUKS” for cross platform sharing of data using device level encryption. |
The FAT filesystem is supported by almost all modern operating systems and is quite useful for the data exchange purpose via removable hard disk like media.
When formatting removable hard disk like devices for cross platform sharing of data with the FAT filesystem, the following should be safe choices.
Partitioning them with fdisk
(8), cfdisk
(8) or parted
(8) (see Section 9.3.1, “Disk partition configuration”) into a single primary partition and to mark it as the following.
Formatting the primary partition with mkfs.vfat
(8) with the following.
/dev/sda1
" for FAT16
-F 32 /dev/sda1
" for FAT32
When using the FAT or ISO9660 filesystems for sharing data, the following should be the safe considerations.
tar
(1), cpio
(1), or afio
(1) to retain the long filename, the symbolic link, the original Unix file permission and the owner information.
split
(1) command to protect it from the file size limitation.
Note | |
---|---|
For FAT filesystems by its design, the maximum file size is |
Note | |
---|---|
Microsoft itself does not recommend to use FAT for drives or partitions of over 200 MB. Microsoft highlights its short comings such as inefficient disk space usage in their "Overview of FAT, HPFS, and NTFS File Systems". Of course, we should normally use the ext3 filesystem for Linux. |
Tip | |
---|---|
For more on filesystems and accessing filesystems, please read "Filesystems HOWTO". |
When sharing data with other system via network, you should use common service. Here are some hints.
Table 10.6. List of the network service to chose with the typical usage scenario
network service | description of typical usage scenario |
---|---|
SMB/CIFS network mounted filesystem with Samba |
sharing files via "Microsoft Windows Network", see smb.conf (5) and The Official Samba 3.2.x HOWTO and Reference Guide or the samba-doc package
|
NFS network mounted filesystem with the Linux kernel |
sharing files via "Unix/Linux Network", see exports (5) and Linux NFS-HOWTO
|
HTTP service | sharing file between the web server/client |
HTTPS service | sharing file between the web server/client with encrypted Secure Sockets Layer (SSL) or Transport Layer Security (TLS) |
FTP service | sharing file between the FTP server/client |
Although these filesystems mounted over network and file transfer methods over network are quite convenient for sharing data, these may be insecure. Their network connection must be secured by the following.
See also Section 6.10, “Other network application servers” and Section 6.11, “Other network application clients”.
When choosing computer data storage media for important data archive, you should be careful about their limitations. For small personal data backup, I use CD-R and DVD-R by the brand name company and store in a cool, shaded, dry, clean environment. (Tape archive media seem to be popular for professional use.)
Note | |
---|---|
A fire-resistant safe are meant for paper documents. Most of the computer data storage media have less temperature tolerance than paper. I usually rely on multiple secure encrypted copies stored in multiple secure locations. |
Optimistic storage life of archive media seen on the net (mostly from vendor info).
These do not count on the mechanical failures due to handling etc.
Optimistic write cycle of archive media seen on the net (mostly from vendor info).
Caution | |
---|---|
Figures of storage life and write cycle here should not be used for decisions on any critical data storage. Please consult the specific product information provided by the manufacture. |
Tip | |
---|---|
Since CD/DVD-R and paper have only 1 write cycle, they inherently prevent accidental data loss by overwriting. This is advantage! |
Tip | |
---|---|
If you need fast and frequent backup of large amount of data, a hard disk on a remote host linked by a fast network connection, may be the only realistic option. |
Here, we discuss direct manipulation of the binary data on storage media. See Section 9.3, “Data storage tips”, too.
The disk image file, "disk.img
", of an unmounted device, e.g., the second SCSI drive "/dev/sdb
", can be made using cp
(1) or dd
(1) by the following.
# cp /dev/sdb disk.img # dd if=/dev/sdb of=disk.img
The disk image of the traditional PC's master boot record (MBR) (see Section 9.3.1, “Disk partition configuration”) which reside on the first sector on the primary IDE disk can be made by using dd
(1) by the following.
# dd if=/dev/hda of=mbr.img bs=512 count=1 # dd if=/dev/hda of=mbr-nopart.img bs=446 count=1 # dd if=/dev/hda of=mbr-part.img skip=446 bs=1 count=66
mbr.img
": The MBR with the partition table
mbr-nopart.img
": The MBR without the partition table
part.img
": The partition table of the MBR only
If you have a SCSI device (including the new serial ATA drive) as the boot disk, substitute "/dev/hda
" with "/dev/sda
".
If you are making an image of a disk partition of the original disk, substitute "/dev/hda
" with "/dev/hda1
" etc.
The disk image file, "disk.img
" can be written to an unmounted device, e.g., the second SCSI drive "/dev/sdb
" with matching size, by the following.
# dd if=disk.img of=/dev/sdb
Similarly, the disk partition image file, "disk.img
" can be written to an unmounted partition, e.g., the first partition of the second SCSI drive "/dev/sdb1
" with matching size, by the following.
# dd if=disk.img of=/dev/sdb1
If "disk.img
" contains an image of the disk contents and the original disk had a disk configuration which gives xxxx = (bytes/sector) * (sectors/cylinder)
, then the following mounts it to "/mnt
".
# mount -o loop,offset=xxxx disk.img /mnt
Note | |
---|---|
Most hard disks have 512 bytes/sector. |
This offset is to skip MBR of the hard disk. You can skip this offset in the above example, if "disk.img
" contains only the following.
An empty disk image file, "sparse
", which can grow up to 5MiB can be made using dd
(1) and mke2fs
(8) by the following.
$ dd bs=1 count=0 if=/dev/zero of=sparse seek=5G $ /sbin/mke2fs sparse mke2fs 1.41.3 (12-Oct-2008) sparse is not a block special device. Proceed anyway? (y,n) y ... $ du --apparent-size -h sparse 5.0G sparse $ du -h sparse 83M sparse
For "sparse
", its file size is 5.0 GiB and its actual disk usage is mere 83MiB. This discrepancy is possible since ext2fs can hold sparse file.
Tip | |
---|---|
The actual disk usage of sparse file grows with data which are written to it. |
The most basic viewing method of binary data is to use "od -t x1
" command.
Table 10.7. List of packages which view and edit binary data
package | popcon | size | description |
---|---|---|---|
coreutils
*
|
V:91, I:99 | 12868 |
basic package which has od (1) to dump files (HEX, ASCII, OCTAL, …)
|
bsdmainutils
*
|
V:67, I:99 | 644 |
utility package which has hd (1) to dump files (HEX, ASCII, OCTAL, …)
|
hexedit
*
|
V:0.3, I:1.9 | 108 | binary editor and viewer (HEX, ASCII) |
bless
*
|
V:0.05, I:0.3 | 1240 | full featured hexadecimal editor (GNOME) |
okteta
*
|
V:0.4, I:3 | 1388 | full featured hexadecimal editor (KDE4) |
ncurses-hexedit
*
|
V:0.09, I:0.5 | 192 | binary editor and viewer (HEX, ASCII, EBCDIC) |
lde
*
|
V:0.04, I:0.4 | 992 | Linux Disk Editor |
beav
*
|
V:0.04, I:0.3 | 164 | binary editor and viewer (HEX, ASCII, EBCDIC, OCTAL, …) |
hex
*
|
V:0.01, I:0.10 | 84 | hexadecimal dumping tool (support Japanese 2 byte codes) |
Tip | |
---|---|
HEX is used as an acronym for hexadecimal format with radix 16. OCTAL is for octal format with radix 8. ASCII is for American Standard Code for Information Interchange, i.e., normal English text code. EBCDIC is for Extended Binary Coded Decimal Interchange Code used on IBM mainframe operating systems. |
There are tools to read and write files without mounting disk.
Software RAID systems offered by the Linux kernel provide data redundancy in the kernel filesystem level to achieve high levels of storage reliability.
There are tools to add data redundancy to files in application program level to achieve high levels of storage reliability, too.
Table 10.9. List of tools to add data redundancy to files
package | popcon | size | description |
---|---|---|---|
par2
*
|
V:0.5, I:1.7 | 284 | Parity Archive Volume Set, for checking and repair of files |
dvdisaster
*
|
V:0.17, I:0.8 | 1272 | data loss/scratch/aging protection for CD/DVD media |
ras
*
|
V:0.02, I:0.2 | 92 | utility to add redundancy files to archives for data recovery |
dvbackup
*
|
V:0.01, I:0.12 | 544 |
backup tool using MiniDV camcorders (providing rsbep (1))
|
vdmfec
*
|
V:0.00, I:0.02 | 88 | recover lost blocks using Forward Error Correction |
There are tools for data file recovery and forensic analysis.
Table 10.10. List of packages for data file recovery and forensic analysis
package | popcon | size | description |
---|---|---|---|
testdisk
*
|
V:0.3, I:3 | 4616 | utilities for partition scan and disk recovery |
magicrescue
*
|
V:0.08, I:0.5 | 336 | utility to recover files by looking for magic bytes |
scalpel
*
|
V:0.03, I:0.2 | 124 | frugal, high performance file carver |
recover
*
|
V:0.08, I:0.8 | 104 | utility to undelete files on the ext2 filesystem |
e2undel
*
|
V:0.06, I:0.6 | 240 | utility to undelete files on the ext2 filesystem |
ext3grep
*
|
V:0.06, I:0.5 | 300 | tool to help recover deleted files on the ext3 filesystem |
scrounge-ntfs
*
|
V:0.03, I:0.4 | 80 | data recovery program for NTFS filesystems |
gzrt
*
|
V:0.02, I:0.14 | 68 | gzip recovery toolkit |
sleuthkit
*
|
V:0.15, I:0.7 | 4872 | tools for forensics analysis. (Sleuthkit) |
autopsy
*
|
V:0.08, I:0.4 | 1372 | graphical interface to SleuthKit |
foremost
*
|
V:0.09, I:0.6 | 140 | forensics application to recover data |
tct
*
|
V:0.04, I:0.2 | 604 | forensics related utilities |
dcfldd
*
|
V:0.03, I:0.16 | 124 |
enhanced version of dd for forensics and security
|
rdd
*
|
V:0.02, I:0.12 | 200 | forensic copy program |
The ISO9660 image file, "cd.iso
", from the source directory tree at "source_directory
" can be made using genisoimage
(1) provided by cdrkit by the following.
# genisoimage -r -J -T -V volume_id -o cd.iso source_directory
Similary, the bootable ISO9660 image file, "cdboot.iso
", can be made from debian-installer
like directory tree at "source_directory
" by the following.
# genisoimage -r -o cdboot.iso -V volume_id \ -b isolinux/isolinux.bin -c isolinux/boot.cat \ -no-emul-boot -boot-load-size 4 -boot-info-table source_directory
Here Isolinux boot loader (see Section 3.3, “Stage 2: the boot loader”) is used for booting.
To make the disk image directly from the CD-ROM device using cp
(1) or dd
(1) has a few problems. The first run of dd
(1) may cause an error message and may yield a shorter disk image with a lost tail-end. The second run of dd
(1) may yield a larger disk image with garbage data attached at the end on some systems if the data size is not specified. Only the second run of dd
(1) with the correct data size specified, and without ejecting the CD after an error message, seems to avoid these problems. If for example the image size displayed by df
(1) is 46301184 blocks, use the following command twice to get the right image (this is my empirical information).
# dd if=/dev/cdrom of=cd.iso bs=2048 count=$((46301184/2))
Tip | |
---|---|
DVD is only a large CD to |
You can find a usable device by the following.
# wodim --devices
Then the blank CD-R is inserted to the CD drive, and the ISO9660 image file, "cd.iso
" is written to this device, e.g., "/dev/hda
", using wodim
(1) by the following.
# wodim -v -eject dev=/dev/hda cd.iso
If CD-RW is used instead of CD-R, do this instead by the following.
# wodim -v -eject blank=fast dev=/dev/hda cd.iso
Tip | |
---|---|
If your desktop system mounts CD automatically, unmount it by " |
If "cd.iso
" contains an ISO9660 image, then the following manually mounts it to "/cdrom
" by the following.
# mount -t iso9660 -o ro,loop cd.iso /cdrom
Tip | |
---|---|
Modern desktop system mounts removable media automatically (see Section 10.1.10, “Removable storage device”). |
When a data is too big to backup as a single file, you can backup its content after splitting it into, e.g. 2000MiB chunks and merge those chunks back into the original file later.
$ split -b 2000m large_file $ cat x* >large_file
Caution | |
---|---|
Please make sure you do not have any files starting with " |
In order to clear the contents of a file such as a log file, do not use rm
(1) to delete the file and then create a new empty file, because the file may still be accessed in the interval between commands. The following is the safe way to clear the contents of the file.
$ :>file_to_be_cleared
The following commands create dummy or empty files.
$ dd if=/dev/zero of=5kb.file bs=1k count=5 $ dd if=/dev/urandom of=7mb.file bs=1M count=7 $ touch zero.file $ : > alwayszero.file
You should find following files.
5kb.file
" is 5KB of zeros.
7mb.file
" is 7MB of random data.
zero.file
" may be a 0 byte file. If it existed, its mtime
is updated while its content and its length are kept.
alwayszero.file
" is always a 0 byte file. If it existed, its mtime
is updated and its content is reset.
There are several ways to completely erase data from an entire hard disk like device, e.g., USB memory stick at "/dev/sda
".
Caution | |
---|---|
Check your USB memory stick location with |
Erase all the disk content by resetting data to 0 with the following.
# dd if=/dev/zero of=/dev/sda
Erase all by overwriting random data with the following.
# dd if=/dev/urandom of=/dev/sda
Erase all by overwriting random data very efficiently with the following.
# shred -v -n 1 /dev/sda
Since dd
(1) is available from the shell of many bootable Linux CDs such as Debian installer CD, you can erase your installed system completely by running an erase command from such media on the system hard disk, e.g., "/dev/hda
", "/dev/sda
", etc.
Even if you have accidentally deleted a file, as long as that file is still being used by some application (read or write mode), it is possible to recover such a file.
For example, try the following
$ echo foo > bar $ less bar $ ps aux | grep ' less[ ]' bozo 4775 0.0 0.0 92200 884 pts/8 S+ 00:18 0:00 less bar $ rm bar $ ls -l /proc/4775/fd | grep bar lr-x------ 1 bozo bozo 64 2008-05-09 00:19 4 -> /home/bozo/bar (deleted) $ cat /proc/4775/fd/4 >bar $ ls -l -rw-r--r-- 1 bozo bozo 4 2008-05-09 00:25 bar $ cat bar foo
Execute on another terminal (when you have the lsof
package installed) as follows.
$ ls -li bar 2228329 -rw-r--r-- 1 bozo bozo 4 2008-05-11 11:02 bar $ lsof |grep bar|grep less less 4775 bozo 4r REG 8,3 4 2228329 /home/bozo/bar $ rm bar $ lsof |grep bar|grep less less 4775 bozo 4r REG 8,3 4 2228329 /home/bozo/bar (deleted) $ cat /proc/4775/fd/4 >bar $ ls -li bar 2228302 -rw-r--r-- 1 bozo bozo 4 2008-05-11 11:05 bar $ cat bar foo
Files with hardlinks can be identified by "ls -li
".
$ ls -li total 0 2738405 -rw-r--r-- 1 root root 0 2008-09-15 20:21 bar 2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 baz 2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 foo
Both "baz
" and "foo
" have link counts of "2" (>1) showing them to have hardlinks. Their inode numbers are common "2738404". This means they are the same hardlinked file. If you do not happen to find all hardlinked files by chance, you can search it by the inode, e.g., "2738404" as the following.
# find /path/to/mount/point -xdev -inum 2738404
The data security infrastructure is provided by the combination of data encryption tool, message digest tool, and signature tool.
Table 10.11. List of data security infrastructure tools
command | package | popcon | size | description |
---|---|---|---|---|
gpg (1)
|
gnupg
*
|
V:38, I:99 | 5072 | GNU Privacy Guard - OpenPGP encryption and signing tool |
N/A |
gnupg-doc
*
|
I:1.5 | 4124 | GNU Privacy Guard documentation |
gpgv (1)
|
gpgv
*
|
V:58, I:98 | 392 | GNU Privacy Guard - signature verification tool |
paperkey (1)
|
paperkey
*
|
V:0.01, I:0.13 | 88 | extract just the secret information out ouf OpenPGP secret keys |
cryptsetup (8), …
|
cryptsetup
*
|
V:3, I:4 | 912 | utilities for dm-crypto block device encryption supporting LUKS |
ecryptfs (7), …
|
ecryptfs-utils
*
|
V:0.12, I:0.2 | 444 | utilities for ecryptfs stacked filesystem encryption |
md5sum (1)
|
coreutils
*
|
V:91, I:99 | 12868 | compute and check MD5 message digest |
sha1sum (1)
|
coreutils
*
|
V:91, I:99 | 12868 | compute and checks SHA1 message digest |
openssl (1ssl)
|
openssl
*
|
V:31, I:90 | 2352 |
compute message digest with "openssl dgst " (OpenSSL)
|
See Section 9.4, “Data encryption tips” on dm-crypto and ecryptfs which implement automatic data encryption infrastructure via Linux kernel modules.
Here are GNU Privacy Guard commands for the basic key management.
Table 10.12. List of GNU Privacy Guard commands for the key management
command | description |
---|---|
gpg --gen-key
|
generate a new key |
gpg --gen-revoke my_user_ID
|
generate revoke key for my_user_ID |
gpg --edit-key user_ID
|
edit key interactively, "help" for help |
gpg -o file --exports
|
export all keys to file |
gpg --imports file
|
import all keys from file |
gpg --send-keys user_ID
|
send key of user_ID to keyserver |
gpg --recv-keys user_ID
|
recv. key of user_ID from keyserver |
gpg --list-keys user_ID
|
list keys of user_ID |
gpg --list-sigs user_ID
|
list sig. of user_ID |
gpg --check-sigs user_ID
|
check sig. of user_ID |
gpg --fingerprint user_ID
|
check fingerprint of user_ID |
gpg --refresh-keys
|
update local keyring |
Here is the meaning of the trust code.
Table 10.13. List of the meaning of the rust code
code | description of trust |
---|---|
-
|
no owner trust assigned / not yet calculated |
e
|
trust calculation failed |
q
|
not enough information for calculation |
n
|
never trust this key |
m
|
marginally trusted |
f
|
fully trusted |
u
|
ultimately trusted |
The following uploads my key "A8061F32
" to the popular keyserver "hkp://subkeys.pgp.net
".
$ gpg --keyserver hkp://subkeys.pgp.net --send-keys A8061F32
A good default keyserver set up in "~/.gnupg/gpg.conf
" (or old location "~/.gnupg/options
") contains the following.
keyserver hkp://subkeys.pgp.net
The following obtains unknown keys from the keyserver.
$ gpg --list-sigs | \ sed -n '/^sig.*\[User ID not found\]/s/^sig..........\(\w\w*\)\W.*/\1/p' |\ sort | uniq | xargs gpg --recv-keys
There was a bug in OpenPGP Public Key Server (pre version 0.9.6) which corrupted key with more than 2 sub-keys. The newer gnupg
(>1.2.1-2) package can handle these corrupted subkeys. See gpg
(1) under "--repair-pks-subkey-bug
" option.
Here are examples for using GNU Privacy Guard commands on files.
Table 10.14. List of GNU Privacy Guard commands on files
command | description |
---|---|
gpg -a -s file
|
sign file into ascii armored file.asc |
gpg --armor --sign file
|
, , |
gpg --clearsign file
|
clear-sign message |
gpg --clearsign --not-dash-escaped patchfile
|
clear-sign patchfile |
gpg --verify file
|
verify clear-signed file |
gpg -o file.sig -b file
|
create detached signature |
gpg -o file.sig --detach-sig file
|
, , |
gpg --verify file.sig file
|
verify file with file.sig |
gpg -o crypt_file.gpg -r name -e file
|
public-key encryption intended for name from file to binary crypt_file.gpg |
gpg -o crypt_file.gpg --recipient name --encrypt file
|
, , |
gpg -o crypt_file.asc -a -r name -e file
|
public-key encryption intended for name from file to ASCII armored crypt_file.asc |
gpg -o crypt_file.gpg -c file
|
symmetric encryption from file to crypt_file.gpg |
gpg -o crypt_file.gpg --symmetric file
|
, , |
gpg -o crypt_file.asc -a -c file
|
symmetric encryption intended for name from file to ASCII armored crypt_file.asc |
gpg -o file -d crypt_file.gpg -r name
|
decryption |
gpg -o file --decrypt crypt_file.gpg
|
, , |
Add the following to "~/.muttrc
" to keep a slow GnuPG from automatically
starting, while allowing it to be used by typing "S
" at the index menu.
macro index S ":toggle pgp_verify_sig\n" set pgp_verify_sig=no
The gnupg
plugin let you run GnuPG transparently for files with extension ".gpg
", ".asc
", and ".ppg
".
# aptitude install vim-scripts vim-addon-manager $ vim-addons install gnupg
md5sum
(1) provides utility to make a digest file using the method in rfc1321 and verifying each file with it.
$ md5sum foo bar >baz.md5 $ cat baz.md5 d3b07384d113edec49eaa6238ad5ff00 foo c157a79031e1c40f85931829bc5fc552 bar $ md5sum -c baz.md5 foo: OK bar: OK
Note | |
---|---|
The computation for the MD5 sum is less CPU intensive than the one for the cryptographic signature by GNU Privacy Guard (GnuPG). Usually, only the top level digest file is cryptographically signed to ensure data integrity. |
There are many merge tools for the source code. Following commands caught my eyes.
Table 10.15. List of source code merge tools
command | package | popcon | size | description |
---|---|---|---|---|
diff (1)
|
diff
*
|
V:90, I:99 | 764 | compare files line by line |
diff3 (1)
|
diff
*
|
V:90, I:99 | 764 | compare and merges three files line by line |
vimdiff (1)
|
vim
*
|
V:13, I:30 | 1684 | compare 2 files side by side in vim |
patch (1)
|
patch
*
|
V:11, I:93 | 204 | apply a diff file to an original |
dpatch (1)
|
dpatch
*
|
V:2, I:15 | 344 | manage series of patches for Debian package |
diffstat (1)
|
diffstat
*
|
V:2, I:14 | 84 | produce a histogram of changes by the diff |
combinediff (1)
|
patchutils
*
|
V:2, I:16 | 292 | create a cumulative patch from two incremental patches |
dehtmldiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | extract a diff from an HTML page |
filterdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | extract or excludes diffs from a diff file |
fixcvsdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 |
fix diff files created by CVS that patch (1) mis-interprets
|
flipdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | exchange the order of two patches |
grepdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | show which files are modified by a patch matching a regex |
interdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | show differences between two unified diff files |
lsdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | show which files are modified by a patch |
recountdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | recompute counts and offsets in unified context diffs |
rediff (1)
|
patchutils
*
|
V:2, I:16 | 292 | fix offsets and counts of a hand-edited diff |
splitdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | separate out incremental patches |
unwrapdiff (1)
|
patchutils
*
|
V:2, I:16 | 292 | demangle patches that have been word-wrapped |
wiggle (1)
|
wiggle
*
|
V:0.02, I:0.12 | 204 | apply rejected patches |
quilt (1)
|
quilt
*
|
V:0.9, I:6 | 860 | manage series of patches |
meld (1)
|
meld
*
|
V:0.5, I:2 | 2304 | compare and merge files (GTK) |
xxdiff (1)
|
xxdiff
*
|
V:0.2, I:1.2 | 1352 | compare and merge files (plain X) |
dirdiff (1)
|
dirdiff
*
|
V:0.09, I:0.5 | 224 | display differences and merge changes between directory trees |
docdiff (1)
|
docdiff
*
|
V:0.02, I:0.19 | 688 | compare two files word by word / char by char |
imediff2 (1)
|
imediff2
*
|
V:0.01, I:0.11 | 76 | interactive full screen 2-way merge tool |
makepatch (1)
|
makepatch
*
|
V:0.02, I:0.2 | 148 | generate extended patch files |
applypatch (1)
|
makepatch
*
|
V:0.02, I:0.2 | 148 | apply extended patch files |
wdiff (1)
|
wdiff
*
|
V:1.9, I:14 | 124 | display word differences between text files |
One of following procedures extract differences between two source files and create unified diff files "file.patch0
" or "file.patch1
" depending on the file location.
$ diff -u file.old file.new > file.patch0 $ diff -u old/file new/file > file.patch1
The diff file (alternatively called patch file) is used to send a program update. The receiving party applies this update to another file by the following.
$ patch -p0 file < file.patch0 $ patch -p1 file < file.patch1
Here is a summary of the version control systems (VCS) on the Debian system.
Note | |
---|---|
If you are new to VCS systems, you should start learning with Git, which is growing fast in popularity. |
Table 10.16. List of version control system tools
package | popcon | size | tool | VCS type | comment |
---|---|---|---|---|---|
cssc
*
|
V:0.01, I:0.05 | 2168 | CSSC | local | clone of the Unix SCCS (deprecated) |
rcs
*
|
V:1.6, I:9 | 772 | RCS | local | "Unix SCCS done right" |
cvs
*
|
V:4, I:24 | 3660 | CVS | remote | previous standard remote VCS |
subversion
*
|
V:9, I:32 | 4256 | Subversion | remote | "CVS done right", the new de facto standard remote VCS |
git-core
*
|
V:6, I:11 | 14036 | Git | distributed | fast DVCS in C (used by the Linux kernel and others) |
mercurial
*
|
V:1.1, I:4 | 332 | Mercurial | distributed | DVCS in Python and some C |
bzr
*
|
V:0.6, I:2 | 20300 | Bazaar | distributed |
DVCS influenced by tla written in Python (used by Ubuntu)
|
darcs
*
|
V:0.2, I:1.6 | 8112 | Darcs | distributed | DVCS with smart algebra of patches (slow) |
tla
*
|
V:0.19, I:1.4 | 1100 | GNU arch | distributed | DVCS mainly by Tom Lord (Historic) |
monotone
*
|
V:0.05, I:0.4 | 4752 | Monotone | distributed | DVCS in C++ |
tkcvs
*
|
V:0.07, I:0.4 | 2384 | CVS, … | remote | GUI display of VCS (CVS, Subversion, RCS) repository tree |
gitk
*
|
V:0.8, I:3 | 756 | Git | distributed | GUI display of VCS (Git) repository tree |
VCS is sometimes known as revision control system (RCS), or software configuration management (SCM).
Distributed VCS such as Git is the tool of choice these days. CVS and Subversion may still be useful to join some existing open source program activities.
Debian provides free VCS services via Debian Alioth service. It supports practically all VCSs. Its documentation can be found at http://wiki.debian.org/Alioth .
Caution | |
---|---|
The |
Here is an oversimplified comparison of native VCS commands to provide the big picture. The typical command sequence may require options and arguments.
Table 10.17. Comparison of native VCS commands
CVS | Subversion | Git | function |
---|---|---|---|
cvs init
|
svn create
|
git init
|
create the (local) repository |
cvs login
|
- | - | login to the remote repository |
cvs co
|
svn co
|
git clone
|
check out the remote repository as the working tree |
cvs up
|
svn up
|
git pull
|
update the working tree by merging the remote repository |
cvs add
|
svn add
|
git add .
|
add file(s) in the working tree to the VCS |
cvs rm
|
svn rm
|
git rm
|
remove file(s) in working tree from the VCS |
cvs ci
|
svn ci
|
- | commit changes to the remote repository |
- | - |
git commit -a
|
commit changes to the local repository |
- | - |
git push
|
update the remote repository by the local repository |
cvs status
|
svn status
|
git status
|
display the working tree status from the VCS |
cvs diff
|
svn diff
|
git diff
|
diff <reference_repository> <working_tree> |
- | - |
git repack -a -d; git prune
|
repack the local repository into single pack |
tkcvs
|
tkcvs
|
gitk
|
GUI display of VCS repository tree |
Caution | |
---|---|
Invoking a |
Tip | |
---|---|
GUI tools such as |
Tip | |
---|---|
Git can work directly with different VCS repositories such as ones provided by CVS and Subversion, and provides the local repository for local changes with |
Tip | |
---|---|
Git has commands which have no equivalents in CVS and Subversion: "fetch", "rebase", "cherrypick", … |
See the following.
cvs
(1)
/usr/share/doc/cvs/html-cvsclient
"
/usr/share/doc/cvs/html-info
"
/usr/share/doc/cvsbook
"
info cvs
"
The following configuration allows commits to the CVS repository only by a member of the "src
" group, and administration of CVS only by a member of the "staff
" group, thus reducing the chance of shooting oneself.
# cd /var/lib; umask 002; mkdir cvs # export CVSROOT=/srv/cvs/project # cd $CVSROOT # chown root:src . # chmod 2775 . # cvs -d $CVSROOT init # cd CVSROOT # chown -R root:staff . # chmod 2775 . # touch val-tags # chmod 664 history val-tags # chown root:src history val-tags
Tip | |
---|---|
You may restrict creation of new project by changing the owner of " |
The default CVS repository is pointed by "$CVSROOT
". The following sets up "$CVSROOT
" for the local access.
$ export CVSROOT=/srv/cvs/project
Many public CVS servers provide read-only remote access to them with account name "anonymous
" via pserver service. For example, Debian web site contents are maintained by webwml project via CVS at Debian alioth service. The following sets up "$CVSROOT
" for the remote access to this CVS repository.
$ export CVSROOT=:pserver:anonymous@cvs.alioth.debian.org:/cvsroot/webwml $ cvs login
Note | |
---|---|
Since pserver is prone to eavesdropping attack and insecure, write access is usually disable by server administrators. |
The following sets up "$CVS_RSH
" and "$CVSROOT
" for the remote access to the CVS repository by webwml project with SSH.
$ export CVS_RSH=ssh $ export CVSROOT=:ext:account@cvs.alioth.debian.org:/cvs/webwml
You can also use public key authentication for SSH which eliminates the remote password prompt.
Create a new local source tree location at "~/path/to/module1
" by the following.
$ mkdir -p ~/path/to/module1; cd ~/path/to/module1
Populate a new local source tree under "~/path/to/module1
" with files.
Import it to CVS with the following parameters.
module1
"
Main-branch
" (tag for the entire branch)
Release-initial
" (tag for a specific release)
$ cd ~/path/to/module1 $ cvs import -m "Start module1" module1 Main-branch Release-initial $ rm -Rf . # optional
CVS does not overwrite the current repository file but replaces it with another one. Thus, write permission to the repository directory is critical. For every new module for "module1
" in repository at "/srv/cvs/project
", run the following to ensure this condition if needed.
# cd /srv/cvs/project # chown -R root:src module1 # chmod -R ug+rwX module1 # chmod 2775 module1
Here is an example of typical work flow using CVS.
Check all available modules from CVS project pointed by "$CVSROOT
" by the following.
$ cvs rls CVSROOT module1 module2 ...
Checkout "module1
" to its default directory "./module1
" by the following.
$ cd ~/path/to $ cvs co module1 $ cd module1
Make changes to the content as needed.
Check changes by making "diff -u [repository] [local]
" equivalent by the following.
$ cvs diff -u
You find that you broke some file "file_to_undo
" severely but other files are fine.
Overwrite "file_to_undo
" file with the clean copy from CVS by the following.
$ cvs up -C file_to_undo
Save the updated local source tree to CVS by the following.
$ cvs ci -m "Describe change"
Create and add "file_to_add
" file to CVS by the following.
$ vi file_to_add $ cvs add file_to_add $ cvs ci -m "Added file_to_add"
Merge the latest version from CVS by the following.
$ cvs up -d
Watch out for lines starting with "C filename
" which indicates conflicting changes.
Look for unmodified code in ".#filename.version
".
Search for "<<<<<<<
" and ">>>>>>>
" in files for conflicting changes.
Edit files to fix conflicts as needed.
Add a release tag "Release-1
" by the following.
$ cvs ci -m "last commit for Release-1" $ cvs tag Release-1
Edit further.
Remove the release tag "Release-1
" by the following.
$ cvs tag -d Release-1
Check in changes to CVS by the following.
$ cvs ci -m "real last commit for Release-1"
Re-add the release tag "Release-1
" to updated CVS HEAD of main by the following.
$ cvs tag Release-1
Create a branch with a sticky branch tag "Release-initial-bugfixes
" from the original version pointed by the tag "Release-initial
" and check it out to "~/path/to/old
" directory by the following.
Tip | |
---|---|
Use " |
$ cvs rtag -b -r Release-initial Release-initial-bugfixes module1 $ cd ~/path/to $ cvs co -r Release-initial-bugfixes -d old module1 $ cd old
Work on this local source tree having the sticky tag "Release-initial-bugfixes
" which is based on the original version.
Work on this branch by yourself … until someone else joins to this "Release-initial-bugfixes
" branch.
Sync with files modified by others on this branch while creating new directories as needed by the following.
$ cvs up -d
Edit files to fix conflicts as needed.
Check in changes to CVS by the following.
$ cvs ci -m "checked into this branch"
Update the local tree by HEAD of main while removing sticky tag ("-A
") and without keyword expansion ("-kk
") by the following.
$ cvs up -d -kk -A
Update the local tree (content = HEAD of main) by merging from the "Release-initial-bugfixes
" branch and without keyword expansion by the following.
$ cvs up -d -kk -j Release-initial-bugfixes
Fix conflicts with editor.
Check in changes to CVS by the following.
$ cvs ci -m "merged Release-initial-bugfixes"
Make archive by the following.
$ cd .. $ mv old old-module1-bugfixes $ tar -cvzf old-module1-bugfixes.tar.gz old-module1-bugfixes $ rm -rf old-module1-bugfixes
Tip | |
---|---|
" |
Tip | |
---|---|
You can checkout only a sub directory of " |
Table 10.18. Notable options for CVS commands (use as first argument(s) to cvs
(1))
option | meaning |
---|---|
-n
|
dry run, no effect |
-t
|
display messages showing steps of cvs activity |
To get the latest files from CVS, use "tomorrow
" by the following.
$ cvs ex -D tomorrow module_name
Add module alias "mx
" to a CVS project (local server) by the following.
$ export CVSROOT=/srv/cvs/project $ cvs co CVSROOT/modules $ cd CVSROOT $ echo "mx -a module1" >>modules $ cvs ci -m "Now mx is an alias for module1" $ cvs release -d .
Now, you can check out "module1
" (alias: "mx
") from CVS to "new
" directory by the following.
$ cvs co -d new mx $ cd new
Note | |
---|---|
In order to perform above procedure, you should have appropriate file permissions. |
When you checkout files from CVS, their execution permission bit is retained.
Whenever you see execution permission problems in a checked out file, e.g. "filename
", change its permission in the corresponding CVS repository by the following to fix it.
# chmod ugo-x filename
Subversion is a recent-generation version control system replacing older CVS. It has most of CVS's features except tags and branches.
You need to install subversion
, libapache2-svn
and subversion-tools
packages to set up a Subversion server.
Currently, the subversion
package does not set up a repository, so one must set it up manually. One possible location for a repository is in "/srv/svn/project
".
Create a directory by the following.
# mkdir -p /srv/svn/project
Create the repository database by the following.
# svnadmin create /srv/svn/project
If you only access Subversion repository via Apache2 server, you just need to make the repository only writable by the WWW server by the following.
# chown -R www-data:www-data /srv/svn/project
Add (or uncomment) the following in "/etc/apache2/mods-available/dav_svn.conf
" to allow access to the repository via user authentication.
<Location /project> DAV svn SVNPath /srv/svn/project AuthType Basic AuthName "Subversion repository" AuthUserFile /etc/subversion/passwd <LimitExcept GET PROPFIND OPTIONS REPORT> Require valid-user </LimitExcept> </Location>
Create a user authentication file with the command by the following.
# htpasswd2 -c /etc/subversion/passwd some-username
Restart Apache2.
Your new Subversion repository is accessible at URL "http://localhost/project
" and "http://example.com/project
" from svn
(1) (assuming your URL of web server is "http://example.com/
").
The following sets up Subversion repository for the local access by a group, e.g. project
.
# chmod 2775 /srv/svn/project # chown -R root:src /srv/svn/project # chmod -R ug+rwX /srv/svn/project
Your new Subversion repository is group accessible at URL "file:///localhost/srv/svn/project
" or "file:///srv/svn/project
" from svn
(1) for local users belonging to project
group. You must run commands, such as svn
, svnserve
, svnlook
, and svnadmin
under "umask 002
" to ensure group access.
A group accessible Subversion repository is at URL "example.com:/srv/svn/project
" for SSH, you can access it from svn
(1) at URL "svn+ssh://example.com:/srv/svn/project
".
Many projects uses directory tree similar to the following for Subversion to compensate its lack of branches and tags.
----- module1 | |-- branches | |-- tags | | |-- release-1.0 | | `-- release-2.0 | | | `-- trunk | |-- file1 | |-- file2 | `-- file3 | `-- module2
Tip | |
---|---|
You must use " |
Create a new local source tree location at "~/path/to/module1
" by the following.
$ mkdir -p ~/path/to/module1; cd ~/path/to/module1
Populate a new local source tree under "~/path/to/module1
" with files.
Import it to Subversion with the following parameters.
module1
"
file:///srv/svn/project
"
module1/trunk
"
module1/tags/Release-initial
"
$ cd ~/path/to/module1 $ svn import file:///srv/svn/project/module1/trunk -m "Start module1" $ svn cp file:///srv/svn/project/module1/trunk file:///srv/svn/project/module1/tags/Release-initial
Alternatively, by the following.
$ svn import ~/path/to/module1 file:///srv/svn/project/module1/trunk -m "Start module1" $ svn cp file:///srv/svn/project/module1/trunk file:///srv/svn/project/module1/tags/Release-initial
Tip | |
---|---|
You can replace URLs such as " |
Here is an example of typical work flow using Subversion.
Check all available modules from Subversion project pointed by URL "file:///srv/svn/project
" by the following.
$ svn list file:///srv/svn/project module1 module2 ...
Checkout "module1/trunk
" to a directory "module1
" by the following.
$ cd ~/path/to $ svn co file:///srv/svn/project/module1/trunk module1 $ cd module1
Make changes to the content as needed.
Check changes by making "diff -u [repository] [local]
" equivalent by the following.
$ svn diff
You find that you broke some file "file_to_undo
" severely but other files are fine.
Overwrite "file_to_undo
" file with the clean copy from Subversion by the following.
$ svn revert file_to_undo
Save the updated local source tree to Subversion by the following.
$ svn ci -m "Describe change"
Create and add "file_to_add
" file to Subversion by the following.
$ vi file_to_add $ svn add file_to_add $ svn ci -m "Added file_to_add"
Merge the latest version from Subversion by the following.
$ svn up
Watch out for lines starting with "C filename
" which indicates conflicting changes.
Look for unmodified code in, e.g., "filename.r6
", "filename.r9
", and "filename.mine
".
Search for "<<<<<<<
" and ">>>>>>>
" in files for conflicting changes.
Edit files to fix conflicts as needed.
Add a release tag "Release-1
" by the following.
$ svn ci -m "last commit for Release-1" $ svn cp file:///srv/svn/project/module1/trunk file:///srv/svn/project/module1/tags/Release-1
Edit further.
Remove the release tag "Release-1
" by the following.
$ svn rm file:///srv/svn/project/module1/tags/Release-1
Check in changes to Subversion by the following.
$ svn ci -m "real last commit for Release-1"
Re-add the release tag "Release-1
" from updated Subversion HEAD of trunk by the following.
$ svn cp file:///srv/svn/project/module1/trunk file:///srv/svn/project/module1/tags/Release-1
Create a branch with a path "module1/branches/Release-initial-bugfixes
" from the original version pointed by the path "module1/tags/Release-initial
" and check it out to "~/path/to/old
" directory by the following.
Tip | |
---|---|
Use " |
$ svn cp file:///srv/svn/project/module1/tags/Release-initial file:///srv/svn/project/module1/branches/Release-initial-bugfixes $ cd ~/path/to $ svn co file:///srv/svn/project/module1/branches/Release-initial-bugfixes old $ cd old
Work on this local source tree pointing to branch "Release-initial-bugfixes
" which is based on the original version.
Work on this branch by yourself … until someone else joins to this "Release-initial-bugfixes
" branch.
Sync with files modified by others on this branch by the following.
$ svn up
Edit files to fix conflicts as needed.
Check in changes to Subversion by the following.
$ svn ci -m "checked into this branch"
Update the local tree with HEAD of trunk by the following.
$ svn switch file:///srv/svn/project/module1/trunk
Update the local tree (content = HEAD of trunk) by merging from the "Release-initial-bugfixes
" branch by the following.
$ svn merge file:///srv/svn/project/module1/branches/Release-initial-bugfixes
Fix conflicts with editor.
Check in changes to Subversion by the following.
$ svn ci -m "merged Release-initial-bugfixes"
Make archive by the following.
$ cd .. $ mv old old-module1-bugfixes $ tar -cvzf old-module1-bugfixes.tar.gz old-module1-bugfixes $ rm -rf old-module1-bugfixes
Tip | |
---|---|
You can replace URLs such as " |
Tip | |
---|---|
You can checkout only a sub directory of " |
Table 10.19. Notable options for Subversion commands (use as first argument(s) to svn
(1))
option | meaning |
---|---|
--dry-run
|
dry run, no effect |
-v
|
display detail messages of svn activity |
Git can do everything for both local and remote source code management. This means that you can record the source code changes without needing network connectivity to the remote repository.
You may wish to set several global configuration in "~/.gitconfig
" such as your name and email address used by Git by the following.
$ git config --global user.name "Name Surname" $ git config --global user.email yourname@example.com
If you are too used to CVS or Subversion commands, you may wish to set several command aliases by the following.
$ git config --global alias.ci "commit -a" $ git config --global alias.co checkout
You can check your global configuration by the following.
$ git config --global --list
See the following.
/usr/share/doc/git-doc/git.html
)
/usr/share/doc/git-doc/user-manual.html
)
/usr/share/doc/git-doc/gittutorial.html
)
/usr/share/doc/git-doc/gittutorial-2.html
)
/usr/share/doc/git-doc/everyday.html
)
git for CVS users (/usr/share/doc/git-doc/gitcvs-migration.html
)
Other git resources available on the web
/usr/share/doc/gitmagic/html/index.html
)
git-gui
(1) and gitk
(1) commands make using Git very easy.
Warning | |
---|---|
Do not use the tag string with spaces in it even if some tools such as |
Even if your upstream uses different VCS, it may be good idea to use git
(1) for local activity since you can manage your local copy of source tree without the network connection to the upstream. Here are some packages and commands used with git
(1).
Table 10.20. List of git related packages and commands
command | package | popcon | size | description |
---|---|---|---|---|
N/A |
git-doc
*
|
I:2 | 5808 | official documentation for Git |
N/A |
gitmagic
*
|
I:0.2 | 560 | "Git Magic", easier to understand guide for Git |
git (7)
|
git-core
*
|
V:6, I:11 | 14036 | Git, the fast, scalable, distributed revision control system |
gitk (1)
|
gitk
*
|
V:0.8, I:3 | 756 | GUI Git repository browser with history |
git-gui (1)
|
git-gui
*
|
V:0.3, I:2 | 1432 | GUI for Git (No history) |
git-svnimport (1)
|
git-svn
*
|
V:0.4, I:2 | 496 | import the data out of Subversion into Git |
git-svn (1)
|
git-svn
*
|
V:0.4, I:2 | 496 | provide bidirectional operation between the Subversion and Git |
git-cvsimport (1)
|
git-cvs
*
|
V:0.18, I:1.4 | 624 | import the data out of CVS into Git |
git-cvsexportcommit (1)
|
git-cvs
*
|
V:0.18, I:1.4 | 624 | export a commit to a CVS checkout from Git |
git-cvsserver (1)
|
git-cvs
*
|
V:0.18, I:1.4 | 624 | CVS server emulator for Git |
git-send-email (1)
|
git-email
*
|
V:0.11, I:1.3 | 368 | send a collection of patches as email from the Git |
stg (1)
|
stgit
*
|
V:0.08, I:0.6 | 844 | quilt on top of git (Python) |
git-buildpackage (1)
|
git-buildpackage
*
|
V:0.14, I:0.9 | 448 | automate the Debian packaging with the Git |
guilt (7)
|
guilt
*
|
V:0.02, I:0.11 | 336 | quilt on top of git (SH/AWK/SED/…) |
You can manually record chronological history of configuration using Git tools. Here is a simple example for your practice to record "/etc/apt/
" contents.
$ cd /etc/apt/ $ sudo git init $ sudo chmod 700 .git $ sudo git add . $ sudo git commit -a
Commit configuration with description.
Make modification to the configuration files.
$ cd /etc/apt/ $ sudo git commit -a
Commit configuration with description and continue your life.
$ cd /etc/apt/ $ sudo gitk --all
You have full configuration history with you.
Note | |
---|---|
|
Note | |
---|---|
The " |
Tip | |
---|---|
For more complete setup for recording configuration history, please look for the |