DSL Tips and Tricks :: declobber - a tool for extension rebuilding

Robert has complained that people are sending the occasional badly-made .dsl extension to the repo containing one or more system-clobbering directories. (An unsubtle hypothetical example might be eg /usr/local/bin, which would wipe out your entire /usr/local/bin tree when the extension is installed).

I suspect these often result from leaving out the --no-recursion tar option when repacking an extension, or from not appropriately trimming a file list for feeding to tar with -T.

Anyway ... I've been using this tool for some time to strip clobber dirs from extensions (similarities to dsl2unc).

Try running it on some of the larger .dsl extensions in the repo - you might get a surprise!

Code Sample

#!/bin/bash

# declobber v0.1 by WDef
# Removes clobber dirs from .dsl extensions

# Just do ./declobber <extension1.dsl extension2.dsl ..>
# Output is extension1_noclobber.dsl etc in the WORK dir

# For livecd use with legacy boot.

#==========================// USER SETTING//===============================

# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl

#==========================// FNS //========================================

dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
# empty dir - check not already on base system
if [ -e "/KNOPPIX/${LM}" ]; then
# this is a clobberring dir !
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
else
# this is a clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}

#=========================// MAIN //=======================================

. /etc/init.d/dsl-functions # for ANSI colors

if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
echo "${RED}declobber is for livecd use with legacy boot option only.${NORMAL}"
exit 1
fi

if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi

results=/tmp/declobber_results

rm -f ${results} # preclean

CURRENTD="${PWD}"
APP=""

for APP in $*; do

# If no path supplied, assume current working dir

if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi

mkdir ${WORK}
touch "${clobber_list}" # zero count initialised

echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1 # comment this out if you're in a hurry

cd ${WORK}
tar -zxpvf $APP
echo -n "${WHITE}Working ..${NORMAL} "
tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!

NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi

# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi

echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"

CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "( These are listed in $clobber_list )" >>${results}
cd ${CURRENTD}
done

echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"

exit 0

Thanks WDef,

I will add this to the tools section in v3.3RC1

It's occurred to me that the requirement to run only as livecd legacy boot is probably overkill and only really matters when the extension contains an empty dir that might be on the system. Also, if other extensions happen to be installed on the system, well, we don't want to clobber those either.

So I might post another version giving this as a warning only when it is required.

Unsightly bug:

Code Sample

tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &

Should read:

Code Sample

find -name \* | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &

Apologies. Cleaned-up version coming.

EDIT: Can't reproduce the alleged-four-in-the-morning bug. Ignoring.

Code Sample

#!/bin/bash

# declobber v0.2 by wdef Feb 9 2007
# Removes clobber dirs from .dsl extensions

# Changes v0.2
# 1. If used on livecd legacy boot or equivalent, can differentiate between isolated empty clobber dir
# that also occurs on base system and one that is non-base (ie added by user or other extension).
# This is noted in the results for information. However, isolated empty dirs are infrequent in
# extensions.
# 2. Help and message related to (1) added.
# 3. Added checks for work dir
# 4. Minor changes to output.

#==========================// USER SETTING//===============================

# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl

#==========================// FNS //========================================

help(){
cat <<"EOF1"
declobber v0.2 - removes clobber directories from damnsmalllinux extensions
by wdef
Usage: declobber <ext1.dsl ext2.dsl ..>
Run as root to preserve all permissions.
Creates a work dir in $DIR
Declobbered extension is output in $DIR
Default DIR=/home/dsl
EOF1
exit
}

check_work_dir(){
# Check work directory looks ok
if [ -z "$DIR" ]; then "${RED}User setting work dir is unset.${NORMAL}"; exit 1; fi
case ${DIR} in
/mnt/*) D=$(echo ${DIR} | awk -F/ '{print "/" $2 "/" $3 }' )
FS=$(grep ${D} /proc/mounts | awk '{print $3}')
if [ -z "$FS" ]; then echo "${RED}$D not mounted.${NORMAL}"; exit 1; fi
case $FS in
ext*|*eiser*);;
*fat*|*dos*) echo "${RED}$DIR on $FS can't preserve perms - use linux filesystem.${NORMAL}"; exit 1;;
ntfs) echo "${RED}Writing to ntfs is not a good idea ${NORMAL}"; exit 1;;
esac;;
/*|/ramdisk/*);;
*) echo "${RED}Err .. where exactly _is_ this $DIR anyway?"${NORMAL}; exit 1;;
esac
if [ ! -d "${DIR}" ] || [ ! -w "${DIR}" ]; then
echo "${RED}$DIR not a dir or not writeable.${NORMAL}"; exit 1
fi
}

not_legacy_live_msg(){
cat <<"EOF2"

This is not a legacy livecd boot.
If your extension contains isolated empty directories that are also found
on the system, declobber cannot be certain if these are clobberring base
system directories or directories added by user/other extensions, or both.
I will note any such dirs in the results.
EOF2
}

checkon_system(){
# Remove empty dirs if already on system
# Make note in results of empty dirs that appear to clobber non-base dirs (eg added by other extensions)
# or if declobber can't tell.

if $LEGCD; then
if [ -e "/${LM}" ]; then
if [ ! -e "/KNOPPIX/${LM}" ]; then
echo ${LM} >>${clobber_list}
echo "Removing empty dir /${LM} - clobbers non-base added directory" >>${results}
else
# this *is* a system clobberring dir !
echo ${LM} >>${clobber_list}
fi
else
echo ${LM} # OK
fi
else
if [ -e "/${LM}" ]; then
echo "Removing empty dir /${LM} - clobbers either base or added directory" >>${results}
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
fi
}

dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
checkon_system
else
# this *is* a self and/or system clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}

#=========================// MAIN //=======================================

. /etc/init.d/dsl-functions # for ANSI colors

if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi

check_work_dir

if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
LEGCD=false;
not_legacy_live_msg
else
LEGCD=true;
fi

if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi

results=/tmp/declobber_results

rm -f ${results} # preclean

CURRENTD="${PWD}"
APP=""

for APP in $*; do

echo ".........................................................................."
echo >>${results}
# If no path supplied, assume current working dir

if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi

mkdir ${WORK}

echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1

cd ${WORK}
touch "${clobber_list}" # zero count initialised
tar -zxpvf ${APP}
echo -n "${WHITE}Working ..${NORMAL} "

tar -ztf ${APP} | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!

NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi

# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi

echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"

CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "See ${clobber_list}" >>${results}
cd ${CURRENTD}
done

echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"

exit 0

Next Page...
original here.