declobber - a tool for extension rebuilding
Forum: DSL Tips and Tricks
Topic: declobber - a tool for extension rebuilding
started by: WDef
Posted by WDef on Feb. 03 2007,21:54
Robert has complained that people are sending the occasional badly-made .dsl extension to the repo containing one or more system-clobbering directories. (An unsubtle hypothetical example might be eg /usr/local/bin, which would wipe out your entire /usr/local/bin tree when the extension is installed).
I suspect these often result from leaving out the --no-recursion tar option when repacking an extension, or from not appropriately trimming a file list for feeding to tar with -T.
Anyway ... I've been using this tool for some time to strip clobber dirs from extensions (similarities to dsl2unc).
Try running it on some of the larger .dsl extensions in the repo - you might get a surprise!
| Code Sample |
#!/bin/bash
# declobber v0.1 by WDef
# Removes clobber dirs from .dsl extensions
# Just do ./declobber <extension1.dsl extension2.dsl ..>
# Output is extension1_noclobber.dsl etc in the WORK dir
# For livecd use with legacy boot.
#==========================// USER SETTING//===============================
# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl
#==========================// FNS //========================================
dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
# empty dir - check not already on base system
if [ -e "/KNOPPIX/${LM}" ]; then
# this is a clobberring dir !
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
else
# this is a clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}
#=========================// MAIN //=======================================
. /etc/init.d/dsl-functions # for ANSI colors
if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
echo "${RED}declobber is for livecd use with legacy boot option only.${NORMAL}"
exit 1
fi
if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi
results=/tmp/declobber_results
rm -f ${results} # preclean
CURRENTD="${PWD}"
APP=""
for APP in $*; do
# If no path supplied, assume current working dir
if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi
if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi
if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi
NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list
if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi
mkdir ${WORK}
touch "${clobber_list}" # zero count initialised
echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1 # comment this out if you're in a hurry
cd ${WORK}
tar -zxpvf $APP
echo -n "${WHITE}Working ..${NORMAL} "
tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!
NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi
# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi
echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"
CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "( These are listed in $clobber_list )" >>${results}
cd ${CURRENTD}
done
echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"
exit 0
|
Posted by roberts on Feb. 03 2007,23:04
Thanks WDef,
I will add this to the tools section in v3.3RC1
Posted by WDef on Feb. 06 2007,12:43
It's occurred to me that the requirement to run only as livecd legacy boot is probably overkill and only really matters when the extension contains an empty dir that might be on the system. Also, if other extensions happen to be installed on the system, well, we don't want to clobber those either.
So I might post another version giving this as a warning only when it is required.
Posted by WDef on Feb. 08 2007,00:51
Unsightly bug:
| Code Sample | | tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} & |
Should read:
| Code Sample | | find -name \* | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} & |
Apologies. Cleaned-up version coming.
EDIT: Can't reproduce the alleged-four-in-the-morning bug. Ignoring.
Posted by WDef on Feb. 09 2007,15:26
| Code Sample |
#!/bin/bash
# declobber v0.2 by wdef Feb 9 2007
# Removes clobber dirs from .dsl extensions
# Changes v0.2
# 1. If used on livecd legacy boot or equivalent, can differentiate between isolated empty clobber dir
# that also occurs on base system and one that is non-base (ie added by user or other extension).
# This is noted in the results for information. However, isolated empty dirs are infrequent in
# extensions.
# 2. Help and message related to (1) added.
# 3. Added checks for work dir
# 4. Minor changes to output.
#==========================// USER SETTING//===============================
# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl
#==========================// FNS //========================================
help(){
cat <<"EOF1"
declobber v0.2 - removes clobber directories from damnsmalllinux extensions
by wdef
Usage: declobber <ext1.dsl ext2.dsl ..>
Run as root to preserve all permissions.
Creates a work dir in $DIR
Declobbered extension is output in $DIR
Default DIR=/home/dsl
EOF1
exit
}
check_work_dir(){
# Check work directory looks ok
if [ -z "$DIR" ]; then "${RED}User setting work dir is unset.${NORMAL}"; exit 1; fi
case ${DIR} in
/mnt/*) D=$(echo ${DIR} | awk -F/ '{print "/" $2 "/" $3 }' )
FS=$(grep ${D} /proc/mounts | awk '{print $3}')
if [ -z "$FS" ]; then echo "${RED}$D not mounted.${NORMAL}"; exit 1; fi
case $FS in
ext*|*eiser*);;
*fat*|*dos*) echo "${RED}$DIR on $FS can't preserve perms - use linux filesystem.${NORMAL}"; exit 1;;
ntfs) echo "${RED}Writing to ntfs is not a good idea ${NORMAL}"; exit 1;;
esac;;
/*|/ramdisk/*);;
*) echo "${RED}Err .. where exactly _is_ this $DIR anyway?"${NORMAL}; exit 1;;
esac
if [ ! -d "${DIR}" ] || [ ! -w "${DIR}" ]; then
echo "${RED}$DIR not a dir or not writeable.${NORMAL}"; exit 1
fi
}
not_legacy_live_msg(){
cat <<"EOF2"
This is not a legacy livecd boot.
If your extension contains isolated empty directories that are also found
on the system, declobber cannot be certain if these are clobberring base
system directories or directories added by user/other extensions, or both.
I will note any such dirs in the results.
EOF2
}
checkon_system(){
# Remove empty dirs if already on system
# Make note in results of empty dirs that appear to clobber non-base dirs (eg added by other extensions)
# or if declobber can't tell.
if $LEGCD; then
if [ -e "/${LM}" ]; then
if [ ! -e "/KNOPPIX/${LM}" ]; then
echo ${LM} >>${clobber_list}
echo "Removing empty dir /${LM} - clobbers non-base added directory" >>${results}
else
# this *is* a system clobberring dir !
echo ${LM} >>${clobber_list}
fi
else
echo ${LM} # OK
fi
else
if [ -e "/${LM}" ]; then
echo "Removing empty dir /${LM} - clobbers either base or added directory" >>${results}
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
fi
}
dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
checkon_system
else
# this *is* a self and/or system clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}
#=========================// MAIN //=======================================
. /etc/init.d/dsl-functions # for ANSI colors
if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi
check_work_dir
if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
LEGCD=false;
not_legacy_live_msg
else
LEGCD=true;
fi
if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi
results=/tmp/declobber_results
rm -f ${results} # preclean
CURRENTD="${PWD}"
APP=""
for APP in $*; do
echo ".........................................................................."
echo >>${results}
# If no path supplied, assume current working dir
if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi
if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi
if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi
NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list
if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi
mkdir ${WORK}
echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1
cd ${WORK}
touch "${clobber_list}" # zero count initialised
tar -zxpvf ${APP}
echo -n "${WHITE}Working ..${NORMAL} "
tar -ztf ${APP} | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!
NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi
# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi
echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"
CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "See ${clobber_list}" >>${results}
cd ${CURRENTD}
done
echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"
exit 0
|
Posted by WDef on Feb. 10 2007,00:29
Wouldn't make any difference other than for a botched extension containing multiple entries having the same filename and path (possible with tar), but uniq -u should be uniq in the pipe feeding dir_filter:
| Code Sample |
#!/bin/bash
# declobber v0.2.1 by wdef Feb 10 2007
# Removes clobber dirs from .dsl extensions
# Changes v0.2.1
# uniq -u -> uniq in pipe feeding dir_filter
# Changes v0.2
# 1. If used on livecd legacy boot or equivalent, can differentiate between isolated empty clobber dir
# that also occurs on base system and one that is non-base (ie added by user or other extension).
# This is noted in the results for information. However, isolated empty dirs are infrequent in
# extensions.
# 2. Help and message related to (1) added.
# 3. Added checks for work dir
# 4. Minor changes to output.
#==========================// USER SETTING//===============================
# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl
#==========================// FNS //========================================
help(){
cat <<"EOF1"
declobber v0.2.1 - removes clobber directories from damnsmalllinux extensions
by wdef
Usage: declobber <ext1.dsl ext2.dsl ..>
Run as root to preserve all permissions.
Creates a work dir in $DIR
Declobbered extension is output in $DIR
Default DIR=/home/dsl
EOF1
exit
}
check_work_dir(){
# Check work directory looks ok
if [ -z "$DIR" ]; then "${RED}User setting work dir is unset.${NORMAL}"; exit 1; fi
case ${DIR} in
/mnt/*) D=$(echo ${DIR} | awk -F/ '{print "/" $2 "/" $3 }' )
FS=$(grep ${D} /proc/mounts | awk '{print $3}')
if [ -z "$FS" ]; then echo "${RED}$D not mounted.${NORMAL}"; exit 1; fi
case $FS in
ext*|*eiser*);;
*fat*|*dos*) echo "${RED}$DIR on $FS can't preserve perms - use linux filesystem.${NORMAL}"; exit 1;;
ntfs) echo "${RED}Writing to ntfs is not a good idea ${NORMAL}"; exit 1;;
esac;;
/*|/ramdisk/*);;
*) echo "${RED}Err .. where exactly _is_ this $DIR anyway?"${NORMAL}; exit 1;;
esac
if [ ! -d "${DIR}" ] || [ ! -w "${DIR}" ]; then
echo "${RED}$DIR not a dir or not writeable.${NORMAL}"; exit 1
fi
}
not_legacy_live_msg(){
cat <<"EOF2"
This is not a legacy livecd boot.
If your extension contains isolated empty directories that are also found
on the system, declobber cannot be certain if these are clobberring base
system directories or directories added by user/other extensions, or both.
I will note any such dirs in the results.
EOF2
}
checkon_system(){
# Remove empty dirs if already on system
# Make note in results of empty dirs that appear to clobber non-base dirs (eg added by other extensions)
# or if declobber can't tell.
if $LEGCD; then
if [ -e "/${LM}" ]; then
if [ ! -e "/KNOPPIX/${LM}" ]; then
echo ${LM} >>${clobber_list}
echo "Removing empty dir /${LM} - clobbers non-base added directory" >>${results}
else
# this *is* a system clobberring dir !
echo ${LM} >>${clobber_list}
fi
else
echo ${LM} # OK
fi
else
if [ -e "/${LM}" ]; then
echo "Removing empty dir /${LM} - clobbers either base or added directory" >>${results}
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
fi
}
dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
checkon_system
else
# this *is* a self and/or system clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}
#=========================// MAIN //=======================================
. /etc/init.d/dsl-functions # for ANSI colors
if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi
check_work_dir
if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
LEGCD=false;
not_legacy_live_msg
else
LEGCD=true;
fi
if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi
results=/tmp/declobber_results
rm -f ${results} # preclean
CURRENTD="${PWD}"
APP=""
for APP in $*; do
echo ".........................................................................."
echo >>${results}
# If no path supplied, assume current working dir
if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi
if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi
if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi
NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list
if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi
mkdir ${WORK}
echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1
cd ${WORK}
touch "${clobber_list}" # zero count initialised
tar -zxpvf ${APP}
echo -n "${WHITE}Working ..${NORMAL} "
tar -ztf ${APP} | sort | uniq | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!
NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi
# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi
echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"
CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "See ${clobber_list}" >>${results}
cd ${CURRENTD}
done
echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"
exit 0 |
Posted by roberts on Feb. 25 2007,22:51
WDef, very impressive work.
One thing I would like to suggest is to add getopts so that one does not need to edit the script to specify output directory for the results.
Here is my suggestion:
| Code Sample |
DIR=/home/dsl
|
Be replaced with:
| Code Sample |
DIR=/home/dsl
while getopts ":o:" opt; do
case $opt in
o)
DIR=$OPTARG
shift $(($OPTIND - ))
;;
\?) echo 'Usage: declobber.sg [ -o directory ] ex1.dsl ext2.dsl ...'
exit 1
esac
done
|
This way the script can be used as
declobber.sh -o /tmp myext.dsl
or even
declobber.sh -o/tmp myext.sh
while still maintaining compatibility without the -o option.
Posted by WDef on Feb. 25 2007,23:08
Thx Robert - good suggestion, easier configurability! Will add and repost.
Posted by jpeters on April 04 2007,07:09
I tried this script on gnucash.dsl, and mydsl-loaded the resulting file (gnucash_noclobber.dsl). Next I attempted to run the exec file cd-ing to the created /gnucash/usr/bin folder and typing "sudo ./gnucash" into xterm. Loading the app failed with the following error:
dsl@box:/ramdisk/home/dsl/gnucash/usr/bin$ sudo ./gnucash ERROR: Could not find slib/require.scm in ("/usr/share/guile" "/usr/share/gnucash/guile-modules" "/usr/share/gnucash/scm" "" "/usr/share/guile/site" "/usr/share/guile/1.6" "/usr/share/guile" ".")
If I mydsl-load the un-declobbered version, the above required files get loaded, and now the app will run. Maybe I'm doing this incorrectly.
EDIT: In comparing the created /usr/share/guile/1.6 files with the KNOPPIX/usr/share/guile/1.6 files, "slibcat" file is missing.
Posted by mikshaw on April 04 2007,14:13
You're running the program "installed" under your home directory rather than under /usr. If the program contains libs or other files that it expects to find in /usr, the program won't work.
Apparently the new package got "/home/dsl/gnucash" included. From first glance, it looks like the $LIST_OUT is incorrectly including the work directory in the filenames....maybe not, but the work directory is getting in there somehow.
Posted by jpeters on April 04 2007,14:52
Running the app from /Knoppix/usr/bin produces the same results.
Posted by WDef on April 06 2007,18:49
Sorry for the delay getting to this, I've been busy.
declobber produces this error:
| Code Sample | tar: usr/share/guile/1.6/ice-9/and-let*.scm usr/share/guile/1.6/ice-9/and-let-star.scm: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors
Error. Removed dirs and difference between in/out unequal, exiting ..
|
It seems some programmer actually thought it was a good idea to put one of the shell's special characters (wildcard *) into a filename:
/usr/share/guile/1.6/ice-9/and-let*.scm
It look to me like tar may be gagging on this filename when it appears in the list for archiving.
I have no idea what this file ('module') is for, but its comment header states that it is deprecated (no wonder) and that usr/share/guile/1.6/ice-9/and-let-star.scm should be used instead.
I suggest rebuilding the extension without this file. Try running declobber on that.
|