Search Members Help

» Welcome Guest
[ Log In :: Register ]

Mini-ITX Boards Sale, Fanless BareBones Mini-ITX, Bootable 1G DSL USBs, 533MHz Fanless PC <-- SALE $200 each!
Get The Official Damn Small Linux Book. DSL Market , Great VPS hosting provided by Tektonic
Pages: (3) </ [1] 2 3 >/

[ Track this topic :: Email this topic :: Print this topic ]

reply to topic new topic new poll
Topic: declobber  - a tool for extension rebuilding, Remove nasty system clobberers< Next Oldest | Next Newest >
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Feb. 03 2007,21:54 QUOTE

Robert has complained that people are sending the occasional badly-made .dsl extension to the repo containing one or more system-clobbering directories.  (An unsubtle hypothetical example might be eg /usr/local/bin, which would wipe out your entire /usr/local/bin tree when the extension is installed).

I suspect these often result from leaving out the --no-recursion tar option when repacking an extension, or from not appropriately trimming a file list for feeding to tar with -T.

Anyway ... I've been using this tool for some time to strip clobber dirs from extensions (similarities to dsl2unc).

Try running it on some of the larger .dsl extensions in the repo - you might get a surprise!

Code Sample

#!/bin/bash

# declobber v0.1  by WDef
# Removes clobber dirs from .dsl extensions

# Just do ./declobber <extension1.dsl extension2.dsl ..>
# Output is extension1_noclobber.dsl etc in the WORK dir

# For livecd use with legacy boot.


#==========================// USER SETTING//===============================

# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl

#==========================// FNS //========================================

dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
# empty dir - check not already on base system
if [ -e "/KNOPPIX/${LM}" ]; then
# this is a clobberring dir !
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
else
# this is a clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}


#=========================// MAIN //=======================================


. /etc/init.d/dsl-functions # for ANSI colors


if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
echo "${RED}declobber is for livecd use with legacy boot option only.${NORMAL}"
exit 1
fi

if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi


results=/tmp/declobber_results


rm -f ${results} # preclean

CURRENTD="${PWD}"
APP=""


for APP in $*; do

# If no path supplied, assume current working dir

if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi

mkdir ${WORK}
touch "${clobber_list}" # zero count initialised

echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1 # comment this out if you're in a hurry

cd ${WORK}
tar -zxpvf $APP
echo -n "${WHITE}Working ..${NORMAL} "
tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!

NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi

# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi

echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"

CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "( These are listed in $clobber_list )" >>${results}
cd ${CURRENTD}
done


echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"

exit 0
Back to top
Profile PM 
roberts Offline





Group: Members
Posts: 4983
Joined: Oct. 2003
Posted: Feb. 03 2007,23:04 QUOTE

Thanks WDef,

I will add this to the tools section in v3.3RC1
Back to top
Profile PM WEB 
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Feb. 06 2007,12:43 QUOTE

It's occurred to me that the requirement to run only as livecd legacy boot is probably overkill and only really matters when the extension contains an empty dir that might be on the system.  Also, if other extensions happen to be installed on the system, well, we don't want to clobber those either.

So I might post another version giving this as a warning only when it is required.
Back to top
Profile PM 
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Feb. 08 2007,00:51 QUOTE

Unsightly bug:

Code Sample
tar -ztf $APP | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &


Should read:

Code Sample
find -name \* | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &


Apologies.  Cleaned-up version coming.

EDIT: Can't reproduce the alleged-four-in-the-morning bug. Ignoring.
Back to top
Profile PM 
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Feb. 09 2007,15:26 QUOTE

Code Sample

#!/bin/bash

# declobber v0.2 by  wdef  Feb 9 2007
# Removes clobber dirs from .dsl extensions


# Changes  v0.2
# 1. If used on livecd legacy boot or equivalent, can differentiate between isolated empty clobber dir
# that also occurs on base system and one that is non-base (ie added by user or other extension).
# This is noted in the results for information.  However, isolated empty dirs are infrequent in
# extensions.
# 2. Help and message related to (1) added.
# 3. Added checks for work dir
# 4. Minor changes to output.


#==========================// USER SETTING//===============================

# Put DIR on a hard drive linux partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl

#==========================// FNS //========================================

help(){
cat <<"EOF1"
declobber v0.2 - removes clobber directories from damnsmalllinux extensions
by wdef
Usage: declobber <ext1.dsl ext2.dsl ..>
Run as root to preserve all permissions.
Creates a work dir in $DIR
Declobbered extension is output in $DIR
Default DIR=/home/dsl
EOF1
exit
}

check_work_dir(){
# Check work directory looks ok
if [ -z "$DIR" ]; then "${RED}User setting work dir is unset.${NORMAL}"; exit 1; fi
case ${DIR} in
/mnt/*) D=$(echo ${DIR} | awk -F/ '{print "/" $2 "/" $3 }' )
FS=$(grep ${D} /proc/mounts | awk '{print $3}')
if [ -z "$FS" ]; then echo "${RED}$D not mounted.${NORMAL}"; exit 1; fi
case $FS in
ext*|*eiser*);;
*fat*|*dos*) echo "${RED}$DIR on $FS can't preserve perms - use linux filesystem.${NORMAL}"; exit 1;;
ntfs) echo "${RED}Writing to ntfs is not a good idea ${NORMAL}"; exit 1;;
esac;;
/*|/ramdisk/*);;
*) echo "${RED}Err .. where exactly _is_ this $DIR anyway?"${NORMAL}; exit 1;;
esac
if [ ! -d "${DIR}" ] || [ ! -w "${DIR}" ]; then
echo "${RED}$DIR not a dir or not writeable.${NORMAL}"; exit 1
fi
}

not_legacy_live_msg(){
cat <<"EOF2"

This is not a legacy livecd boot.
If your extension contains isolated empty directories that are also found
on the system, declobber cannot be certain if these are clobberring base
system directories or directories added by user/other extensions, or both.
I will note any such dirs in the results.
EOF2
}


checkon_system(){
# Remove empty dirs if already on system
# Make note in results of empty dirs that appear to clobber non-base dirs (eg added by other extensions)
# or if declobber can't tell.

if $LEGCD; then
if [ -e "/${LM}" ]; then
if [ ! -e "/KNOPPIX/${LM}" ]; then
echo ${LM} >>${clobber_list}
echo "Removing empty dir /${LM} - clobbers non-base added directory" >>${results}
else
# this *is* a system clobberring dir !
echo ${LM} >>${clobber_list}
fi
else
echo ${LM} # OK
fi
else
if [ -e "/${LM}" ]; then
echo "Removing empty dir /${LM} - clobbers either base or added directory" >>${results}
echo ${LM} >>${clobber_list}
else
echo ${LM} # OK
fi
fi
}



dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/${LM}" ]; then
if [ $(ls -1A ${WORK}/${LM}| wc -l) -eq 0 ]; then
checkon_system
else
# this *is* a self and/or system clobberring dir !
echo "${LM}" >>${clobber_list}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}


#=========================// MAIN //=======================================


. /etc/init.d/dsl-functions # for ANSI colors


if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi

check_work_dir


if [ -e /etc/sysconfig/unionfs -o ! -e /KNOPPIX/bin/ash ]; then
LEGCD=false;
not_legacy_live_msg
else
LEGCD=true;
fi


if [ $EUID -ne 0 ]; then
while true; do
echo "${WHITE}You need to be root to conserve all permissions."
echo -n "Quit? (y/n)${NORMAL} "
read
case $REPLY in
y|y*|Y|Y*) exit 1;;
n|n*|N|N*) break;;
*) echo "Invalid response";;
esac
done
fi


results=/tmp/declobber_results


rm -f ${results} # preclean

CURRENTD="${PWD}"
APP=""


for APP in $*; do

echo ".........................................................................."
echo >>${results}
# If no path supplied, assume current working dir

if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME
LIST_IN=${WORK}/list_in
LIST_OUT=${WORK}/list_out
clobber_list=${WORK}/clobber_list

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi

mkdir ${WORK}


echo
echo "${WHITE}Removing any nasty clobbering dirs from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo
sleep 1

cd ${WORK}
touch "${clobber_list}" # zero count initialised
tar -zxpvf ${APP}
echo -n "${WHITE}Working ..${NORMAL} "

tar -ztf ${APP} | sort | uniq -u | tee ${LIST_IN} | dir_filter >${LIST_OUT} &
rotdash $!

NEWNAME=${NAME}_noclobber.dsl
tar -T ${LIST_OUT} --no-recursion --numeric-owner -cpf- | gzip -9 >"${NEWNAME}" &
rotdash $!
if [ ! -s "${NEWNAME}" ]; then echo "${RED}Problem making $NAME_noclobber.dsl" >>${results}; continue; fi

# Debug - check that clobbers removed = difference between in and out
DIFF=$(cat ${LIST_IN} ${LIST_OUT} | sort | uniq -u | cat - ${clobber_list} 2>/dev/null | sort | uniq -u | wc -l)
if [ $DIFF -ne 0 ]; then echo "Error. Removed dirs and difference between in/out unequal, exiting .."; exit 1; fi

echo "${GREEN}Now test your new ${YELLOW}${NEWNAME}!${NORMAL}"

CDRS="$(cat ${clobber_list} 2>/dev/null |wc -l)"
CLOBBERS=${CDRS##*[ ]}
echo "Removed ${YELLOW}$CLOBBERS${NORMAL} clobber dir(s) from ${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}${NEWNAME}${NORMAL}" >>${results}
echo "See ${clobber_list}" >>${results}
cd ${CURRENTD}
done


echo
echo "=========================================================================="
echo
echo "${WHITE}DECLOBBER ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "=========================================================================="
echo "${GREEN}Finished.${NORMAL}"

exit 0
Back to top
Profile PM 
11 replies since Feb. 03 2007,21:54 < Next Oldest | Next Newest >

[ Track this topic :: Email this topic :: Print this topic ]

Pages: (3) </ [1] 2 3 >/
reply to topic new topic new poll
Quick Reply: declobber  - a tool for extension rebuilding

Do you wish to enable your signature for this post?
Do you wish to enable emoticons for this post?
Track this topic
View All Emoticons
View iB Code