.dsl file editing utilities

Forum: myDSL Extensions (deprecated)
Topic: .dsl file editing utilities
started by: clivesay

Posted by clivesay on Oct. 04 2004,00:17

I like to spend my time finding ways to automate redundant tasks.

If you like to edit and repackage .dsl/.tar.gz files like I do, you probably have Ke4nt's howto doc printed out and laying in a high visibility area of your desk.

I love using emelfm and created a few shortcut buttons to speed up this task. I thought I would share with everyone. You can add these as buttons or, as I did, add them to the User Commands section of the emelfm menu.

Name: Unpack .dsl
Command: tar -zxvf %f

Name: View .dsl Contents
Command: tar -ztf %f > dirlist.txt

Name: Repack .dsl
Command: tar -T dirlist.txt --no-recursion --numeric-owner -cvf- | gzip > %{}

Thanks to clacker for educating me on how to create the last command. It will prompt you to enter the name of your new tar.gz/.dsl.

Have fun!

Chris

Posted by ke4nt1 on Oct. 04 2004,02:36

Nice...

Thanks and 73 ..
ke4nt

Posted by ke4nt1 on Nov. 18 2004,03:21

bookmark

Posted by ico2 on Nov. 18 2004,20:59

naaaaaah, i still prefer deb2dsl.

but good job though

Posted by clivesay on Nov. 19 2004,00:50

ico2 -

You can use these commands to repackage your .dsl after you personalize it with an icon or whatever. :;):

Chris

Posted by ico2 on Nov. 20 2004,23:13

ooooooooooooooh, i might consider that for dsl-making.

Posted by clacker on Nov. 29 2004,23:06

Here are two tricks I found. I don't know how "button friendly" they are, but I thought I'd toss them here since they really are dsl files editing utilities.

The first looks for files in a dsl package that already exist on your system. I thought this would help avoid overwriting files:

Code Sample

#!/bin/bash
# finds files in an archive that already exist on the system
# you can use the file "deleters" to remove repeats

tar -ztf $1 > files
cat files | while read i
do
if [ -e "/$i" ]
then
echo $i >> deleters
fi
done

The second is for when you want to make a dsl that uses ke4nt1's gtk.dsl package. That helps keep your dsl size down when you use gtk2.0 or a later version of perl:

Code Sample

#!/bin/bash
tar -ztf $1 > files_new
tar -ztf gtk.dsl > files_gtk
cat files_new | while read i
do
grep $i files_gtk >> deleters
done

That technique should also work with the tcl dsl in the same way.

Then unzip your archive and use tar to remove the duplicates from your package:

Code Sample

mv somenewdsl.dsl somenewdsl.tar.gz
gunzip somenewdsl.tar.gz
tar -vf somenewdsl.tar --delete -T deleters
gzip somenewdsl.tar
mv somenewdsl.tar.gz somenewdsl.dsl

Posted by ke4nt1 on Nov. 30 2004,03:14

sweet...

I have used similar tactics from the command line.
This will help with upgrading the gtk2.dsl as well when it becomes neccessary.

tnx fer sharing..

73
ke4nt

Posted by SaidinUnleashed on Nov. 30 2004,04:11

GTK 2.6 comes out soon does't it?

/me shudders because he is sure it will be fatter than it already is...

I wish more apps would use < FLTK >.

-J.P.

Posted by clacker on Nov. 30 2004,14:24

I can see now that the code I wrote in the first section isn't finding duplicates, at least not all of them. I think I needed -s instead of -e, but that code doesn't work either. I also added quotes and a / around the $i in the first code section so that it would look for files from the root directory.

Posted by ico2 on Nov. 30 2004,17:30

we should definately improve dsls package management. i can produce something much better if you ppl want?

Posted by ico2 on Nov. 30 2004,17:32

i can change it so that it will support the deleting of dsls without killing deps, and also make it prevent accidental downgrading of dependancies. can someone in dsl now post a copy of the mydsl-load script so i can have a look (cannot be bothered to reboot).

Posted by jls legalize on Dec. 01 2004,00:46

is dsl open-source?

Posted by ico2 on Dec. 01 2004,09:30

yep

Posted by clivesay on Feb. 10 2005,15:46

clacker -

Did you ever come up with a working update to your code?

Chris

Posted by clacker on Feb. 11 2005,23:41

I didn't. I think what is needed is to search for existing files (-e), links (-s), and probably other things as well. Then you would need to sort them and remove duplicates. It's at the very edge of my bash skills, and I didn't spend any more time on it lately. It looks like a good idea, but I can't take it to compleation.

Echoing what SaidinUnleased said, I wish there was a neat flua program that could unpack, look for duplicates, make menu items, make .xtdesktop entries (starting xpaint with a icon template), and repackage dsl files. Can we take your emelfm ideas and transfer them?

Posted by clivesay on Feb. 11 2005,23:48

Quote (clacker @ Feb. 11 2005,17:41)

Echoing what SaidinUnleased said, I wish there was a neat flua program that could unpack, look for duplicates, make menu items, make .xtdesktop entries (starting xpaint with a icon template), and repackage dsl files. Can we take your emelfm ideas and transfer them?

I never thought of that. It would be a great little program to use for us hopelessly addicted .dsl types. I would love to do some more gtk2 type apps but picking through them is tedious and I hate sending ke4nt a jumbled mess. I'm sure he would find something like this handy also. Knowing him, though, he probably has a handy little utility of his own under tight security!

Chris

Posted by softgun on Feb. 20 2005,07:33

Quote (clivesay @ Oct. 03 2004,20:17)

I like to spend my time finding ways to automate redundant tasks.

If you like to edit and repackage .dsl/.tar.gz files like I do, you probably have Ke4nt's howto doc printed out and laying in a high visibility area of your desk.

Please tell me where this dsl howto written by Ke4nt is?

I am trying a dsl which needs to have a userid and password added to it during apt-get. Therefore the deb2dsl script will not work.

How can I save the running program with the added userid, database name etc as a dsl and not make one from the basic debs as deb2dsl does?

I will also try to unpack and repack after changes are made. What are the changes we are allowed to make using the methods you suggest.

Thanks

Posted by clivesay on Feb. 20 2005,13:46

Softgun -

You would have to do a forum search to find it. There has been so much posted about extensions I don't even know what the best keywords would be to find it without retrieving a bunch of threads. It may be in the Documentation. I haven't looked. I will share my personal cheatsheet with you(and the world).

Quote

Create a work directory such as /home/knoppix/WORK/

Untar the file: tar –zxvf <filename>.tar.gz

*********************************************

Make your changes and tar the file:

tar cvfz <filename>.tar.gz *

*********************************************

Eliminate unnecessary directories from the tar.gz

Output tar.gz contents: tar –ztf <filename>.tar.gz > <output filename>.txt

Edit the text file by removing directories that already exist in the filesystem such as /usr/bin

*********************************************

Repackage tar.gz file:

tar –T <output filename>.txt –no-recursion –numeric-owner –cvf- | gzip > <filename>.tar.gz

********************************************

Identifying changes in the file structure

Before changing the filesystem

find / -print > b4.txt

After the change

find / -print > aft.txt

To view the changes

diff b4.txt aft.txt > change.txt

You have two options from what I know.

1. Package the debs up and create a script to unpack and install them when you run the .dsl and enter the information your program requires. You can see the alsadebs.dsl and aliendebs.dsl packages as an example.

2. Take a snapshot of your filesystem before you install the package, install the package, and then take a snapshot afterwards and output the difference between the two. Then you would package up the debs with deb2dsl and then you would have to go into the filesystem and pick out each changed file in the filesystem and add or replace existing files in your .dsl. Taking a snapshot of your system can be done with the last entries of my cheatsheet. This option is not for the faint of heart unless you are just looking for a challenge.

You can search the forum for a lot of great info but the info above should get you started.

Good Luck

I can't believe I am describing this stuff!! Just a year ago I was a clueless idiot, now I have graduated to just idiot!

Chris

Posted by softgun on Feb. 20 2005,16:49

Thanks Chris,

I am not sure i got it :-(

I hope to graduate to idiot stage by the end of this year. Wish me luck!!

I may come back with problems in a day or too when I try out your snapshot stuff.

Click! Click!

:-)

Posted by softgun on Mar. 03 2005,08:15

Quote (clivesay @ Oct. 03 2004,20:17)

I like to spend my time finding ways to automate redundant tasks.

After unpacking nd viewing the file how do i exactly change things (using what tool) before repackaging it? Suppose i want to add a whole directory with subdirectories to the dsl befor repacking it?

Thanks for any clarification/help

Posted by reidar on Mar. 04 2005,08:55

Today is a joyful day for me! I finally decided to take a dive plunge into the world of .dsl-creation, based on the cheatsheet presented by Chris in this thread. The result is that I am now the proud creator of my own textmaker.dsl. (Since I have bought a licence of textmaker, I cannot share this, but I wanted to find out if I could create a .dsl from apps that I use daily and that DSL doesn't have, and Textmaker was a good starting point...)

I installed the program in the home directory, created a WORK/ directory there and put the entire textmaker/ folder there. Created an opt/ folder inside of this /home/dsl/WORK directory and moved the entire textmaker/ folder into that one. I also made a tmp/mydsl.menu folder where I put a file that I thought would give me a mydsl-menu entry (based on the investigation of a couple of other dsls). I also created a /home/dsl/.softmaker folder inside of WORK (textmaker creates such a folder, and it contains the registation-info etc.). In the end I deleted unneccesary language files from the textmaker directory, did a tar cvfz textmaker.dsl *, and I was done. Everything worked like a charm, except for one little thing: there wasn't created a mydsl-meny entry for Textmaker. Anyone knows why? I put a textmaker file inside of tmp/mydsl.menu that looked like this:

[exec] (Textmaker) {/opt/textmaker/tml}

But it didn't create a meny entry....

Everything else worked out fine though, and I am the proud owner of the textmaker.dsl file, which makes a killer wordprocessor on my old machine!

-r

Posted by clivesay on Mar. 04 2005,13:06

Quote (reidar @ Mar. 04 2005,02:55)

Congrats!

As far as your menu, you may have done something I have done myself. In the menu file, you have to enter the path as you have done and then hit the enter key to return to the next line before saving.

Also, although your .dsl my be working, I suspect it isn't 'clean' since you used the cvfz command only. Ke4nt had a great primer on .dsl creation in the forum. I am not sure exactly where. It was quite awhile back.

He taught us an important lesson you want to make sure you understand. It involved avoiding changing permissions on existing directories.

Run the tar ztf command to create a text file of the contents of your .dsl file. Open the file and look inside. You want to eliminate any lines that contain existing directory paths in the base filesystem.

/usr/bin/ <-----Bad
/usr/bin/textmaker <-----Good

/usr/bin already exists in the filesystem so no need to have it in your .dsl. However, /usr/bin/textmaker is new to the FS so you want to keep it.

After you have removed the offending lines, use the tar -T command to repackage your .dsl.

Now you have a nice clean file ready to submit to the community if you like!

Chris

Posted by reidar on Mar. 04 2005,14:07

Thanks a lot Chris! Your tip about my menu-problem was excellent. I did what you said (pressed Enter), went over the process of creating the .dsl again, tar -ztf, removed "dangerous" entries like tmp/, opt/ etc., used your repacking-script in emelfm and got a new clean (I believe) and perfectly workable textmaker.dsl. I also did the same for planmaker, so now I have them both. I would like to share them, but I am not sure if that is allowed, since Textmaker and Planmaker are commercial products for which I have paid? Do you know? If it is ok I will submit them of course! planmaker.dsl is 3.85 megs and textmaker.dsl is 8.13 megs. (The latter includes Norwegian dictionary, so I suppose it could be made somewhat smaller.) They are both excellent in speed and features in my opinion, especially on older machines, like the ones I use :-)

Again, this is a happy day, and it has become even more happy!!!

-r

Posted by clivesay on Mar. 04 2005,14:35

I imagine those apps are not free to share. I just meant for any other apps you may put together. I have made that same menu error before and needed ke4nt to whack me upside the head to get me to see what I had done wrong.

It's great to have another extension builder amongst us.

Chris

Posted by WoofyDugfock on Mar. 11 2005,07:44

Pls ignore - posted in error

Posted by clacker on Mar. 16 2005,15:55

Heres what I came up with for finding duplicates between files like a new dsl and gtk2.dsl or tcltk.dsl:

Code Sample

#!/bin/bash
#
# find duplicates between two dsl files
#
tar -tzf $1 > files1
tar -tzf $2 > files2
cat files1 | while read i
do
grep $i files2
done

I think the problems I was having before was due to an additional ./ placed in front of the lines of my files by the old deb2dsl script. This worked for me. It could stand some error checking like are there two command line arguments and do they end in dsl or tar.gz, but I couldn't make that work. My bash chops ain't up to snuff.

I also wrote one to find duplicates between the dsl file and the system. What's nice here is if you unpack the dsl to a temporary directory and run this script, you not only see the file names, but a list of details for the existing file and the file in the dsl. The differences pop out (different sizes, same name but linked to a different file)

Code Sample

#!/bin/bash
#
# find duplicates between two dsl files
#
rm -f sysdups
tar -tzf $1 > files
cat files | while read i
do
if [ -e /$i ]
then
echo $i >> sysdups
fi
done

cat sysdups | while read i
do
echo
ls -l /$i
if [ -e $i ]; then ls -l $i; fi
done

Posted by WoofyDugfock on Mar. 22 2005,08:57

1. For finding duplicates I've just been trying out the very good duplifinder.sh (http://mwynwood.com/blog/?page_id=12).

This compares md5 hashes of all files in the argument paths so for example I just found two pairs of identical library files with different names (and one pair were not symlinks) inside the .dsl I'm trying to make (BTW my first .dsl!).

This seems useful given that there are (as I just discovered) libs sharing the same basename in different locations but having very different contents eg stdio.h

To find files that are identical & common to ~/expanded_tarball and eg /usr/local/lib, just ./duplifinder.sh -d ~/expanded_tarball /usr/local/lib

BTW duplifinder.sh appears to need gnu-utils.dsl to be installed.

2. To quickly scan for possible duplicates on the system against an unpacked .dsl's dirlist.txt, I'm trying the following with dirlist.txt as the argument:

#!/bin/bash
# possduplicate.sh
{
echo "The following files have the same basename as those listed in $1:"
while read LN; do
find / -name `basename $LN` -print 2>/dev/null
done <$1
} >possible_duplicates.txt

3. To find libs etc that are not mentioned in the source code (and hence are only ? for configure/make/install or other ancillary use), I've moved the src out of the unpacked tarball (not a .deb obviously) into another directory and deleted the files that look like Makefile.c, deleted all libs & imports themselves from the remaining unpacked tarball directory, removed the path to the binary itself from the WIP .dsl's dirlist.txt, and then ran:

#!/bin/bash
# Identify libs that are only used for configure/make/install.
# Search configure/make/install argument directories $2 for
# instances of basenames of files listed in argument $1 that are not # mentioned in src $3.

{
while read LN; do
ENDNM=`basename $LN`
if grep -r -q $ENDNM $2; then
if ! grep -r -q $ENDNM $3; then
echo "$ENDNM was found in $2 but not $3."
fi
fi
done <$1
} >poss_buildonly_libs.txt

Can then consider deleting the unneeded libs etc from dirlist.txt before packing the .dsl (after all the files are installed where they should be). This last one is probably a bit iffy/naive/risky for all I know.

Dunno if the above is of interest but there it is. Probably old news.