Creating a Compressed Extension from Source

From DSL Wiki

   This page in other languages: Español

Contents

[edit]

Introduction

Before getting started, lets consider some reasons why we might choose to go with a *.tar.gz (or 'green') extension rather than the standard *.dsl way. First, there is the issue of memory usage. Installation of a *.dsl extension causes the execution of the script 'mkwriteable', which copies a larger part of the filesystem into the ramdisk to allow writing to these files. This of course uses more RAM. Opening up the additional files also opens up the system to potential damage. When a program or library can be overwritten, it can possibly cause problems. A *.tar.gz extension does not execute mkwriteable, and so base system libs and programs cannot be modified. These extensions should write only to /opt, /tmp, and /home (/etc is possible, but not recommended due to lack of ram space).

On the other hand, there are reasons for not choosing this method. Many programs are built in such a way that they have to modify certain files or write to certain directories which are not open to *.tar.gz. Or perhaps they don't need to be installed into these directories, but require some changes to these files in order to run properly. There is also the fact that a lot of people don't know how or just don't want to have to compile source, or maybe they don't have the tools to do so. It can be easier to gather the files installed though a package system and put them in the same places in a myDSL package.

Building a green extension can sometimes be as simple as packing up a pre-compiled binary with the appropriate ownership. More often, though, it's a bit more difficult. Since pre-compiled programs usually look for their support files in the directories to which they were installed (/usr/share, /usr/lib, etc.), it's sometimes better just to start from the source and compile it specifically to run in /opt. This procedure also allows you to make DSL-friendly changes to the program which might not be so simple after the application has been compiled.

Here is one method...Building From Source. This document assumes a program which can be installed through the use of a typical "configure; make; make install" method. There are other ways to install programs from source, and it would take a book to explain them all. At this time the majority of Linux applications use this method. Another assumption is that you have access to a computer system which has a development environment suitable for compiling Linux applications. As an example, we'll be building a green extension from the ImageMagick source.

[edit]

Read The Documentation

First step is to do a little reading. Grab the source archive of your program (imagemagick.org), extract the archive, and make sure to read the README file and any files dealing with license and copyright. Linux programs are not all open source, and doesn't always mean you can redistribute the package. GPL programs are a pretty safe bet that you can modify and redistribute. Next to read is the INSTALL file, which sometimes contains just generic text, but often gives you some useful information about the specifics of the installation procedure, and finally, just before getting to work, run the configure script (if one exists) with the '--help' option. This will list all of the options you can send to configure so you might not have to go back and modify the Makefiles by hand. You may want to redirect the output of this command to a file, which you can have in sight while testing various configure options: "./configure --help > conf-help.txt". As with anything else, the more you compile, the more confortable you will become with it, and the more you will understand what you can do to build a better extension. One configure option which will almost always be used in a green extension is '--prefix=', which sets the install location of the application.


[edit]

Configure

Having done your research, feel free now to execute configure. In the directory containing this script, do "./configure --prefix=/opt/imagemagick" and it should begin checking your system for necessary components. If it spits up an error about not having a compiler, you don't have the needed development environment with which to compile applications. Unfortunately this how-to is not equipped to explain all the necessary steps to putting together a workable development system.

Assuming you have at least a basic development system set up, the script will likely give you feedback on your system, and eventually either end successfully or cough up an error telling you what you are missing. If you have successfully run configure, the script will have created the makefiles which make uses to compile the program. If you get errors concerning a component which you believe is not vital to the program, you might try disabling that component and run configure again. The '--help' option of configure should tell you which features can be disabled this way (e.g. ./configure --prefix=/opt/imagemagick --disable-x). Otherwise you may need to install some support files. Again, this is beyond the scope of this document, and you really should consider learning a bit about compiling software in Linux.

NOTE: One useful bit of knowlege I've just recently (finally) come to understand is the use of variables while configuring the build. For example, I've been including libpng.so.3 with some of my DSL packages because I couldn't get programs to compile against earlier versions, for compatability with libpng in DSL. I finally installed libpng 1.0.12 into /opt on my dev system, and now when I compile for DSL i use the command CPPFLAGS=-I/opt/png-1.0.12/include LDFLAGS=-L/opt/png-1.0.12/lib ./configure --blah --blah and it uses the 1.0.12 version of libpng instead of the one which is generally found first in my dev system.

[edit]

Compile

So let's just say configure went smoothly. You now can do 'make', and if that succeeds you can do 'make install' as root, installing the app into /opt/imagemagick. If you happen to have successfully compiled this program within a DSL system, you can probably launch the program without trouble. However, if the application was built on another system, you may have some serious testing ahead of you. If this is the case, copy the /opt/imagemagick directory into DSL and first see if DSL has the support files needed to run the application. You can do this by typing "ldd /opt/imagemagick/bin/display". This will list the libs required by the program, and tell you whether those libs are found in DSL. If you notice some "not found" messages concerning libraries which were installed into the program's lib directory, you may need to tell ldd that they exist: LD_LIBRARY_PATH=/opt/imagemagick/lib ldd /opt/imagemagick/bin/display The ldd program should now at least find any libs which were installed with the program. If you see additional "not found" messages, you'll now need to find those libs and copy them over to DSL. The most convenient place to put them would be in the same lib directory with the program's own libs. If the program does not have any libs, you can create an /opt/imagemagick/lib directory and place the additional libraries in there. At runtime you will most likely need to set the LD_LIBRARY_PATH to tell the program where to find the libs. Fortunately Imagemagick builds and links its own libs, so any extra libs can be dropped into its lib directory without needing to change the LD_LIBRARY_PATH. In the case of ImageMagick there are a couple of libs required which are not part of the base DSL, so they will need to be copied from your development system into /opt/imagemagick/lib.

It is possible that your program requires a huge number of libraries, or very large libraries. In this case you may want to consider whether it might be a good idea to try to whittle off some fat. Think about what is necessary in a "Damn Small" system and what can be considered extra bulk. If some of these libraries are not terribly important and can be disabled through the configure script, it may not be a bad idea to go back and recompile the application without those libs.


[edit]

Test the Application

When you have all of the necessary libs copied into a directory in DSL where they can be found, you're ready to test the program itself. This testing should be done in an unmodified DSL system, such as one booted from liveCD without any restoration or extra software installed. From a terminal window (so you can see the output of the program), type the command to launch the program, including the path since /opt/imagemagick/bin is not in dsl's $PATH. Any error messages? Hopefully not. If so, try the LD_LIBRARY_PATH command mentioned in the last section to see what is missing in DSL. If you have all the necessary libs in a place where the program can find them, chances are your program will load. If not, you may have problems beyond the scope of this document, and may want to consider looking for help at damnsmalllinux.org or #damnsmalllinux irc channel on freenode.

If the application loads up fine, things are looking good. Please do not assume that this means everything is ready to go, however. There is always the possibility that a program will eventually call another program which is not required merely to start your program. It's always a good idea to thoroughly test your application's features and functions before assuming that it is complete. Open some files, save some files, edit files within your program, if that's what the program does. Whatever the program's use is, it should be tested with as many of its functions as possible.

In the spirit of keeping it small, you might also consider stripping your binary file(s), if you have the strip program handy. This will remove comments and debugging objects from the program, reducing its size. After doing this you should test more, just in case something went wrong.


[edit]

Build the File Structure

Now, does your program work? Good. Now we can package it up as a DSL extension.

First we decide whether or not to include an icon. If the program is a non-interactive console application there is not much reason for an icon. Most of the time, however, it will be an interactive program which may be launched from the desktop. Ultimately it comes down to whether or not you want to bother adding the icon, and whether you have a suitable image for it. Let's assume that you want an icon, just for the sake of this tutorial. The icon file must be in a format readable by xtdesk (png, gif, jpg, and xpm can be used...probably some other formats as well), and is most conveniently placed in the /home/dsl/.xtdesktop directory. The only other file necessary is the *.lnk file which Xtdesk uses to place and name the icon on the desktop. The lnk file is a simple text file with a lnk extension (something.lnk). The format of the file is as follows: 

        table Icon
        Type: Program
        Caption: ImageMagick (the name which will appear next to the icon)
        Command: /opt/imagemagick/bin/display (command to launch the program, in this case it is the imagemagick gui)
        Icon: /home/dsl/.xtdesktop/imagemagick.xpm (path to the icon file)
        X: 420 (X position)
        Y: 384 (Y position)
        end

There are other options available for the file, but this is suitable for a program launcher. If you needed to point the application to some libraries at runtime then you can add that path to the 'Command' line: LD_LIBRARY_PATH=/opt/iamgemagick/lib /opt/imagemagick/bin/display This sets the LD_LIBRARY_PATH variable specifically for this one application. Another option is creating a wrapper, as mentioned before. The wrapper simply contains a command or two to set up the environment, after which it launches the real program. Remember, ImageMagick has it's own linked libraries, so this should not be necessary this time.

Next we need to copy all the installed files into a directory structure with which to create a tarball. Since the application was installed entirely within a single directory(+ $HOME files where applicable) there's no need to hunt for the files. Become root and create a directory somewhere out of the way, in $HOME for example. Let's assume the work directory will be named '/home/dsl/work'. 

        sudo su
        mkdir -p /home/dsl/work/opt # the opt part is just to save a step in the future.
        cd /home/dsl/work/


[edit]

Important

Pay close attention to the paths you use. Everything done from this point is done from within /home/dsl/work/. A command done to './opt' or 'opt' is NOT the same as a command done to '/opt'. '/opt' is a system directory, and 'opt' or './opt' is the opt directory within /home/dsl/work.

If you have files going into /home/dsl you'll need to make this dir as well: 

        mkdir -p ./home/dsl/.xtdesktop # xtdesktop is of course for your icon, if any.

Now copy the installed files. 

        cp -Pr /opt/imagemagick ./opt/
        cp -Pr /home/dsl/whatever ./home/dsl/ #only if there are other files needed in /home/dsl

If you want to have a menu item it will be placed in tmp/mydsl.menu/. The decision to create a menu item is much the same as the decision to use an icon. If the application is non-interactive and generally requires various commandline options, a menu item probably won't be useful. Again, let's assume a menu item will be used: 

        mkdir -p tmp/mydsl.menu
        touch tmp/mydsl.menu/imagemagick

tmp/mydsl.menu/imagemagick is the name of your menu item, used by mydsl-load. While at the time of this writing the menu item in a tar.gz extension does not need to have the same name as the package, it is recommended simply to be consistent with uci extensions (and because things may change in the future which require it). The contents of the menu item are identical to the lines within the fluxbox menu file: 

        [exec] (display name) {command}

Note: .dsl extensions do have to have the same name as its corresponding menu item for it to be valid.

If you need to specify a library path it can be done in the same way as in the icon file. There may be trouble with the menu item being used along with other DSL extensions, where two menu items end up on the same line in the menu and only the first one is displayed. This can be avoided by adding a line break after the menu line in tmp/mydsl.menu/imagemagick

Assuming all files are copied into their proper directory structure, it's now time to check ownerships of the files. Since root created a home/dsl dir, we know that at least this will need to be chowned to dsl. A quick way to do this is to simply give root ownership to everything and then change the ownership of the few dsl-owned files afterward. /opt and everything within it is owned by root.root (0.0) /home is owned by root.root (0.0) /home/dsl and everything in it is owned by dsl.staff (1001.50) /var and everything in it, except for the menu file itself, is root.root (0.0) This should give proper ownership of all files: 

        chown -R 0.0 ./{opt/,tmp/}
        chown -R 1001.50 ./home/dsl/
        chown 1001.50 ./tmp/mydsl.menu/something

We use numeric ownership in this example in the event that the system on which the extension is being built does not have user 'dsl' and group 'staff'.

[edit]

Create the File List

Next we need to create a file list of everything which will go into the package. 

        find . > files.txt

This uses the find command to create a file list of everything in the current directory and below. Now we need to make some edits in order to remove lines which are either unnecessary, or could possibly cause trouble with ownerships. Open files.txt in a text editor and start removing some selective lines. Depending on how many files are included, this may be a tedious process. The file will look something like this (numbers are added here for reference): 

       1 .
       2 ./files.txt
       3 ./home
       4 ./home/dsl
       5 ./home/dsl/.xtdesktop
       6 ./home/dsl/.xtdesktop/imagemagick.xpm
       7 ./home/dsl/.xtdesktop/imagemagick.lnk
       8 ./tmp
       9 ./tmp/mydsl.menu
       10 ./tmp/mydsl.menu/imagemagick
       11 ./opt
       12 ./opt/imagemagick
       13 ./opt/imagemagick/bin
       14 ./opt/imagemagick/bin/display
       15 ./opt/imagemagick/man
       16 ./opt/imagemagick/man/man1
       17 ./opt/imagemagick/man/man1/display.1

1. You absolutely should remove this. Installation will be performed from the root directory, which already exists. You do not want to overwrite this directory, particularly if you made a mistake with permissions. Delete this line.

2. This is the file list...not wanted inside the archive. Delete.

3-5. Same as 1 - any directory which already exists is not needed.

6&7. These ones you want to keep.

8. Same as 1

9. This will be created automatically with 10, so it's not necessary

10. Keep this one.

11-13. Delete.

14. Keep.

15-17. Delete. Man pages can be found in hundreds of places online.

(Note: you should also remove any blank lines, as these make tar produce confusing error messages.)


Keeping your extension "damn small" is the name of the game. The same can be applied to other help documents, includes, and anything else which isn't vital to running the program. Testing the app without certain parts available will help determine what is not terribly important.

So what you would end up with in this case is 4 lines, 6,7,10, and 14. Of course this is only a simple example, and is not complete even for the imagemagick package, but the method can be applied to larger lists.

[edit]

Alternative Way to Create the File List

If your program contains lots of files and directories, going through removing the directories gets tedious. An easier solution is to do: 

        find . ! -type d > files.txt

NB. For this to work, you need to have installed the gnu-utils.dsl package or equivalent.
This adds to files.txt only things that aren't directories (the ! negates). However you still have to go through files.txt as a sanity check, and to remove 'files.txt' itself.

Be aware that that this method will (correctly) add files and symlinks to the list. It will also add any oddities like sockets or FIFOs, in the unlikely event that they're present. But any empty directories won't be included; so if you actually want one of these, you'll need to add it by hand.

[edit]

Create the Archive

       tar cvf imagemagick.tar --no-recursion --numeric-owner -T files.txt
       gzip -9 imagemagick.tar

Now you're ready to test the final product. Hopefully it installs properly with the MyDSL system, including any menu items and icons you may have used. If it fails to run properly, or if other programs stop working after the installation, it's likely that there was a problem with ownerships of the files in the archive. You'll need to go back a few steps and carefully check your files.

If this is an extension for the official DSL repository, be sure to create an info file (check some of the others for reference) and md5sum: 

       md5sum imagemagick.tar.gz > imagemagick.tar.gz.md5.txt

Submitted by Mikshaw

[edit]

A Nice, But Not Necessary, Suggestion:

While you're at it, make two tarballs ("tar.gz" files): One containing executables and one containing all the text files. You can keep the text tarball for your own use and send it to others who ask for documentation.

Retrieved from "http://damnsmalllinux.org/wiki/index.php/Creating_a_Compressed_Extension_from_Source"