Would it be time for DSL to shed its skin?
Forum: water cooler
Topic: Would it be time for DSL to shed its skin?
started by: curaga
Posted by curaga on July 13 2007,18:11Would it be time to throw away Debian compatibility?
I've watched these threads about non-compatible libs & headers, which are because the extension was made separately from the base.
Also many have wished for updated libs & apps.
A solution for that would be to follow compile-from-scratch philosophy.
If a new base was compiled, it would be up-to-date, small, and work well.
As uClibC has become mature enough to support nearly all apps, it could be used, and that along with optimization for size and 486 for all libs & apps would drop the size of DSL quite much.
Doing something like this would increase the gap between DSL and other distros, like Puppy..
From this new base we could separate a build system, totally compatible with the base without any conflicts.
So, to sum up:
Pros: smaller size -> can fit more into DSL
up to date
fully compatible build system
Cons: it would be a lot of work
extensions would have to be redone
For me dropping Debian compat is a pro, but for others it might be a con.. so I didn't list it in either one.
It's a radical idea, not likely to happen, but tell still what you think.
Posted by lucky13 on July 13 2007,19:22Those are two separate issues: Debian compatibility and whether DSL is in need of overhaul. Both issues are complicated by how different people view DSL.
If it's viewed primarily as a live CD, it doesn't matter what you do or how you do it so long as it works; so nothing needs to be done. If it's frugal or USB or so on, then it needs some minor upgrades to freshen it up enough that newer apps can be compiled for it; it wouldn't require a total overhaul to do that.
The third option is one I think is probably one headed for deprecation because it will eventually require what you suggest, and because it duplicates too many better current alternatives: hard drive install. This isn't DSL's niche, and DSL increasingly pales in comparison to other distros because people want newer versions of apps than are in Woody (or even Sarge -- hence adoption of Ubuntu's repositories instead of Debian's by other distros like Mepis). While MyDSL extensions tend to work pretty well for hard drive installs, that's not how they're intended. And I think more of them will increasingly be in forms (UCI, UNC) not best suited for hard drive installs.
The total overhaul would require far too much work both on the ISO and current extensions. I'm not convinced that newer apps and libraries would be smaller. I'm also pretty sure that in making things tighter with newer (more bloated) libraries, someone would take exception to something being left out that would require recompiling it just to make something with a certain feature. So then we're right back where we started. At least now people can try to grab something from Knoppix 3.4.
So I would say no to a total overhaul and yes to deprecating hard drive installs (even though I have several partitions with DSL installed traditionally) in the very near future. I think Robert's already doing what he can in freshening things up. And maybe this talk should follow the release of 4.0, not precede it.
Posted by humpty on July 13 2007,21:08The HD installs (though i hate them) does serve a purpose for low ram systems.
Did you know ram for old laptops cost twice that of newer ones?
And that's if you're lucky enough to find them!
Having ties with Debian is not necessarily a bad thing.
It means extensions can be easily converted to work on a Debian system.
(I don't know if you can say the same for puppy).
Separating the library from the base system is an interesting idea.
Although I can't think of any benefits and it would probably cause
more confusion with apps having to match the library module.
Unless ofcourse each library upgrade is backwards compatible,
which really defeats the purpose of being small.
A new DSL (like DSL-N) would have to be launched.
Posted by roberts on July 13 2007,22:24Seems to me, that we have already had this discussion.
It is not fair to compare a 2.6 kernel, gtk2 distribution, of any kind, and the applications offered therein, with a 2.4 kernel, gtk1 system like DSL.
If you want/need the latest applications and don't mind the bloat and your computer can handle it, then there are many, last count over 500, linux distributions for you to choose from. Please stop the 2.6 gtk2 comparisions.
Additionally, I tried the DSL-N (2.6/gtk2) route with no Debian and no gtk1. All I got has gripes about not being Debian compatible. Even thought it was never claimed to be. I got much grief because users tried to load gtk1 extensions and of course would not work. I got complaints because no syslinux boot floppy version, and on and on.
Based on these experiences...
I conducted a poll, on the direction of DSL and a newer 2.4 kernel was decided. Realize many things have moved to 2.6 kernel and therefore compiling the latest may remain as issue. I have already run into that with compiling some of the additional non-kernel modules.
Again, it is a matter of what works, what is smallest versus the latest and the inheirt bloat. DSL has always chosen the smaller and what works. DSL supports many smaller memory limited machines.
I realize that I cannot please everyone. With the soon to be released alpha of DSL 4, I have tried to make DSL easier to use with a totally new UI and sporting a 2.4.34 kernel. Many changes throughout. During the alpha cycle, I will likely release with different default configurations. I will do this to showcase the areas that have changed the most.
I am sure there will be some who will hate it and some who will love it. It may take some time to get used to it. And for some, they will see no change at all. It depends on how you run DSL. DSL has always been about offering many choices. And it shall remain so I am not planning on deprecating any of the major functions in DSL.
So goes the life of a developer. Much work. Little thanks; and even less pay
Posted by curaga on July 14 2007,07:13When I said update, I was thinking about minor updates, like latest gtk1 etc, not moving to gtk2 and linux 2.6.
Building with uclibc makes everything smaller, even when dynamically linked.
if there's some app that would require gazillion libs not in core DSL, and the extension would grow because of all those, would there be need for a uClibC compiling environment where extensions like that could be compiled statically??
That way they would be smaller than with glibc dynamic linking and no need for starting wrappers etc..
And the result would work on all versions of DSL etc..
Posted by curaga on July 15 2007,06:57no one has any comments about my last question?
or might it be that this thread isn't shown in "new posts" anymore.. well now it is
Posted by mikshaw on July 15 2007,15:01I don't see much difference between the bloat from including dynamic libs in a package and the bloat from including libs statically? in the executable.
Posted by stupid_idiot on July 16 2007,10:40Being a non-programmer, it sounds like voodoo to me - the thing about uClibc decreasing size of object files.
Let me explain myself more clearly:
We are not going to gain much in size reduction just from the smaller size of the uClibc main libraries as compared to Glibc.
The real size gains, as you have said, is from the smaller size of all code compiled using uClibc functions - hence the code size reduction is multiplied for each app.
The idea sounds very promising, but you are advocating a complete rework of DSL. Not just the base system; I mean, following what you said - all extensions have to be reworked.
But I don't like the idea. I mean, just think: All the memories from DSL, all gone. I know, all the extensions can be cleanly recompiled with uClibc and reposted. But they wouldn't be what they are, any longer. Besides, I don't think I have the right to take another person's extension and recompile it. It's more serious than this - If you do one, you have to do them all; it's like people can't even say no. I know, I could express this in a better way, but I do mean all this seriously.
I don't want to discourage, hamper, or hinder you in any way. In fact, I have fantasized giving DSL the full uClibc treatment myself, in the past. Perhaps we could do this as a shadow project, a side attraction alongside DSL. Something low-key but relevant to the people with this kind of interest (i.e. crypto-obsession) in small size.
I also propose an experiment just to honour the whole idea:
Try compiling a uClibc Firefox using < buildroot > - naturally, using the same buildconfig and the same version (1.0.6) as DSL's Firefox - and see how much the size is reduced.
Lastly, there is one last thing that hasn't been tried yet:
Try auditing all the code in DSL for code size.
I believe the binaries themselves are already compiled with '-Os', so there's no room for improvement with the binaries.
But some libraries I think were pulled straight out of Debian packages. Possibly we could get some marginal size gains there. Quite a small potential reduction - but it's there.
Basically, optimize the Glibc platform before moving on.
I know I am spending an unhealthy amount of time over something that is counter-productive to 'getting things done and changing the world', but.. Oh dear. Can't think of anything.. But! Curaga, assuming there's cooperation, we can get alot of libraries done quickly! That is all I want.
I don't mean to be pushy. I mean, if you don't like it at all, then just say, 'No, it's just too crazy!', or something like that.
Moderators, I apologize for inflaming this mania, this obsession, this abstruse, counter-productive, impractical concern with small size. I know that I've thrown away ettiquette and propriety for the sake of the last little bit of pseudo-elegance that does no one much of any good. My only fear is that DSL will be made into an imprint of my megalomania. I don't want that. Curaga, let's tune this baby up!
Posted by curaga on July 16 2007,13:34Nice one, stupid_idiot!
More in DSL has been drawn from Debian than you think. Maybe around 80% of the base. Plain -Os would gain something, but uClibC would give even more.
Let's. About your firefox idea: I haven't compiled firefox even once yet (I'm an Opera fan). And they mention that the one in DSL was modified to use only gtk1 and "took the pains of reducing it's space requirements as much as possible".. I don't know what they did, so an exact replica would be hard.
Quickie comparison of glibc-2.5 and uclibc-0.9.29 source tarballs: ~15mb and ~2mb.. So even replacing the main libs would have an effect. Also gcc-4 produces smaller code than gcc-3..
But you didn't say anything about the question of a static uclibc build environment for extensions needing a lot of libs not in core DSL, how about it?
Posted by stupid_idiot on July 16 2007,15:53Don't worry - Firefox builds out of the box with gtk1. I've tried this with 1.0.x, 1.5.0.x, and 2.0.0.x. As for what the DSL devs did in 'modifying' Firefox, I don't know exactly what they did - Perhaps these were only cosmetic changes?
Re: Building Firefox
1. Download source for 1.0.6 < here > (http://ftp.osuosl.org). 1.0.6 is not longer on ftp.mozilla.org - 1.5.0.x is the earliest.
2. Read this guide on how to write a '.mozconfig' file (used for configuring Firefox): < 'Configuring Build Options' >
You really should take note of the section 'Building with an Objdir' for the sake of convenience - If you want to discard a bad build, you can just delete the objdir and start over - Or, you can have multiple builds with differently-named objdirs.
3. In DSL's Firefox, go to 'about:buildconfig'.
Do you see the section "Configure arguments?" These should be pasted verbatim into '.mozconfig' using the proper syntax.
For each individual argument, start a new line, like this:
The '.mozconfig' file should be placed in the Firefox source directory.
4. When you are done pasting all the arguments from 'about:buildconfig' into '.mozconfig':
Make sure you are in the Firefox source directory and that your '.mozconfig' file is placed in here as well.
For building Firefox on uClibc, the procedure is the same.
The most straightforward way is probably to build from inside a uClibc environment. You can generate an entire uClibc system using < buildroot > (link to buildroot snapshots; direct SVN access also available - see < here >).
Building the uClibc environment using buildroot is similar to building the Linux kernel.
1. Extract and cd into source.
2. `make menuconfig` (Choose the packages that you want in your uClibc 'distro'/environment).
At the end of it, a root filesystem image is produced (you can choose the filesystem type - ext2, iso9660, cramfs, etc - when doing `make menuconfig`).
Then you should set it up so you have an environment you can `chroot` into.
1. `mount -o loop rootfs.ext2 mnt_dir` and copy the contents onto a spare hard disk partition.
2. Do the same as above, except just copy the contents into a directory somewhere.
3. 'rootfs.ext2' is fully used, but you can create free space:
`e2fsck -f rootfs.ext2` (`resize2fs` will insist that this be done first.)
`resize2fs rootfs.ext2 size_in_[kilo|mega|giga]bytes`
Then, just chroot into the image file itself. Hilarious!
`mount -o loop rootfs.ext2 mnt_dir` (Is it mounted read-write? Please do check!)
Posted by stupid_idiot on July 16 2007,16:20
I would opt for a shared uClibc extension. The size of this extension would (in compressed form) be at most in the ~500K region. This is better than tacking the static uClibc onto every extension, which would swell every extension by a few hundred KB. The only objection I could have would be aesthetic. I mean, two(..?!) parallel libc libraries? The aggravation that people experience at this inelegant solution had better be outweighed by size reductions - which might not be as dramatic as imagined. Conversely, if this works out, it again begs the question - If uClibc is so great, why not use it in core DSL as well? Identity crisis, confusion, etc.
But above all, thank you very much for discussing these things.
Geez, I look at what I posted this morning, and I don't even know what I'm talking about anymore. I was contradicting myself all over the place. It's more interesting to hear what you want to do, instead. Would like to help out in little ways if possible.
Posted by curaga on July 17 2007,09:28Well, what do I want to do?
I'm not too sure about Firefox..
Maybe make a base ala modified LFS. Maybe start a distro. Maybe conquer the world.
I've got barely two weeks of summer holiday left, and I wanted to learn Quenya this summer (haven't even started yet..). So much I want to do and so little time. Gotta prioritize. First, I'm gonna go see Harry Potter and the Order of the Phoenix...
interpret all that as: I'm gonna do something, but not too hasty master Meriadoc..
Why don't you do that Firefox experiment? You seem like a better person for it..
Posted by stupid_idiot on July 17 2007,11:17I'll see what I can do about compiling uClibc Firefox in the next few days.
Oh, and enjoy your school holidays - They seem more precious as you get older.
Speaking as someone in his last year at college.
Posted by stupid_idiot on July 18 2007,09:37It's near 6PM here. I just started the buildroot process. I estimate 4 or 5 hours to completion.
Update: 12:30AM here. Build just completed.
This is the configuration I used for buildroot (I am only listing options used that differ from the defaults):
The above packages can be built automatically. There are more packages which (ideally) should be selected - but have trouble building using buildroot. (At first, they don't compile because libtool has not been built - a build order problem. After I deselect these packages, build everything including libtool, then reselect them, I get strange automake errors.) Therefore, please note that the following packages have to be built manually once we chroot into the new uClibc environment:
For graphics (listed in build order - Xorg depends on font libs; glib and gtk depend on X libs):
3. Xorg (Because we need the libraries and headers. Also, even though DSL uses libraries from XFree86, the libraries and headers in Xorg should be fully compatible.)
4. glib 1.2
5. gtk 1.2
I might run into trouble building Xorg. I have never built Xorg before. There's not much else to worry about.
Posted by florian on July 18 2007,18:07Hi!
Would it be possible to cut the size of the DSL base by giving it the 'uclibc treatement' and at same time create a glibc MyDSL extension that could be used for compatibility with existing legacy glibc-based MyDSL extensions? If so that would seem like the ideal solution.
Just my 2-cents idea of the day!
Posted by curaga on July 18 2007,18:12on the other hand, I'm good at building Xorg. So you can have Firefox and I'll take X..
florian: that would be ideal, but since programs hard-code glibc into /lib/libc.so, which will be uclibc and thus cannot be made a link to glibc, wouldn't work...
Posted by florian on July 18 2007,18:41I still think that if DSL goes with a uclib base, MyDSL should provide glibc.unc for users that needs compatibility with legacy extensions
When compiling extension and base program for "uclib DSL", we should make sure to link to /lib/uclibc.so
The compatibility glibc.unc extension could create /lib/libc.so that the legacy MyDSL programs use.
Posted by curaga on July 18 2007,18:49@florian: you're missing the point. If the link was replaced to another libc, only static and programs compiled for that libc would work. So after using that only the old extensions would work, not even tar or ls or any basic util would be available...
As libc.so is a shared library, programs load it everytime the programs are loaded..
@stupid_idiot: which target triplet and optimizations did you use?
As it's gonna be built from ground up, we should do everything we can to improve performance & size (triplet should be i486-pc-linux-uclibc, best optimization IMO would be "-Os -march=i486 -mtune=i686 -fomit-frame-pointer"
Posted by florian on July 18 2007,21:22@Curaga: I think you missed my point in previous point as I never spoke about a symbolic link in my previous post
My point is that compilation should always explicitely define the uclibc name by uclib.so (can be done in gcc using something like -nodefaultlibs or -nostdlib flag). This is an hassle but this is the only way to ensure backward compatibility! .
This way legacy extensions would access libc.so from the glibc.unc MyDSL compatibility extension, while core apps and newly "properly" compiled extension would still work against uclib.so
Posted by stupid_idiot on July 18 2007,23:15
In previous times, I did use the same flags you suggested.
Sadly, the additional optimized code from '-march' and '-mtune' increases the size of binaries somewhat. So does '-fomit-frame-pointer' - 'libwxgtk1.uci' is 2.0M with fp omitted and 1.8M without. So it led to me cheating; the added convenience doesn't hurt neither - I just use '-Os'.
That is when compiling libraries.
When compiling binaries, I also use
With buildroot, I used the default suggested CFLAGS:
Unfortunately, what buildroot does is prepend these flags to the command line. So it tends to get overridden by the default flags ('-O2 -something -something') of the packages. I am sure there is an easy way, somewhere, around this.
Posted by curaga on July 19 2007,08:44What? omitting frame pointers increases size? Weird, it's supposed to leave them out so size would be smaller..
I think we can have little bigger code for speed, unless it's bigger than with glibc originals of course..
Posted by curaga on July 19 2007,09:23I wondered about your results, so I did a test, built gzip:
flags: binary size:
"-s -Os -fomit-frame-pointer" 45720
"-s -Os" 45880
"-s -Os -march=i486 -mtune=i686 -fomit-frame-pointer" 45720
so omitting frame pointers did decrease size, and optimizing didn't increase size......
Posted by curaga on July 20 2007,07:33Another comparison:
unoptimized wide ncurses library was 333kb with glibc and 284kb with uClibc..
Posted by stupid_idiot on July 20 2007,14:51C compiler: gcc 4.2.0 (self-compiled)
C++ compiler: g++ 3.3.5 (Debian Sarge)
Size taken with `du -b file` after running `sstrip file`.
`sstrip` ("Maximal 'strip'ing utility.") can be found < here > (www.busybox.net).
Bold entries are C++ apps.
'-mcpu' is used for C++ apps because g++-3.3 doesn't understand '-mtune'.
It seems like '-fomit-frame-pointer' hits C++ apps. C apps seem unaffected.
Could this be due to a difference in compiler versions (gcc-4.2-4.2.0 vs g++-3.3-3.3.5)? I will have to compile g++-4.2 first and then find out.
`make` ; `sstrip unzip` ; `du -b unzip`
'-Os -fomit-frame-pointer' 90596
'-Os -march=i486 -mtune=i686' 90596
'-Os -march=i486 -mtune=i686 -fomit-frame-pointer' 90596
`make 7za` ; `sstrip bin/7za` ; `du -b bin/7za`
'-Os -fomit-frame-pointer' 1062377
'-Os -march=i486 -mcpu=i686' 845289
'-Os -march=i486 -mcpu=i686 -fomit-frame-pointer' 1062377
`make` ; `sstrip unrar` ; `du -b unrar`
'-Os -fomit-frame-pointer' 187227
'-Os -march=i486 -mcpu=i686' 154459
'-Os -march=i486 -mcpu=i686 -fomit-frame-pointer' 187227
Posted by curaga on July 20 2007,17:07That might be because of old g++, I don't seem to recall it raising sizes..
when can I have your root image to play with? X etc..
Posted by stupid_idiot on July 20 2007,23:11
ASAP, but I have only too many assignments these few days.
Where could I upload it when it's done? (Gmail is not possible because the file is too large.)
Posted by curaga on July 21 2007,06:30What? Gmail has 2Gb space, doesn't it?
how about making it iso9660/bzipped ext2/cramfs? (whichever is smallest and fits for the purposes, cramfs cannot be used if the size is over 256mb uncompressed etc..)
Posted by curaga on July 21 2007,06:47I can handle zisofs too, and if gmail has a file size limit, just "split" it
Anyone can upload files <100mb to rapidshare, btw..
I think a base done with LFS style would be better, with chance to optimize and mess with everything, but since my three tries so far have failed, let's mess with this buildroot one..
Posted by stupid_idiot on July 21 2007,07:16On top of the buildroot base, I am manually compiling freetype, fontconfig, libid3tag, and libmad. I am not touching Xorg, so glib-1.2 and gtk-1.2 will have to come later.
I will send this modified buildroot image (it is ext2) by Gmail, split into 20M pieces (I will use `lxsplit` - is this okay?).
Thanks for your patience.
1. Busybox `ar` lacks [some?] switches that are present in the GNU version. As a result libiconv could not compile - Busybox `ar` gave an "unknown switch" error. It seems buildroot leaves GNU `ar` out of the image in favour of busybox's `ar`. I did:
`rm mountpoint/usr/bin/ar` (This is a symlink - Remove first, or you will overwrite '/bin/busybox'! No - `cp -r` doesn't work.)
`cp buildroot/build_i386/binutils-2.17-target/binutils/ar mountpoint/usr/bin`
This solved the problem.
It is possible there are more Busybox commands that need to be replaced with the GNU versions.
Update: Busybox `tar` sometimes complains about "Invalid magic" while GNU `tar` works. So I've replaced '/bin/tar' with GNU `tar`.
2. After I compiled the nano editor, it wouldn't run in the chroot environment - "Error opening terminal - rxvt."
Seems like the default ncurses installation doesn't have the rxvt terminfo entries.
So I copied over the terminfo entries for anything rxvt-related from my Debian root:
`cp /usr/share/terminfo/r/rxvt* mountpoint/usr/share/terminfo/r`
`nano` works properly after this.
3. From my own bad experience - If you chroot into the uClibc image, and then close the xterm window without properly using `exit`, further instances of `bash` will become progressively laggier. To make things right again, do `lsof mountpoint` and `kill -9` the bash instance(s).
4. I have a bunch of 'alias bla=blabla' and 'export CXXFLAGS=blabla' and 'export CPPFLAGS=blabla' lines in '/etc/profile' which I use. Please modify or remove them at your convenience.
I just ran into a problem building GNU tar. It has to do with missing wide char functions in uClibc. I will have to rebuild uClibc with wide char support.
The buildroot documentation describes how to reconfigure uClibc < here >.
Posted by curaga on July 21 2007,08:23How big is it?
I think it would be best if you split (using gnu split, so I could combine more easily) to 100mb which would go to Rapidshare and the rest to your Gmail, as Rapidshare has amazing bandwith..
Also please at least gzip it before splitting.. oh and don't email anything to me, my mailbox is 3mb and already half full!
For your errors/stuff, all that would've been avoided if the base was LFS... Not that it matters..
Posted by stupid_idiot on July 21 2007,08:46Alright - It will be on rapidshare by tonight (it's 5PM here); stay tuned.
Restarted buildroot - Adding 'coreutils' and other GNU stuff (gawk, less, nano, patch, sed, etc).
There is still OpenSSL. And OpenSSL needs Perl. Dear Curaga, I am leaving OpenSSL up to you.
It is coming along, but will probably not be ready until tomorrow morning.
I think I will PM you the rapidshare url when it is ready.
1. The buildroot snapshot tarball I am using is 'buildroot-20070721.tar.bz2'.
2. I would like to send you the buildroot configuration ('.config') and uClibc configuration ('uClibc-0.9.29.config'). What is the best way to send them? Thank you.
(The buildroot configuration file '.config' is placed at 'buildroot/.config'.)
(The uClibc configuration file 'uClibc-0.9.29.config' is placed at 'buildroot/toolchain/uClibc/uClibc-0.9.29.config'.)
3. I think I will upload my 'buildroot/dl' directory as a .tar.gz to rapidshare. It will contain a 'dl' directory with all the source tarballs inside. This will save you the download time.
Posted by curaga on July 21 2007,14:01I don't like buildroot that much, I don't think I'd have any use with those configs..
How did you select 2.4.31 headers? when I tried buildroot's menuconfig the only header selectable was 2.6.21..
Hey, no hurry. I just got Harry Potter and the Deathly Hallows, so I couldn't do anything anyway
Build perl and openSSL if you can, or if it only depends on time..
Posted by stupid_idiot on July 21 2007,14:52
I am sorry, but it takes very long on my system - P4 'Willamette' 1.7. Could you add Perl and OpenSSL on your own? (Unless, you are worse off in terms of CPU?) I would build them if I were more free, but it is already midnight here, I still have assignments to complete, and I have to go watch a show tomorrow (parent is taking me to see it), so my weekend is effectively dried up.
Posted by stupid_idiot on July 21 2007,15:5812AM
I am stumped by coreutils.
Sadly, I have no more time within this weekend to finish this work. Curaga - I can't help you anymore with this uClibc work. I am very sorry about this.
Posted by curaga on July 21 2007,17:36wtf? P4 1.7G takes long to build perl and openssl? I have a (p3) celeron 1066Mhz, and perl takes about 9min and openssl about 6min on my comp...
And as I said, I'm gonna be reading HP now, I can't work on this either, so no hurry..
Posted by stupid_idiot on July 21 2007,23:03
What? Then I am gravely mistaken! My apologies.
Posted by curaga on July 22 2007,10:01I've finished reading HP7, so whenever you're ready..
Posted by stupid_idiot on July 23 2007,12:09To anyone reading this:
So as not to keep you in suspense, I have decided to suspend my activities WRT uclibc because I want to keep the focus on my schoolwork. Curaga knows about this. That's just me. Curaga is still interested and may work on uClibc in the future, but I am not going to do the talking for him. I suppose he will post again if he starts on anything interesting. So just look out for any new threads about uClibc. Thanks!