squashFSForum: DSL Ideas and Suggestions Topic: squashFS started by: petergunn Posted by petergunn on Dec. 19 2004,03:55
Noticed an interesting article bencmarking squashFS on Kerneltrap...< http://www.kerneltrap.org > performance numbers are here... < http://kerneltrap.org/files/PERFORMANCE.README.txt > The interesting bit mentions DSL... "2. Damn Small Linux liveCD performance tests ext3 uncompressed size 126 MB CRAMFS compressed size 52.19 MB Squashfs2.0 compressed size 46.52 MB Squashfs2.1 compressed size 46.52 MB" Looks like it might give us an extra meg or two and be faster all round as well. :-) Posted by dvazquez on Dec. 20 2004,18:17
cool ,maybe can be implemented in dsl More space, more options for dsl. sorry for my english Posted by ico2 on Dec. 22 2004,12:40
coooooooooooooooooooooooooooool, what more can i say. we can have scite back, chatzilla, fireftp and tons of other really really coool stuff.
Posted by ico2 on Dec. 22 2004,12:46
although the drivers might take up space.
Posted by Fordi on Jan. 02 2005,04:01
using Knoppix 3.4's .config file and the linux-2.4.27 sources, along with the squashfs2.1 patch, I'm currently building a squashfs.o for testing on DSL. If all goes well, we'll be able to use squashfs.On the other hand, I thisnk it might be better to keep KNOPPIX as a cloop. I'll explain later. Meanwhile, why I want squashfs: The primary obstacle to having LOTS of applications in a MyDSL CD is ram. Especially on older machines, if you've got less than 128M, you're pretty much up the crick. One could repackage the key applications as cloop files, but then you've got a system limitation of seven bits of software (/dev/cloop1-/dev/cloop7. /dev/cloop0 (or simply /dev/cloop) is occupied by KNOPPIX). But with squashfs, you are NOT limited to the number of cloop devices. I don't know what the maximum limit is, but I'll be trying to find out. Meanwhile, with squashfs, it's possible to make MyDSL occupy almost NO EXTRA RAM PER PACKAGE. I'll explain: I don't know if you're quite getting me here, so I'll put it together: Create a squashfs file containing, say, gaim.dsl (expanded into position, of course). The /home dir would be packaged in user.tar and placed in the root of our squashfs. This is the "new-format" package I'm hoping for. The user downloads that file. The system runs mkwritable (4M overhead). In my vision, it would run mkwritable upon boot. The system then mounts it in /opt/gaim, and symlinks its contents, omitting user.tar (no .gz, as the file is already compressed and the gz headers would increase the total image size) to their analogues in the filesystem. user.tar is expanded to your /home dir. You now have gaim. gaim is not occupying ANY RAM (well, some for user data, but not much). Install a few more sqfs-type packages, still very little extra RAM use. I'll have more info on this once I've experimented more. Meanwhile, I'm supposed to be rewriting mydslmkr as a local perl/shell app (maybe with a bit of C thrown in there too if serenary). I'll post updates as the occur. Posted by cbagger01 on Jan. 02 2005,05:41
Sounds like an idea worth exploring, for sure.Do you know if squashfs images can be located on an ntfs partition? Posted by John on Jan. 02 2005,08:14
Fordi, let us know how your experiments go -- combining squashFS extensions with a cloop iso sounds very interesting.
Posted by Fordi on Jan. 03 2005,14:53
Ok, here's the news:squashfs.o works with an unmodified liveCD system. I'm in the midst of debugging a pair of scripts that will be packaged with "squashfs.dsl" (along with mksquashfs and squashfs.o) that will automatically load converted .dsl packages as "sdm" packages (Squashfs DSL module). dsl->sdm conversion occurs as follows: -the dsl is extracted to a temp dir. -home, opt, var and tmp are bound within user.tar (.gz isn't necessary; the filesystem gets compressed anyway) these folders are then removed -an empty "sdmrc" is created. (for later package-scripting support) the temp dir is made into a squashfs. sdm installation is performed as follows: -if squashfs.o isn't loaded, load it -if /etc/init.d/mkwriteable hasn't been run, run it -the sdm is mounted in /opt/packname -the user.tar is extracted into the filesystem ( tar -mUkxvf user.tar -C / ) -the mounted sdm is symlinked, file-by-file, into the filesystem -sdmrc is executed (If you think this is a security issue, think what would happen if you had a dsl file that replaced your .xinitrc. Trust me, you can root any machine with a well-designed package.) -code grabbed from mydsl-load does the work in putting together the MyDSL menu and refreshing xtdesktop. (this is what's being debugged) If you have the squashfs.dsl module on a liveCD, it inserts a script into the init-process that scans the CD for sdm packages much like the way DSL does for its original packages. This is currently being debugged, as is the MyDSL menu code. The rest appears to be working fine (it's all being tested with three dsl-converts: gift, gaim and the Gimp 2.0.) Posted by Fordi on Jan. 03 2005,14:56
Well, squashfs is inherently read-only, so I don't see why not. As long as you have ntfs.o and squashfs.o (as well as any special hardware drivers you need to access the ntfs drive, ie scsi drivers or usb drivers), you could do it. You're thinking maybe a method of install using grubw32 on an NTFS partition? Posted by Fordi on Jan. 03 2005,15:02
I never mentioned why I suggest to keep DSL's core image as cloop!1) I poked around in linuxrc once already. That's quite enough. 2) It'd be real cool to make squashfs packages an upgrade that doesn't require getting a new iso of DSL 3) QEMU supports cloop images. All the work I did making the split live/embedded mods would have been for nought. (this is a cop out and can be worked around - just not right now, damnit) 4) It's easier to distribute betas of a 128k package than a 50M liveCD. Yes. It can be done. Yes, it probably should be done. I'd just rather not work on it right now. Maybe once I'm done with building a working squashfs.dsl and rewriting the MyDSL maker for local use. Posted by cbagger01 on Jan. 03 2005,18:15
Fordi,Seems like you have a lot of irons in the fire, all things considered. The reason why I ask is because I believe that cloop files cannot be read from an ntfs partition for some reason and if squashfs files CAN be read from an NTFS partition then it makes it possible to run a DSL poorman's install from a C:\KNOPPIX on a Win2000/XP computer that has no FAT partitons. It also makes *.uci type extensions mountable when they are stored on an NTFS partition. Posted by Fordi on Jan. 03 2005,19:25
I did not know that cloops couldn't be loaded from ntfs partitions.At what point in boot are we talking here? (it's a simple maneuver to toss ntfs.o in the minirt and toss an insmod for it just above the cloop.o load. It would have to be done similarly for squashfs.o as well) Posted by Fordi on Jan. 03 2005,19:25
Funny thing. I no longer have a WinXP machine with which to test that.
Posted by AwPhuch on Jan. 03 2005,22:53
This is a bad thing for development... Go beat up your little brother and steal his! Brian AwPhuch Posted by cbagger01 on Jan. 03 2005,23:31
I think that this is true but I don't remember if I every tested cloop & ntfs together. I'll have to try it out to confirm.I definately remember reading something to this effect, like this: < http://www.knoppix.net/forum/viewtopic.php?p=9035&highlight= > but things may have changed since then. I do know that you can boot the KNOPPIX compressed image from an NTFS partition, but not as C:\KNOPPIX Instead, you save the entire *.iso file to your ntfs partition. Then, upon bootup, Knoppix will do a NTFS -> ISO file system -> cloop mount which somehow works even though a NTFS -> direct cloop mount does not work. Hmm... Posted by The DSL bootprocess on Jan. 04 2005,04:22
hehe, that silly bugger, all confusing people. Using init's the problem.Init, as well as the entirety of inittab is INSIDE the KNOPPIX file. ntfs.o has to be loaded by the linuxrc inside minirt24.gz before it starts searching for places from which to mount the KNOPPIX file. Here's how I've figured the bootprocess: Bootloader kernel loads and mounts the initrd (minirt24.gz), attempts to exec /bin/init /bin/init is either an alias for, or a script that loads linuxrc linuxrc does a lot, but in essence, it finds the root partition, mounts it, and passes control to init, switching to runlevel S When changing runlevels, init executes what commands are found in the inittab. In the case of DSL (and debian in general), it executes that which is contained in /rc{runlevel}.d Init starts by switching to runlevel S, so the contents of /etc/rcS.d are executed in order (K00-K99, then S00-S99). For both Knoppix and DSL, this is all the hotplug stuff you see on boot. For DSL, it's also where the myDSL stuff is loaded. Then init switches to the runlevel defined by the "initdefault", in the case of DSL, 5. This is where services specific to our GUI are loaded (sound, printer, etc). You can also place stuff to run before the gui starts here. See below before you go mad-dashing with scite or beaver. You'll see in inittab that the next thing is to run bash -login. This executes .bashrc and .bash_profile from ${HOME} (in the case of the DSL LiveCD, this is /home/dsl). .bash_profile starts X (if you're not connecting via SSH). startx, among other things, checks to see if there's an xconfiguration, if not, calls xsetup.sh. After that, it starts the xserver and executes ~/.xinitrc boom. That's how DSL boots. Now you know where to place anything based on where it should be in that process. Posted by Fordi on Jan. 04 2005,04:44
Actually, I just read the first reply to that post. I'm going to make an NTFS partition with another computer and get back to you.
Posted by Fordi on Jan. 04 2005,05:35
Ugh. You know, after all that work, doing things that aren't supposed to work, you know what I got hung up on?user.tar was creating an ownership problem when extracted from runlevel 5. A simple sudo and it was fixed. The only way I knew was X wouldn't start. Took me two HOURS to figure that out. I hate that shit. Anyway, I'm nearly done. Right now I'm getting script execution right, then I gotta work on the sdm dynamic loader script. Then I can release the product. Posted by noclobber on Jan. 04 2005,17:37
A dumb question:Can the number of cloop devices in DSL be increased? Posted by roberts on Jan. 04 2005,18:33
Of course, I started with 4 devices. Because at the beginning there were not many ci extensions. When I introduced the uci, I added 4 more. It is a caution approach that we take. We wanted to see performance running 8 cloops in liveCD environment. I will now bump the number up again. I would imagine that it is limited by the device number size like the other devices are, and of course, performance.
Posted by AwPhuch on Jan. 04 2005,21:58
So that would be a static setting, and not a dynamic one? Kinda like IP_CONNTRACK for iptables Brian AwPhuch |