unionfs examples
Forum: DSL Tips and Tricks
Topic: unionfs examples
started by: clacker
Posted by clacker on April 17 2006,02:50
I thought that the unionfs.dsl I made needs a little explanation and a few examples. The reason it needs explanation is because some people really hate unionfs (one person commented it was why they stopped using Knoppix) and I'm not suggesting it gets incorporated into the dsl base. Things work fine the way they are and there's no need to change this drastically now. It would cease to be dsl if you did that. I think unionfs has some great uses besides basing the distro around it. The examples might help people think of even better ways to use it to make things easier.The easiest way to describe unionfs is to say that it combines two or more directories so that they appear to be a single directory. On the surface that might not seam like such a big deal. Say you have a directory called dir_one containing a file called one and a directory called dir_two with a file called two. If you make a third directory called dir_sum , you can run the following command to make the contents of dir_one and dir_two to appear to be in dir_sum:
| Code Sample |
| sudo mount -t unionfs -o dirs=dir_one=ro:dir_two=ro unionfs dir_sum |
Now running the ls command on dir_sum shows one and two. This is typical of the type of example you see and while it shows the basics, it misses some of the finer points.
It is also possible to combine read/write directories with read only ones, the result of which is a directory that appears to contain all of the files in the read only directory, but also allows writes to it. One massivly usefull application of this is in the remastering process on lower memory, no ext2 formatted drive machines. During the remastering process, I typically create a source directory and copy the contents of /KNOPPIX into a source/KNOPPIX directory. This takes up a lot of memory since I remaster to my ramdisk. I found I could save a substantial amount of resources by doing the following:
| Code Sample |
| mkdir KNOPPIXtemp mkdir source mkdir source/KNOPPIX sudo mount -t unionfs -o dirs=KNOPPIXtemp=rw:/KNOPPIX=ro unionfs source/KNOPPIX |
Using this method rather than the normal route of
| Code Sample |
| mkdir source sudo cp -Rp /KNOPPIX source |
I saved 245724 kilobytes according to the free command (the unionfs way used 119140Kb and the normal way used 364876 Kb). Now you can add things to the source/KNOPPIX directory and they get put into the KNOPPIXtemp directory. You can delete files and a .wh file for the deleted file gets created in KNOPPIXtemp that removes the file from the virtual drive. The rest of the remastering process can take place normally, and the new source/KNOPPIX directory behaves the way you would expect.
Another neat example I found was when you want to have a /usr/src/linux-2.4.26 directory but you dont't want to take up all that room in the ramdisk. By using unionfs, you can not only add that directory by combining the files from a CD or from somewhere in KNOPPIX (say you remastered with it) but you can write to the directory as well.
| Code Sample |
| sudo mkdir /usr/src sudo chown dsl /usr/src sudo chgrp staff /usr/src mkdir /usr/src/linux-2.4.26 mkdir srclinuxtemp sudo mount -t unionfs -o dirs=srclinuxtemp=rw:/mnt/cdrom1/linux-2.4.26=ro unionfs /usr/src/linux-2.4.26 |
You can now run everything you need to in the /usr/src/linux-2.4.26 directory to compile drivers and the only thing that gets added to the ramdisk are the changed and created files. So there is a huge space savings in terms of memory.
There are other possibilities, like creating a CD that has your programs and files on it, and the using a temp directory and unionfs to make the files writeable even though they are on a CD. Or patching a CD by using unionfs to combine the CD with a directory of patched files.
Posted by mikshaw on April 17 2006,12:51
hmmmmm....I don't see the usefulness of adding another tool to accomplish the same thing you can already do with cp -s Anything else it does besides share files between directories?
Not saying it's not useful...just trying to figure out if it would be useful to me.
Posted by clivesay on April 17 2006,13:23
This is all a little over my head, I think, but if I understand the first example correctly, would this allow you to run .dsl files as .uci's without using additional ram by making the filesystem writable? It seems you are saying you can take exisitng dirs and basically point them to another location? If that was true, then all .dsl files could be like uci's? I've never quite understood the whole unionfs thing 
Posted by mikshaw on April 17 2006,14:02
Correct me if I'm wrong, but i don't think it would work that way. You create a separate directory, so that directory would not be in PATH, LD_LIBRARY_PATH, etc. The original directory is untouched, so the system would not behave any differently. You would still need a wrapper for many uci-like packages to work.As far as I can tell, this is similar to what mkwriteable does for the base DSL system, in that you can write to these directories but the original files are untouched.
Posted by clacker on April 17 2006,19:42
mikshaw, the real use is to start with a directory that isn't writeable and make it writeable, but it also makes the files that were in the read only directory writeable. If you used the cp -s command, yes, you would wind up with a writeable directory populated with links to the files in the read only directory. But you wouldn't be able to change or write to those files without deleting the link and replacing it yourself with the real file. With unionfs, if you needed to overwrite a something.conf file that was in the original directory, it would do it without an error. Your way would say that the file is write only.So sure, the ln -s or cp -s way works, but only if none of those files need to change. Take the example of the /usr/src/linux-2.4.26 directory. If I had a .config file in there when I started, I wouldn't be able to write over it after running make menuconfig. Using unionfs you could. For the remastering and creating the source/KNOPPIX directory I gave, your way doesn't work because the mkisofs command creates an iso of the links and not the files as well as not allowing file modifications.
clivesay, yes you can make uci files writeable using unionfs. This is similar to what slax does in their distro. You would mount the something.uci file (really a compressed file system created with mkisofs and create_compressed_fs) to /opt/something_ro rather than to /opt/something like you normally would, create /opt/something and /opt/something_rw directories, and use unionfs to merge them into a writeable directory in /opt/something. You could do:
mount -t unionfs -o dirs=/opt/something_rw=rw:/opt/something_ro=ro unionfs /opt/something
Now the /opt/something directory is writeable, as are the files it already contained. Any changed files get added to the /opt/something_rw directory.
Posted by JB4x4 on April 17 2006,23:04
I was messing with unionfs, and came up with something. First I created XFree86.uci, it mounts as /opt/XFree86/usr. Then I ran...mount -t unionfs -o dirs=/usr=ro:/opt/XFree86/usr=ro unionfs /usr
What I have ended up with is a working XFree86 without having to make /usr writeable. This saves alot of memory, but I was wondering if there are there any adverse effects. (I am running a frugal install.) The only thing I noticed is that I had to add "umount unionfs" to powerdown.sh to umount /usr so that XFree86.uci properly un-mounts.
Posted by clivesay on April 17 2006,23:43
That's EXACTLY what I was wondering about Thanks for experimenting.
Posted by clacker on April 17 2006,23:57
That totally blew my mind. I didn't know that a union could contain the directory it points to. You might want to make the /usr directory rw (read write) instead of ro (read only).I think /usr became writeable when you loaded the unionfs.dsl. Loading dsl files runs mkwriteable which makes that directory and others writeable, but their contents are still links into /KNOPPIX. The fact that you could modify the files that were from your uci's usr directory is really nice, though.
Just out of cuiousity, did you run free at all to check the amounts of memory at various points in the process compared to the traditional way? Like loading XFree86.dsl and running free, then restarting and running it with your uci and running free again to check the difference?
Like clivesay said, thanks for experimenting.
EDIT:
I don't think that works. It's a nice idea but I don't believe the union can contain its own contents. I also found out it's really easy to mess things up when your changing the /usr directory and where it's link points.
Posted by JB4x4 on April 18 2006,02:57
Clacker, I actually didn't tell you the whole story, I took your .dsl extension, and remade it into a .uci extension 
. It mounts in opt and loads the module with "insmod", leaving /usr read-only. I didn't check free on the old setup (XFree86.dsl), but I know it was using 30-50 meg of swap as soon as X started. Running frugal now, after just loading unionfs.uci and xfree86.uci, free shows Mem: 78164 total, 61984 used, 16180 free - Swap 156200 total, 0 used 156200 free.I have even managed to shave about 30 seconds off of the boot time from the old debian styled install I ran before trying frugal, and probably close to a min when I first started using frugal and having to load XFree86.dsl at boot.
As far as I can tell when looking through /usr, it seems unionfs made a seemless integration between /usr and /opt/XFree86/usr (the mount of XFree86.uci). I even tried going to the console running "umount unionfs", "startx", and there were errors, "No X server found". Re-run the "mount -t unionfs ........" command, and everything works fine again.
Posted by mikshaw on April 18 2006,03:07
That's pretty awesome
Posted by roberts on April 18 2006,03:17
I agree.With 64 cloops we can sure have some interesting UCI's
Nice work guys!
Posted by hswerdfe on April 26 2006,18:01
WOW, I have just read this uinionfs thread and I think it is very cool what could be done with it.so. would it be then possible to mount "/" read write?
see this post for why I would want to do this
< post >
I could maybe then do other cool stuff
like
like mount part of my USB stick as "/"
| Code Sample |
mount -rw -t vfat /dev/sda1 /mnt/sda1 mount -t unionfs -o dirs=/=ro: /mnt/sda1/dsl/=rw unionfs / |
install stuff
| Code Sample |
apt-get update apt-get install emacs |
Reboot the computer
and remount "/" as above
then run emacs?
like a normal app from the command line...
I see the possibility of a way to install apps without remastering?
and without using .dsl or .uci
so like ALL of debains 40,000 some odd apps are avalible to install, in an easy way...
WARNING : I haven't tried any of the above, I havn't even used the unionfs.dsl yet, and I was not even really aware of unionfs till a little while ago.
I just got all excited about what it could mean?
so..am I way out to lunch, or is something like this possible?
what about problems like ..permissions on vfat? and files that have the same name space?
Posted by spock on April 29 2006,16:55
For how to use unionfs this page is interesting :< http://www.unionfs.org >
Inspired by this thread I've made a mydsl.uci containing XFree86.dsl, gtk2-0705.dsl, gkrellm-gtk2-2.2.2.dsl and all my fonts. This extension is merged with /usr and /lib in 2 unions and gives me a gtk2 version of DSL with XFree that boots faster and uses way less ram and swap than with dsl extensions.
With unionctl I can also load / unload my former dsl extensions easily converted into uci.
Thanks a lot for making unionfs available in DSL.
Posted by roberts on April 29 2006,17:31
Sounds great spock!Any chance to have seperate XFree86, Gtk2 uci versions for our repository? Maybe I need to make a new extension definition of say .uuci. Which would indicate that the new extension requires unionfs? With 64 cloops I would like to see a move in this direction.
Posted by clivesay on April 29 2006,17:36
| Quote (roberts @ April 29 2006,12:31) |
| Maybe I need to make a new extension definition of say .uuci. Which would indicate that the new extension requires unionfs? |
Maybe a mkmydsl type script where people but their .dsl files in a directory and it builds a .uuci file containing all the extensions for a customized DSL?
Looks like there are lots of possibilities.
Posted by roberts on April 29 2006,20:11
I am going to look into this as it relates to UCI type extensions and how much integration into core is possible given the size limitation.I know I could make it work like dsl-n. But size constraints will prevent that level.
Posted by clacker on April 30 2006,22:15
I tried making a version of mkwriteable for unionfs but I ran into a few snags. Most of the directories were easy to handle: I created two new directories in /ramdisk for each one, the first using the base name (for the new mount point) and the second with "_holder" added to the name (for the writeable area). Then I used the /KNOPPIX/whatever directory in the unions.for example, for the bin directory I might do:
| Code Sample |
| mkdir /ramdisk/bin mkdir /ramdisk/bin_holder mount -t unionfs -o dirs=/ramdisk/bin_holder=rw:/KNOPPIX/bin=ro unionfs /ramdisk/bin ln -sf /ramdisk/bin / |
Then I tried to streamline this for all of the directories I needed. The /etc and /var directories were harder, and require some care at shutdown. For /etc I had to make a copy of the existing /etc directory and copy the /etc/mtab into it. For whatever reason I wasn't able to overwrite the link from /var -> /ramdisk/var. I don't know why that happened either but I did work around it.
| Code Sample |
| #!/bin/bash #first handle the easy directories for i in bin boot lib sbin usr do mkdir /ramdisk/$i mkdir /ramdisk/${i}_holder mount -t unionfs -o dirs=/ramdisk/${i}_holder=rw:/KNOPPIX/$i=ro unionfs /ramdisk/$i ln -sf /ramdisk/$i / done # Then take care of /etc by making a copy of it mkdir /ramdisk/etc_holder mkdir /ramdisk/etc cp -rp /etc /ramdisk/etc_copy mount -t unionfs -o dirs=/ramdisk/etc_holder=rw:/ramdisk/etc_copy=ro unionfs /ramdisk/etc # copy mtab or their may be problems since there is another entry in the current one cp /etc/mtab /ramdisk/etc/mtab rm -rf /etc ln -sf /ramdisk/etc / # to undo the above section you need the following: # rm /etc # cp -r /ramdisk/etc_copy /etc # umount /ramdisk/etc # now handle the /var directory mkdir /ramdisk/var mkdir /ramdisk/var_holder mv /ramdisk/var /ramdisk/var_old mount -t unionfs -o dirs=/ramdisk/var_holder=rw:/KNOPPIX/var=ro:/ramdisk/var_old=ro unionfs /ramdisk/var # to undo the above section you need the following: # umount /ramdisk/var # rm -r /ramdisk/var # mv /ramdisk/var_old /ramdisk/var |
It's a lot like mkwriteable, using links to unionfs mounted directories in the /ramdisk directory. Once you have the /ramdisk/bin mounted, you can use unionctl to meld more /opt/whatever/bin directories with it.
Is there a file that gets run at shutdown, before anything is unmounted automatically? If so I could add the lines I nned to unmount the /ramdisk/etc and /ramdisk/var direstories. The other directories work fine.
I didn't think it was as easy as mounting / as a union as read/write because /ramdisk is under / and there would be a nasty loop with the writeable part of the mount then.
Posted by roberts on April 30 2006,23:12
Given the fact that DSL targers low(er) ram, old(er) machines.My interest is more in the area of easily deploying uci type extenions. Using unionfs to avoid the library and binary pathing issues. Also trying to achieve that mkwriteable is not necessary. That only mountpoints are used in the union. This would be a watershed for our target base of less capable machines. For example, I just made an abiword.uuci.
I could certainly make dsl like dsl-n and have the entire fs writeable via a boot option in linuxrc. But then it would be easily overrun on trying to install every conceivable type of extension or package. I am not sure of proceeding in that direction.
Of course anyone can always write code to do what they want. Just wanted to share my thoughts and direction in this area.
Posted by roberts on April 30 2006,23:18
| Quote |
| Is there a file that gets run at shutdown, before anything is unmounted automatically? If so I could add the lines I nned to unmount the /ramdisk/etc and /ramdisk/var direstories. The other directories work fine. |
There are two levels of such a file.
/usr/local/bin/exitcheck is when just before X is stopped.
This will allow X type messages and prompts to still function before shutdown of X
/opt/powerdown.sh is the system wide power down script.
This is where I clean up dynamic icons and menus,unmount uci's etc.
Posted by meo on May 18 2006,22:41
Hello clacker!Very nice work! Thanks for explaining some of the use of unionfs.dsl as well. It might come in handy for me since I'm making remasters quite frequently. Thanks again!
Have fun all you guys,
regards meo