complex dsl vs uciForum: myDSL Extensions (deprecated) Topic: complex dsl vs uci started by: softgun Posted by softgun on Feb. 11 2005,11:41
Please tell me how to setup a dsl which has to be "started" to work. I had this problem wih the postgresql dsl I made, where I have to start the postgresql postmaster to get it to work. When I try, the system crashes.Is there a limit to the size in bites to a dsl? Is a dsl of 10MB too big? Do large apps have to be in uci format? Any size limit which defines which mode to go for? dsl tar.gz or uci? Posted by ke4nt1 on Feb. 11 2005,13:43
There are many .dsl and .tar.gz files that are way over 10MB's in the repository.The only limiting factor is the amount of ramspace you have available to run them. When extensions get to be upwards of +400MB's in size, it's best to try to build them into a .uci format, just to save ramspace, unless your box is loaded with a couple o' GB's of ramsticks, and you just don't care. 73 ke4nt Posted by softgun on Feb. 13 2005,14:30
Thank you! 400MB?!!! Amazing!!!! You are not messing with zeros are you? :-) Posted by ke4nt1 on Feb. 13 2005,16:49
No, tis true....Here is my .uci dir list ... root@box:/mnt/hdd3/uci# ls -al drwxr-xr-x 2 root root 4096 Jan 29 11:48 . drw-r--r-- 10 root root 4096 Jan 29 11:48 .. -rw-r--r-- 1 root root 96719405 Jan 27 01:48 RailroadTycoon2.uci -rw-r--r-- 1 root root 270220360 Jan 27 01:35 enemy-territory.uci -rw-r--r-- 1 root root 8653633 Jan 27 01:45 firefox.uci -rw-r--r-- 1 root root 181299036 Jan 27 01:54 heretic2.uci -rw-r--r-- 1 root root 322690545 Jan 29 04:07 heroes3.uci -rw-r--r-- 1 root root 107102388 Jan 27 02:05 hhb2003.uci -rw-r--r-- 1 root root 31260092 Jan 27 01:45 jre1_5_0.uci -rw-r--r-- 1 root root 388663002 Jan 27 01:55 myth2.uci -rw-r--r-- 1 root root 69797802 Jan 27 01:45 openoffice.uci -rw-r--r-- 1 root root 6050668 Jan 27 01:45 opera.uci -rw-r--r-- 1 root root 11952260 Jan 27 01:45 python.uci -rw-r--r-- 1 root root 43355622 Jan 27 01:45 qcad.uci -rw-r--r-- 1 root root 453881837 Jan 27 02:09 quake3.uci -rw-r--r-- 1 root root 103191741 Jan 27 02:05 ut2003.uci -rw-r--r-- 1 root root 300231559 Jan 27 02:05 ut2004.uci 73 ke4nt Posted by softgun on Feb. 14 2005,08:14
I believe!What this means in other words, is that I can have a dsl upto 400MB in size........... This gives me mouth watering anticipation of things that could be :-) Ah! But yet i cannot work out how to get my postgresql dsl to start :-( The problem with the deb2dsl script is that it makes a "/static"/ package from downloaded debs. How to get them working is the problem? Is there something about dsls I need to know that will explain these to me? I feel I am THAT close.......... Posted by ke4nt1 on Feb. 14 2005,16:49
I haven't found a "limit" yet.. I'm currently working on a "doom3.uci" which will be several gigabytes in size. Same for many other titles and applications. On your app, when you normally apt-get install it, ( postgresql ) are you run thru any pre or post-install setup screens? ( these may be what you need to emulate ) deb2dsl, amazing as it is, does not work with extensions that have multi-part install routines.. 73 ke4nt Posted by softgun on Feb. 15 2005,11:59
Thanks for asking about postgresql.What i am trying to do is make zope and postgresql run, connect them up and add a package to zope (web application server). apt-get update apt-get install zope-psycopgda apt-get install zope-parsedxml apt-get install python2.2-imaging apt-get install postgresql These are automatically setup. /etc/init.d/postgresql start Then I have to create a database file, make a few directories, change some ownership of certain directories, add a zope product etc. Any guidance is greatly appreciated. Thank you again! Posted by softgun on Feb. 18 2005,16:14
What is the way to create such a dsl? I could not find a HowTo about this in the HowTo section. Posted by clivesay on Feb. 18 2005,16:53
The simpliest way would be to package up the .deb files and have a script that installs them. Take a look at the alsadebs or aliendebs .dsl packages in the repository to see an example.A much more tedious method would be to take a snapshot of the filesystem before you install the package and then a snapshot afterwards to see what has changed. Then you would have to go through and pick out the changed files one by one. I suggest the first option unless you just really want to dive into a project with the second option. Chris Posted by joer on Feb. 25 2005,04:44
Why can't we just have UCI type extensions?They are so much more efficient. Is there any reason? Posted by mikshaw on Feb. 25 2005,04:51
There is a limit to the number of cloop devices available...8 or 9 i think.
Posted by softgun on Feb. 27 2005,09:47
8 or 9? That is a lot to me!But there are other restrictions like the fact that ALL the files must be in one single directory structure running in the opt directory, right? Posted by joer on Feb. 27 2005,19:32
As of Dec 2004 klik went to all cmg format.Seems to me cmg format is the same as DSL uci format. So it can be done. Too bad klik is "KDE Live Installer for Knoppix" We don't have KDE. But lets see DSL has had UCI since May 2004? Looks like DSL has been sitting on a great idea. I guess UCI should be renamed to FLID. Fluxbox Live Installed for DSL??? Still seems a better way to me, and I guess to others too. Posted by mikshaw on Feb. 28 2005,03:54
...except for the fact that UCI has nothing to do with Fluxbox. It can run regardless of what graphical system you have, and without a graphical system if your application doesn't require it.
Posted by Fordi on Feb. 28 2005,06:45
Ahh, I don't think that's such a good idea. See, the 400M upper limit is likely based on the assumption that:1) you have 512M of ram on the target machine 2) that's the uncompressed size of your application (or compressed and 1G) 3) The space is taken up by Large Files 4) yours will be the only dsl package on said disc / boot. If these are not all true, you should likely adjust your upper limit accordingly. It's all about the target machine, you know. Posted by Fordi on Feb. 28 2005,07:08
cloop has a 2G limit to uncompressed size, I think, and you are limited to 8 cloop devices You can get up to 255 normal loop devices, too. (to unlock the extra loops requires: -A change to the boot parameters (add max_loop(s?)=n) -An on-boot script that will create the devices necessary to handle the extra loops (makedev is not available on the core DSL distro, so it would require a script in /etc/init.d, a link to that script in /etc/rcS.d, and a copy of makedevs from the dsl-utils.dsl package, all stored within a customized .dsl pack) I did a little research on this a short while back and had someone unveil the key to this trick in the "Talk" page. And, yes, this will be in my own microlinux distro (so far, I've got a pre-beta running to X from eight or so megs, and is compatible with .debs, .dsls, .ucis, .rpms, dsl-type .tar.gzs (must be renamed to dtz for getting-there's sake), my own sqsh (kept cos it's simpler to just use the loops as sqshfs than to translate the loops to cloops - it has the same directory limitation as uci) and sqx (same as sqsh, but links into the filesystem proper) formats. Running to X, but with little else but the WM (which is post-boot mounted and configurable from outside the main FS file. I currently have Fluxbox, KDE, xpde and icewm). Yes. KDE. As an sqx, and eating about 10% of the filehandles (1k blocks in a quick ramdisk kernel hack greatly reduces the amount of space utilized by symlinks. To be honest, I don't know why one has to have blocks from RAM. Sure, 1 Byte blocks would reduce performance - too many system calls - but 1k works fine on my 500 MHz machine.) I also had to do a dirty little hack to the uClibc that I've been using. If a symlink is opened for write, it checks if the target is read-only, and if so, replaces the symlink with a copy of the file and reopens it, returning the file handle. Recursive function, nice and elegant. Even detects looped links and complains about them. The upshot is that the user can run an app like KDE mostly from a read-only source, but in the even the WM has to write to its own FS links, it can, and does, nice and transparent-like. So yeah. Nice stuff. Too bad it's still buggy (mostly getting apps to work is similar to pulling teeth. Things are supposed to work all transparently, but the second you install a decompression-type package - like a dsl, tar.gz, rpm or deb - over a big linked app, things get all funny. S'probably from RAM use, so I think I just have to have a target machine calculator. (Idiot light-type + details. you know. OK! This disc will run fine on the target machine, WAIT! This disc will boot fine, but initializing too many of the optional apps will crash your machine, STOP! This disc will not boot on your target machine) Sorry I been so quiet lately. While I've been doing that, on my Windows machine, I've been toying around with GBA development using HAM. Fun stuff, really. Nearly have a port of the old DOS game, Brix, completed. Posted by l0st on Mar. 21 2005,12:04
resurrection of article & slight off topicHere I have, bringing links from the dead: BTW I'm not a programmer of any sort so anything I'm saying will be in layman terms. 1) < Relink >Compiles everything into the same directory, effectively making a self contained app. Works like .uci uncompressed I guess. CLI interface and stable, works hand in hand with make. 2) < Autopackage >Autopackage is something like the way .exe's work in Windows, only that it's also self contained in the same directory. Version 0.7 with CLI, GTK+ or Qt frontend, stable API with 10+ working samples. 3)< Klik > < (1) > is something of a < Click'N'Run >, only that it's free in both senses of the word. Incredibly akin to .uci but has a far larger repository, already tested with Knoppix, MEPIS and Kanotix. Stock Debian is rumoured to work also; KDE may or may not be required. So why am I talking about this? First things first Damnsmall is an incredibly useful distro and has a not-so-small following, with a very neato .dsl and .uci system, complementing an already powerful apt system. But myDSL will never, ever reach the scale of that of such repositories mentioned above; nothing short of a miracle or sudden upsurge of myDSL contributions of massive proportions will ever achieve that. That's why I'm suggesting here to ya'll that we include another package management system! It WON'T replace the current system if I'm reading things correctly, its purpose will only be to benefit the DSL community more by providing huge treasure troves of software that is easily reachable. Posted by roberts on Mar. 21 2005,13:06
I have been an advocate for our ci and uci. In fact if you look at the timeline of these features, DSL has had ci and uci before there was even a Klik and run.I believe the vast majority of the Kilk repository will be KDE based. So how much would we actually gain? With the added capability of completely "unloading" the uci in RC1. I would think that is should become "the" most desired form of an extension. I believe all .tar.gz could be easily turned into a uci which would benefit all users. But also, another thing to think about. We are a community driven distro. It is easy to Monday Morning Quarterback and say it should be this way. We have grown in many directions. And mostly to be easily embraced by the user community. It is indeed quite simple to make a tar.gz, then graduate to a .dsl. But look at the few uci s that have been contributed. So being community driven then those types that are easy to make become the most popular and not necessarily the best for package management. We are still evolving and will continue to embrace technology that our user community is comfortable with. You can look for many more uci's in the near future as they provide an easy pacakge management as described in your above quoted articles. One last thing, our uci's do not require special software to download, nor do they even require any window manager to use. Posted by clivesay on Mar. 21 2005,13:19
Interesting stuff.I looked at some of your links. I'm not sure what the actual system requirements are for things like klik. DSL may be too stripped down. I also noticed it appears to only be stable for Knoppix 3.3 right now while DSL is based on 3.4. May be a really cool option for the future. Since Frugal now supports a persistent /home and /opt directory I have begun converting .dsl's to tar.gz's. So far I have been fortunate that apps have only required export LD_LIBRARY_PATH or a symlink or two. I've been able to convert tuxpaint, wmdrawer and gcompris. Ke4nt pitched in and converted gcompris to a uci! I know also that lgames will convert I just haven't packaged it up yet. EDIT: I just saw roberts posted at the same time I did and agree with him. Posted by softgun on Mar. 21 2005,16:22
I agree that we should go ahead with our repositories and extensions. dsl is not like the others as it is a little distro which thinks big! I have been experimenting with the creation of postgresql and zope for some time with debs and failed. now I have downloaded the source distro and am trying to make them into ucis! ./configure --prefix -/opt/oio/pgsql but it needs other dependencies like zlib and python-dev and also creates a /var/lib/pgsql directory during make and make install. Does this mean this cannot be made into a uci? Posted by clivesay on Mar. 21 2005,16:58
softgun -As far the file that goes into /var/lib, you could try moving it to the /opt directory and see if the app still runs. If it complains that it needs to see /var/lib you can keep the directory in /opt but symlink it to /var/lib. That should satisfy the program. After that, just create a little startup script to create the link and launch the app and you are all set. Posted by cbagger01 on Mar. 21 2005,17:34
The problem with "/var" directories is that they are usually created for the purpose of saving log information or other writes.If you symlink a /var subdirectory back to a read-only subdirectory inside the UCI mountpoint /opt/ProgramName/var/MyVarDir you will not be able to write to it. I don't know if this statement is true for this application, but my generic solution would be to include a check to see if the /var/programname subdirectory does not exist and if so: mkdir /var/programname and then cp over any "writable" logfiles before starting the applicaiton. You would included this in the beginning of the wrapper where you usually put your LD_LIBRARY_PATH stuff. Posted by softgun on Mar. 22 2005,16:52
I will try all that you say. The fact that uci are read only maybe a cause of some problems in a database program!
Posted by softgun on Mar. 27 2005,16:25
Hello Robert, A few questions when you have the time to answer :-) I would like it very much if you could kindly explain what you mean by completely "unloading" means to me/us? If ucis arre read only, does it mean postgresql -which is an RDMS - is not suitable to be developed as a uci? Or if we do create one can we have it write the database to another writable directory? Is this the solution? In developing uci, as they have to be in a single directory, how can tar.gz files which do not have this requirement, be turned iinto a uci? Do we have to nake ucis by using source downloads and putting them into a single directory with config --prefix=/directory ? Thank you. Posted by roberts on Mar. 27 2005,16:47
Using emelfm double click on a uci extension and it loads up ready to use including menu and icon. Double click on it again and it is gone. No menu, no icon, no mount, no empty directory. That is what I mean my completely unloading the uci.
Database or anything else that need writeable access would require one to do an analysis of those parts requiring such. I have made in the past both a Postgres and MySQL cdrom based applicance. Also, there are many other Knoppix derived database products. So it is a matter of being presice with the use of syslinks.
I think you are confusing the general term of .tar.gz vs the .tar.gz dsl extensions. The DSL tar.gz extension by definition can only be located in the two natively writeable areas being /opt and /home. Both of two directores is the structure used in the uci type extension.
Not necessarily. Many of them can use PATH and LD_LIBRARY_PATH. Each one would need its own analysis on how best to proceeed. Posted by softgun on Mar. 28 2005,10:38
Robert,Thanks for your quick reply. :-) You have already made some wonderful contributions to DSL, and the unloading feature is very nice, very neat. I am trying to develop postgresql/zope/python based application called OIO to work in DSL. Postgresql is accessed through Zope via a backend databse adaptor. The python uci is already available. I think Zope too is suitable to become a uci as it can install in one directory. The problem is postgresql - have tried a dsl, tar.gz but when I try to "run" the file it crashes. I will keep trying, as OIO is my favorite software and DSL my favorite distro!! |