Ultra-fast backup/restoresForum: DSL Ideas and Suggestions Topic: Ultra-fast backup/restores started by: WDef Posted by WDef on Sep. 21 2005,13:03
< lzop > is the *fastest* compression utility around. It compresses on average three times faster than gzip.I've hacked backup/restore (again) to use lzop. If lzop is not installed, the hack defaults to using gzip. The speed increase, especially when backing up, is substantial. Yet the resulting archives are only slightly larger. Results (p3, device=hd): Non-encrypted backup with gzip --> 42.04MB = 46.7 seconds Same non-encrypted backup with lzop --> 46.72MB =5.8 seconds !! Non-encrypted restore with gzip --> 5.8 seconds Same non-encrypted restore with lzop --> 2.24 seconds !! Encrypted backup with gzip --> 42MB = 54 seconds Same encrypted backup with lzop --> 46.72 MB = 14.1seconds !! Encrypted restore with gzip --> 14.2 seconds Same encrypted restore with lzop --> 11.1 seconds !! The hacked scripts are backwards-compatible with gzipped tarballs. This enables: 1. Painless user migration from plain or encrypted gzipped backups to lzopped backups - just place the extensions (see below) on your mydsl partition and reboot. 2. Old copies of gzipped tarballs continue to be restorable. 3. Remastering to add the scripts to the base does not mean lzop must be added or always loaded. Without lzop, backup/restore should proceed using gzip. To test: ~~~~~~ The altered scripts and lzop are inside extensions. 1. Download < this > (r click, "save as") and re-name fastbackup.dsl md5sum 7d8996a88d83c1e9a706a849ef70d656 (please check) also < this > and re-name lzop.dsl md5sum a750aaf85deb4cd7041802c5d9bf17ef (please check) 2. Place both extensions on your mydsl partition and reboot including the mydsl={your_mydsl_partition} and restore/protect options, as appropriate for your existing gzipped tarball. A normal restore using gzip should then occur. 3. When finished, hit 'backup' from the system menu or reboot. Notice the increase in backup speed. Check your backup partition - you should find backup.lzo.des (if using 'protect'), or a new file backup.tar.lzo 4. On booting up again, the hacked script will try to find & restore one of backup.lzo.des, backup.gz.des, backup.des (old), backup.tar.lzo and backup.tar.gz (in that order). If the candidate archive doesn't exist, or there is no 'protect' password or lzop where required, then the next candidate is found and restored if possible. See the screen messages for details. Unofficial scripts. No warranty, use AYOR, make copies of your tarballs beforehand, etc. (Many thanks go to Robert and others who have encouraged me recently). Posted by SaidinUnleashed on Sep. 21 2005,14:32
Looks like a lot bigger to me. I'd rather wait the extra few seconds for a smaller backup. I don't have the space to store an extra meg or two. -J.P. Posted by WDef on Sep. 21 2005,15:06
That's wait 41 extra seconds on that size tarball and a p3. It's an 8-fold speed increase in that example, for a ~10 per cent increase in tarball size. I'd like some figures on a huge tarball. People with tight backup headroom (and they're not everybody) don't have to load lzop. I'd be grateful if people would test the work, which took some time and effort. --------------------------------------------------------- BTW: It might be safest for now to pass the backup device name directly through the boot options eg restore=hdb6 or protect=hdb6 Posted by cbagger01 on Sep. 21 2005,17:30
Has anyone tried uzing "gzip" with a different compression setting?I find that "gzip -2" compression level is a good compromise between speed and compression ratio. I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this. Posted by WDef on Sep. 21 2005,17:41
There was a reasonably thorough survey published in Linux Journal a few months ago, using results over a number of different data types for a variety of different compression progs and levels. Log curves etc - more rigorous than< this >test on Windows, in which lzop was also the fastest. gzip didn't come close speed-wise - lzop was the fastest of the lot of them. There was another prog that achieved the very best compromise between speed and compression but it uses up gigantic resources - 900 MB ram etc I'm talking about large speed increases without too unacceptable a drop in compression ratio. A 10% increase in tarball size is not at all unacceptable to me. Why not test lzop - can't hurt can it. Posted by friedgold on Sep. 21 2005,22:03
I'd never heard of lzop until this post so I thought I'd try some comparisions of my own (including vs gzip -2). I got the following results
This is on my 1.8GHz duron PC. Clearly lzop offers much quicker compression times than even gzip -2. I can see how on a slow PC this could be beneficial. This needs to be balanced with the draw backs : lzop is a uncommon format making it hard to manipulate the files in the backup from outside DSL. lzop offers lower compression ratios than gzip making it less useful when space is limited (e.g. on a pendrive). Posted by WDef on Sep. 22 2005,07:04
Again, I'm not necessarily proposing that lzop should be added to the base.It would be very easy to change the hack to be able to use lzma, bzip2 or any other comp programs as well as lzop and gzip, depending on what compression extension was loaded. You could then take your pick of max speed or max compression, or a compromise. How's that for a big picture. ie This does not have to be solely about lzop in the longer term, so could we please not get bogged down in a discussion about the merits of various programs. What I am asking for now are testers for the present extensions. So far nobody has said they've tried these yet. How about some feedback? ----------------------------------------- To respond to Friedgold's points: Thanks for the figures. These will vary according to the mixture of data types of course.
If you've restored using lzop and decide to backup using gzip, just remove the lzop binary thus
and hit backup. You should get a gzipped tarball back. Then either don't load lzop.dsl at next boot, or you can delete /move the previous lzopped tarball from the backup device. The altered script should then restore much as the original did. Of course a change of comp prog could easily be automated from (eg) the backup/restore widget. Similarly, if you've restored ordinarily with gzip, you can load the two extensions and hit backup to get an lzopped tarball. Lzop is available < for Windows >, and various *nixes etc. There's a statically-linked i386 linux binary anyway.
Pendrives are very slow to write to, and getting rapidly bigger, so some who back up on pendrives might be looking for whatever speed gains they can get. Posted by Patrick on Sep. 22 2005,08:38
I just installed the 2 dsl's and tried it out. Works great and fast. my backup.tar.gz = 8.92 Mb and the same backup.tar.lzo = 9.74 Mb. Processor used to hit the roof when backing up (100%) now that's in the past. Backup time improved greatly. I run from a 512 Mb usb-stick (DSL 1.5) and i like the improvement in speed. (keeping this one!)Thnx man! Posted by Patrick on Sep. 22 2005,10:07
All i need now is the right command-phrase in emelfm to "unpack in other panel"I use emelfm a lot and need to see inside the backup.tar.xxx (or unpack it temporaraly) Can you help? Posted by WDef on Sep. 22 2005,15:44
'Configure' button --> Filetypes -->AddDescription Lzop tarballs Extension tar.lzo, tzo -->Add (under "Actions:} Name Unpack in another panel Command x cd %D; lzop -dc %d/%f | tar xvf - --> OK --> Done --> OK --------------------- Note it's lzop -dc, not lzop -c BTW Patrick - Thanks for testing. Posted by Patrick on Sep. 22 2005,16:22
briljant, thnx
Posted by cbagger01 on Sep. 23 2005,05:12
wdef,I apologize. I wasn't trying to hijack the topic but it appears that I did. I don't know how lzop does it, but it looks like it is a real improvement over gzip for the "backup" part of the "backup/restore" process. However, if the DSL developers are unwilling to go with lzop over gzip in the base ISO, it might still be a good idea to change the "gzip" command in the official DSL backup scripts over to a "gzip -2" command. This would be a good idea for a future release of DSL. It would result in a 7 / 18 = 39% improvement in backup speed while still maintaining gzip compatibility. But I digress again. Posted by mikshaw on Sep. 23 2005,05:26
or even better (as far as i see it), maybe have the backup compression be chosen by the user, like the downloads directory is now chosen by the user. Personally i don't have any issues with the speed of gzip -9, and prefer it to something faster when the size is smaller. At the same time, there are also people who prefer to go farther to use something like bzip2, which is even smaller, but slower. If anything is going to change, I don't see any reason why it couldn't be changed in a way that benefits both those who favor speed and those who favor size.
Posted by cbagger01 on Sep. 23 2005,06:02
OK, how about this compromise:a "gzip -$level" where level is a variable that is extracted from the .dslrc settings file? Of course, my strong preference is to have "2" be the default value in the file, because it is ~40% faster but with only a ~5% size penalty. Posted by WDef on Sep. 23 2005,09:28
Up to the developers, it wouldn't be difficult. Some configurability is better than none? I'll put it in the hacked script if you like.
No problem - stuff needs to be said. I'm just keen to get feedback/bugs on the filetool.sh mods.
Your opinion would count. For me, it's not an issue - I'll just continue to load my extensions. These could go into the repository - I've emailed Ke4NT re lzop.dsl. If Robert agrees (he is the original filetool.sh author), fastbackup.dsl perhaps could go there later.
Had similar thinking:
I spoke a bit soon with the "very" in "very easy" because of the different command switches/syntax with some of these progs (p7zip etc). But a restricted generalisation is doable - I think I'll try it. Interesting what Patrick said about CPU load:
Anyone else noticed this? Posted by WDef on Sep. 23 2005,12:31
CBagger, Mikshaw et al< Here's > another script extension, altered so compression levels can be set in the file /home/dsl/.backuprc These get applied to gzip (and lzop if it is loaded). Run it in a root shell:
Docs say that lzop -1 probably wouldn't make much difference, and it didn't. So I'm happy with default lzop (which is -3). Setting the compression levels for busybox gzip doesn't make any difference to speed etc. Gnu-utils are needed. Unless gzip from gnu-utils is put into the base, there's no point in changing the comp level of gzip in the default filetool.sh. ---------------- lzop -3 ( ~5 secs on my tarball; level 3 is the default) still backs up around three times faster than gnu-utils gzip -1 (~15 secs) - as the hype says - with the lzopped tarball 9.3% bigger. Posted by cbagger01 on Sep. 23 2005,22:29
I think that putting the GNU gzip into the base ISO would be a good thing. In addition to the whole speed issue, some users now get an "invalid magic" error from busybox gzip and no such error when using GNU gzip.In the past, there was a similar problem with the "tar" command and the developers decided to replace busybox tar with GNU tar, so there is hope for gzip. Posted by WDef on Sep. 24 2005,12:25
Agreed, for both reasons. It's not very big either, but I haven't checked to see if the libs are different (not on dsl now). Meanwhile I've done a generalised version of that hack - now a re-write really - of filetool.sh. It will do backups with gzip, lzop, bzip2 at different comp levels where possible, options can be set by the user, and I may get it to offer lzma, gpg and rzip choices as well. Can do bzip2, lzop or gzip -9, or whatever. I was only interested in faster backups myself but .... I'll post it in a new thread once I've tested some more (next week), assuming people are interested. Posted by WDef on Sep. 30 2005,15:48
< Here > is a bugfix and a number of improvements to the fast backup scripts:- Fixed bug that incorrectly gave "failed" message on successful restore of encrypted lzop archive. - Only stores a device name in /opt/.backup_device if backup/restore using that device is successful. Prevents trying to auto backup to a bad device on shutdown, with a consequent loss of data (also an issue in the normal backup scripts). - New dsl-restore.sh scans for lzopped as well as gzipped archives - no need to point 'protect' or 'restore' at the device in the boot options (eg 'restore=hdb6'). But requires a remaster of your iso to overwrite the normal dsl-restore.sh for it to work this way. Other changes are described at the top of the scripts. md5sum is 8d7f239220fbb65dc3880e8785584bd7 Posted by Patrick on Oct. 12 2005,08:26
How do i make a backup.tar.gz from an exsisting backup.tar.lzo?
Posted by lub997 on Oct. 15 2005,04:23
//Posted something in wrong forum, so edited it out, but it won't let me delete it completely.
Posted by WDef on Oct. 15 2005,12:59
Just do: sudo rm -f /bin/lzop Then backup. Without lzop it'll just make & restore gzipped tarballs. |