Search Members Help

» Welcome Guest
[ Log In :: Register ]

Mini-ITX Boards Sale, Fanless BareBones Mini-ITX, Bootable 1G DSL USBs, 533MHz Fanless PC <-- SALE $200 each!
Get The Official Damn Small Linux Book. DSL Market , Great VPS hosting provided by Tektonic
Pages: (5) </ [1] 2 3 4 5 >/

[ Track this topic :: Email this topic :: Print this topic ]

reply to topic new topic new poll
Topic: Ultra-fast backup/restores, More code suggestions< Next Oldest | Next Newest >
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Sep. 21 2005,13:03 QUOTE

lzop is the *fastest* compression utility around. It compresses on average three times faster than gzip.

I've hacked backup/restore (again) to use lzop. If lzop is not installed, the hack defaults to using gzip.

The speed increase, especially when backing up, is substantial. Yet the resulting archives are only slightly larger.

Results (p3, device=hd):

Non-encrypted backup with gzip --> 42.04MB = 46.7 seconds
Same non-encrypted backup with lzop --> 46.72MB =5.8 seconds !!

Non-encrypted restore with gzip --> 5.8 seconds
Same non-encrypted restore with lzop --> 2.24 seconds !!

Encrypted backup with gzip --> 42MB = 54 seconds
Same encrypted backup with lzop --> 46.72 MB = 14.1seconds !!

Encrypted restore with gzip --> 14.2 seconds
Same encrypted restore with lzop --> 11.1 seconds !!

The hacked scripts are backwards-compatible with gzipped tarballs. This enables:

1. Painless user migration from plain or encrypted gzipped backups to lzopped backups - just place the
extensions (see below) on your mydsl partition and reboot.

2. Old copies of gzipped tarballs continue to be restorable.

3. Remastering to add the scripts to the base does not mean lzop must be added or always loaded. Without lzop, backup/restore should proceed using gzip.

To test:
~~~~~~

The altered scripts and lzop are inside extensions.

1. Download this
(r click, "save as")  and re-name fastbackup.dsl
md5sum 7d8996a88d83c1e9a706a849ef70d656 (please check)

also this and re-name lzop.dsl
md5sum a750aaf85deb4cd7041802c5d9bf17ef (please check)

2. Place both extensions on your mydsl partition and reboot including the mydsl={your_mydsl_partition}
and restore/protect options, as appropriate for your existing gzipped tarball. A normal restore using gzip should then occur.

3. When finished, hit 'backup' from the system menu or reboot.  Notice the increase in backup speed. Check your backup partition - you should find backup.lzo.des (if using 'protect'), or a new file backup.tar.lzo

4. On booting up again, the hacked script will try to find & restore one of backup.lzo.des, backup.gz.des, backup.des (old), backup.tar.lzo and backup.tar.gz (in that order).  If the candidate archive doesn't exist,  or there is no 'protect'  password or lzop where required, then the next candidate is found and restored if possible.

See the screen messages for details.

Unofficial scripts. No warranty, use AYOR, make copies of your tarballs beforehand, etc.

(Many thanks go to Robert and others who have encouraged me recently).
Back to top
Profile PM 
SaidinUnleashed Offline





Group: Members
Posts: 1428
Joined: Mar. 2004
Posted: Sep. 21 2005,14:32 QUOTE

Quote (WDef @ Sep. 21 2005,08:03)
Non-encrypted backup with gzip --> 42.04MB = 46.7 seconds
Same non-encrypted backup with lzop --> 46.72MB =5.8 seconds !!

Non-encrypted restore with gzip --> 5.8 seconds
Same non-encrypted restore with lzop --> 2.24 seconds !!

Encrypted backup with gzip --> 42MB = 54 seconds
Same encrypted backup with lzop --> 46.72 MB = 14.1seconds !!

Looks like a lot bigger to me.

I'd rather wait the extra few seconds for a smaller backup. I don't have the space to store an extra meg or two.

-J.P.


--------------
They say if you play a Microsoft CD backwards, you hear satanic messages. That's nothing, cause if you play it forwards, it installs Windows.

Unleash the power of the TILDE~~~
Back to top
Profile PM AOL MSN YIM 
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Sep. 21 2005,15:06 QUOTE

Quote
..wait the extra few seconds ...


That's wait 41 extra seconds on that size tarball and a p3. It's an 8-fold speed increase in that example, for a ~10 per cent increase in tarball size.  I'd like some figures on a huge tarball.

People with tight backup headroom (and they're not everybody) don't have to load lzop.

I'd be grateful if people would test the work, which took some time and effort.

---------------------------------------------------------

BTW:  It might be safest for now to pass the backup device name directly through the boot options eg

restore=hdb6

or

protect=hdb6
Back to top
Profile PM 
cbagger01 Offline





Group: Members
Posts: 4264
Joined: Oct. 2003
Posted: Sep. 21 2005,17:30 QUOTE

Has anyone tried uzing "gzip" with a different compression setting?

I find that "gzip -2" compression level is a good compromise between speed and compression ratio.

I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this.
Back to top
Profile PM 
WDef Offline





Group: Members
Posts: 798
Joined: Sep. 2005
Posted: Sep. 21 2005,17:41 QUOTE

Quote
I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this.


There was a reasonably thorough survey published in Linux Journal a few months ago, using results over a number of different data types for a variety of different compression progs and levels. Log curves etc - more rigorous than this test on Windows, in which lzop was also the fastest.

gzip didn't come close speed-wise - lzop was the fastest of the lot of them.

There was another prog that achieved the very best compromise between speed and compression but it uses up gigantic resources - 900 MB ram etc

I'm talking about large speed increases without too unacceptable a drop in compression ratio.  A 10% increase in tarball size is not at all unacceptable to me.

Why not test lzop - can't hurt can it.
Back to top
Profile PM 
21 replies since Sep. 21 2005,13:03 < Next Oldest | Next Newest >

[ Track this topic :: Email this topic :: Print this topic ]

Pages: (5) </ [1] 2 3 4 5 >/
reply to topic new topic new poll
Quick Reply: Ultra-fast backup/restores

Do you wish to enable your signature for this post?
Do you wish to enable emoticons for this post?
Track this topic
View All Emoticons
View iB Code