Faster Copy ?

Forum: DSL Tips and Tricks
Topic: Faster Copy ?
started by: humpty

Posted by humpty on May 06 2006,03:32

Still can't figure out why linux is so slow copying large files between FAT32 disks (?)

But using the dd command does help (approx 1.7x);

~/tools/fastcopy.sh

Code Sample

#!/bin/bash
# A faster copy using dd
#
# Add this button command in EmelFM
# aterm -e /$HOME/tools/fastcopy.sh %f %D

echo copying..
echo " " $1
echo " to" $2

dd if="$1" of="$2/$1" bs=5M

echo hit return..
read

# * only use for 1 file at a time *

Posted by cbagger01 on May 08 2006,04:15

Do you have DMA turned on for the drives?

Either boot with

dsl dma

or type:

sudo su
hdparm -d1 /dev/hda
hdparm -d1 /dev/hdb

etc after bootup

Posted by humpty on May 10 2006,19:41

yep, dma was confirmed in the boot msgs

but there seems to be a lot of thrashing going on.
these are two seperate disk drives.

Posted by WDef on May 16 2006,12:25

Is a second write of the same file (made straight after the first) to the same disk much faster?

As you probably know, Linux tries to maximise its use of memory, partly by caching disk writes to memory, so subsequent disk writes of the same file should be fast.

Posted by humpty on May 20 2006,18:39

2nd write is defintely faster. (but why would i do that?)
it's the 1st write i'm interested in, cos i have large files to get from
one disk to another. typically 300 to 700mb (and i only want to copy them once).

so anyway, i just loaded up a virgin copy of dsl2.4, nothing else added,
and here are the timings for 1st copies straight after each re-boot.

file 1 = 319100928 bytes
'cp'=30 secs 'dd'=15 secs XP=8 secs

file 2 = 565446002 bytes
'cp'=85 secs 'dd'=35 secs XP=15 secs

'dd' was using a 5Mb bucket size.
emelFM standard copy has the same time as a bash 'cp' command.
XP using the windows explorer.

i didn't really want to compare this with windoze, but i'm curious to what the trick is. windoze is nearer to the 40 mbyte/sec performance i expect between two seperate HDs, they both run at dma100 and was checked with hdparm (udma5).

obviously the cp performance on linux got worse for the larger file.
there was a lot of disk thrashing, maybe the caching? maybe too small a bucket? maybe some other background task?, though i made sure nothing else was going on.

Posted by WDef on June 24 2006,09:44

Sorry this is an old thread - I'm not in the forum much right now.

If you're still interested Humpty -
are you running with swap? If so try the same experiment with noswap (I will when I'm next on dsl). If the swap is thrashing that's probably what's slowing it down.

I suspect dd and cp use memory (and hence swap) quite differently but I'd have to get out my books to check and I'm on holidays

Posted by humpty on June 24 2006,21:43

no swap.

ha, i have given up on this. Web searches don't turn up anyhting.
The standard hdparm test do give 50 mbytes/s and 40 mbytes/s for each disk, but this I suspect are only the speeds of the interfaces.
The only hunches I have left are either a vfat driver that's not optimized or may something to do with multi-threaded dd.

Posted by WDef on July 03 2006,09:24

The kerneltrap forum can be good for this sort of thing. You might try posting there. If you do find out what it is, please post back here since it's interesting.