backup on the web
Forum: DSL Ideas and Suggestions
Topic: backup on the web
started by: ypx
Posted by ypx on April 05 2005,16:23
i saw < this > today.and i now wonder if there's a way to use a similar concept in DSL :
to put the backup file on a distant ftp space, and to be able to restore it from there upon booting ?
is it allready possible ? if so, how ?
and if not, what do you think about it ?
for my part, i think that would be usefull, but i would like this system to use a personal ftp space, not a public one.
Posted by mikshaw on April 05 2005,16:36
I suppose it's not much different than the current backup system, only it uses ftp instead of a local drive. Doesn't seem like it would take a terrible amount of effort to implement.
Posted by roberts on April 05 2005,17:00
I actually implemented this around 09/04 which be around 0.8. series. When I released it to testing, the feedback I got was that it do not work with MS ftp servers. It worked fine with unix ftp servers. So, I guess, again, MS pollutes internet standards. I did not release it. It does use encryption to protect the backup file. It was small enough to be included in the base iso. Another thing that I found frustrating is that most ISP give you a small ftp or personal webspace, versus, a very large email space.
Posted by RoGuE_StreaK on April 06 2005,02:30
| Quote (roberts @ April 05 2005,13:00) |
| ...most ISP give you a small ftp or personal webspace, versus, a very large email space. |
I don't suppose there would be a way of being able to extract data from a gmail account? ie., your backup is emailed as an attachment to your account in a very specific format (title, sender etc), then to restore backup it searches your gmail account for this data and loads it?
Maybe that way too you don't need to worry about specifying which server you were using (ie. if there wasn't a centralised DSL settings server), all you have to do is enter your gmail address and password?
Posted by cbagger01 on April 06 2005,16:52
There is a project to manipulate files that are stored in gmail messages.I believe that it is called gmailfs and it allows you to "mount" your gmail mailbox as it if was a disk drive.
But my guess is this project requires a kernel recompile and will be hard to keep up and running if Google decides to change their e-mail interface one day.
Posted by isager on April 06 2005,18:29
In my opinion it would make much more sense to support FTP and perhaps something like WebDAV. This is proven technology and I do think roberts agrument about not working with MS ftp is a bit thin :-)
Posted by cbagger01 on April 07 2005,03:03
The problem isn't so much the buggy MS FTP server.The problem is that people don't KNOW about the buggy MS FTP server so it produces a ton of "help me, my web backup doesn't work" messages for the developer.
It is easy to trivialize this drawback if you aren't the guy getting all of the complaints.
Posted by ypx on April 07 2005,19:38
I agree with cbagger on this one, that's why i did not insist when i saw Robert's answer. Hope that this will one day become possible...
Posted by roberts on April 07 2005,20:10
Ok. Lets try an experiment. A call for beta testers.I have uploaded my program that I wrote last September. I have been using it without problems with my ISP.
I have packaged it as an extension. If the results are good I will add it into the iso. Here's what it is.
The extension webdata.tar.gz contains:
1 webdata the Lua-fltk program
2. webdata.lst a file list similiar to filetool.lst, i.e., a list of files to be store on the web/ftp site.
3. .netrc used only during transmission for automatic ftp session.
All there are installed into and can be run from /home/dsl
You must be using dsl-1.0RC2 as this already has the encryption program that I am using. I am planning more with this in the next full release of DSL.
You must have an ftp account with your ISP and know your password.
Note this program is to be considered beta. use at your own risk.
But please be a good beta tester and let me know yea or nay.
You should download the webdata.tar.gz and use mydsl-load on it or emelfm however you normally load your extensions. Three files should be installed into /home/dsl/
First take a look at webdata.lst. It is just a sample of what files to backup to the ftp site. It is up to you. Since, you must already have a net connection, it is different from filetool.lst
To run this beta program open an xterm as normal user dsl and type
./webdata
A GUI appears:
1. Enter the ftp url for your ISP.
2. Enter your ftp login name for this ISP
3. Enter your ftp password for this ISP
4. Enter a password to be used access to the encrypted data.
Select BACKUP
The program displays the ftp connection messages.
If you get this far, it probably means it is going to work for you.
If you see error messages in the xterm window please report back what they are.
To restore the data, simply run the program again and select RESTORE.
Be sure that you know your encrypted password otherwise no one is going to be able to help you.
This has worked reliably for me. I hope we can get some good feedback here.
If so I will do more cleanup and integrate it better into the system.
Robert
Oops, almost forgot, you can get it here. < webdata.tar.gz >
Posted by gantrithor on April 08 2005,12:54
i'll say, slax has nice ideasroberts gets a cookie for recognizing that
it might sound a little uncourteous and rude coming out like this but i assure you it's not what i meant, but some kind soul might be able to give a wee bit of bandwidth for the poor guys who just happen to have ms' shitty servers
2 cents
Posted by cbagger01 on April 08 2005,16:22
That would be the other missing piece of the puzzle.Mr. Slax developer doesn't mind hosting a few THOUSAND user settings files at 8MB per user.
Good luck getting someone here to ante up for that storage + bandwidth. A white knight could appear and help out but I wouldn't hold my breath if I were you.
But this (Roberts) approach works fine if you have your own personal (limited) web/ftp space and the server is not MSFTP.
In my case, I have 10MB available with my cable service but most of it is taken up by baby pictures which leaves me SOL unless I buy some space elsewhere.
Posted by RoGuE_StreaK on April 09 2005,01:58
That's why I was asking about gmail or similar, could even set up one account for all DSL users to use, 1GB (or is it more now?), say a 2MB limit per person, gives 500 users the ability to backup settings and plugins to the web.Not sure about the issue of gmail etc changing their login/collection methods - there are plenty of POP email systems that claim to be able to setup to get your hotmail / et al, so I'm thinking this would restrict any changes these webmail systems could make, as they now have third-party software relying on their current login/collection systems. Of course, they could just say "stuff you" and go ahead and change anyway...
Posted by roberts on April 09 2005,02:25
gmailfs or even the ftp proxy uses libgmail which requires python which is too big for inclusion in the base iso. Unless someone wants to translate libgmail to something we support locally...
There are plenty of free webhosting with ftp access.
I found another one which gives 20MB free and I was able to ftp 14.5MB backup single file.
We will see if they kick me off. But so far... so good!
I see many more free webhosts restrict single file size to 1MB or don't allow true ftp.
Posted by drx456 on April 09 2005,15:44
how to backup entire /home/dsl directory? How to setup the .netrc file?
Posted by roberts on April 09 2005,16:54
The .netrc file is auto created when running the webdata script and then destroyed upon completion. This is for security reasons and also not to interfere with the use of the ftp gui program that you may need for other servers or uses. I don't think it would make sense to backup the entire home/dsl as the purpose is to allow you to move from machine to machine and have access to the data that would be machine independent.
But as always, I give you the flexibility to do whatever you want. Just edit the webdata.lst file it is the same format and concept as the filetool.lst file. If you are not familiar with the filetool.lst then I would strongly suggest to learn and practice on that first. You can read about it in the Getting Started guide.
There is an old saying, "I give you enough rope that if you want to hang yourself. You can!"
Posted by RoGuE_StreaK on April 09 2005,21:18
Actually there's an idea, is there a way to exclude something in the filetool.lst / webdata.lst?You can add /home/dsl so that you backup all the bits and pieces you may have in there, but you would also get the dirs and settings in there that you may not want to backup, say /home/dsl/.xtdesktop - if you could exclude certain files/dirs, could make it more flexible, as you don't need to specify every single file you want to backup?
(as when you specify a directory in the lst, it includes everything in that directory and it's subdirectories)
Posted by Caspar_s on April 10 2005,04:55
So when does this happen? Can you use this as a replacement for your "normal" restore? Or would this be in addition to a fewer files in your backup.tar.gz? Most of your net connection files in the backup, and your fonts, desktop, etc in your webdata? (sorry, haven't figured out how to do the dual boot yet, and have to be the fileserver for the wife, so I haven't tested this out yet)
Posted by DeeJay on April 15 2005,07:50
| Quote (roberts @ April 07 2005,16:10) |
| Ok. Lets try an experiment. A call for beta testers. I have uploaded my program that I wrote last September. I have been using it without problems with my ISP. I have packaged it as an extension. If the results are good I will add it into the iso. <snipped> Note this program is to be considered beta. use at your own risk. But please be a good beta tester and let me know yea or nay. <snipped> This has worked reliably for me. I hope we can get some good feedback here. If so I will do more cleanup and integrate it better into the system. Robert |
In the apparent absence of other beta feedback can I be the first to say "Yea". Thanks Robert, it works fine for me. A really neat demonstration of the power of lua-fltk, which I had never heard of until yesterday. I've already got a custom version of webdata that holds the URL and Username for my ftp space, to save me re-typing them. [A suggested luxury tweak would be to take the most recent user input - perhaps not the passwords - store them somewhere ( .webdatarc ? ) and present them as default choices at the next run.]
Personally, I am happier knowing how the data is encrypted and where it is stored using dsl+webdata than I would be with the slax+webconfig model where my data ends up on someone else's server.
Testing has been carried out using DSLv1.0 running 'embedded' in Qemu under Win2k.
Posted by roberts on April 15 2005,15:15
Thank you. Thank You. Thank You.Finally something other than chatter; a real beta tester with meaningful results.
Yes, I will make persistent your URL and login so that you don't have to type it each time. This is currently a rough cut just to test the concept.
Personally, I think it is a great idea and have been using it since I wrote it last Sept.
I really want to integrate it into the base iso. The size of the lua-fltk is certainly tiny.
So, now I know it works for a least two people!
Any other real beta tester, PLEASE speak up!
Posted by green on Sep. 10 2005,14:24
Been playing with the Webdata Backup/Restore utility.I am using DSL 1.5 Live toram (not frugal) on the client machine.
I setup an FTP server using DSL 1.4 on my local network.
I can successfully FTP the backup file to the server and it is written as data.des which is ~48Mb and written to RAM. (file size seems rather large!)
The FTP server is running toram from LiveCD. There is a hard drive in the FTP server, hda1.
I would prefer to have the data.des file written to hard drive vs. RAM.
What do I need to tweak to accomplish this?
Posted by green on Sep. 12 2005,14:29
I just have to comment that the webdata utility is quite rewarding to play with. I have been looking for a simple and convenient way to do nothing but save firefox bookmarks/profile without having to carry around a USB memory thingy. The webdata utility serves that purpose very well.
Posted by Borderpatrol on Sep. 12 2005,23:28
As soon as I can boot into DSL I am going to play with that. I just need to do some stuff for work that I should be doing now. Im going to be home from school tomarrow (wisdom teeth yanking) so I plan on trying it then. We shall see. Hopefull Comcast will hold up to the challenge.
Posted by green on Sep. 13 2005,14:18
I have Road Runner as my ISP. They give customers 5Mb of free webspace and allow FTP to it. I've been using that and an FTP server at home. Both seem to work fine from home and from not at home.
Posted by Borderpatrol on Sep. 14 2005,14:31
My net was down so I couldn't play with it. But I can FTP to it at have i believe i have 25+ MB
Posted by isnoop on Dec. 03 2005,23:06
My FTP doesn't allow uploading to the root, so I've modified the script to allow folks the option to provide a different path.Here's my slightly modified version:
< http://isnoop.net/pub/webdata.tar.gz >