directory listing on http/ftp severForum: Other Help Topics Topic: directory listing on http/ftp sever started by: mikshaw Posted by mikshaw on May 19 2005,15:57
Looking for a way in DSL to get a file listing of an arbitrary directory on an http or ftp server, without user interaction. The directory might not contain an index file, so it would need to be something similar to the ls command on local/network drives.I've tried using 'wget -S <URL>', which works for some systems but not for DSL. ftp has a 'ls' command, but i don't think it can be used non-interactively. Lynx has a -dump option, but apparently this was not carried over to Links. So i'm out of ideas. Anyone know another way? Thanks. Posted by cbagger01 on May 19 2005,16:45
Many ftp clients have a "batch mode" where you can execute a series of commands in a way that is similar to a shell script.It may be possible to do a "ls" from inside ftp and dump the output to a file. Posted by mikshaw on May 19 2005,21:51
yes, but that requires user interaction, unless the ls can be passed as an ftp parameter. I don't think it can.This is something which will be run from a script as part of a larger process, and it needs to be done by something available in a standard DSL system....preferably something that does not have a user interface. If there is nothing in DSL that will do this, then i just wasted the last two days =o) Posted by friedgold on May 19 2005,22:23
Shouldn't it be possible to do with this IO redirection. i.e something likeEdit: Just been playing around with this and my previous suggestion didn't quite work. Try this ftp -n<<EOF open my.ftp.server user username password ls exit EOF Posted by clivesay on May 19 2005,22:48
mikshaw -the problem is fixed. You just need to have gnu-utils.dsl installed Chris Posted by mikshaw on May 20 2005,02:09
gnu-utils is a big file. Although it's cool that it will fix it, ideally something already present in DSL should be implemented. I'll try to work with friedgold's suggestion if i can.Edit: doesn't seem to work with http, and requires username/password, even with the -a option. That's not acceptable. Edit2: Just to clarify what I'm doing.... I've written a flua script to update a frugal installation. It contains a list of DSL mirrors, and it needs to obtain a list of files in the directories specified in that mirror list. Some are http and some are ftp. It works ok as-is if you have gnu-utils installed, but the default wget does not have the -S option required to grab a list of files in the directory. The user simply clicks on the mirror, and the list of ISO files is downloaded. This must be done automatically, and without additional effort on the part of the user in order to make the script actually worthwhile to use. If the user has to do too much work he may as well just download a file with wget or a browser, mount it, and copy the necessary files over. The main point is to make it as automatic as possible. Posted by friedgold on May 20 2005,02:53
For the username / pass the standard thing to do is use the username anonymous and the pass as a email address.For example if I use firefox to go to a ftp site it actually uses the username anonymous and the password mozilla@example.com. Similarly wget uses anonymous and @wget. For http servers I'm not too sure what kind of output you want. What does wget -S <URL> do that wget <URL> doesn't? For me -S doesn't seem to make a difference to the downloaded file, only to the infomation that is shown on std output (i.e. is shows the server responses) Posted by mikshaw on May 20 2005,03:09
That had crossed my mind, but only briefly. I'll try again with anonymous, but it doesn't seem to connect to http servers at all.Showing the server response is what i need, since the server gives you a list of files available in that directory. It's in html format, but thanks to Robert's mydslBrowser the script is able to parse the html easily enough. It's not downloading any files at this point, so wget <url> is not useable. The file is downloaded only after the user sees what is available and chooses a file. In order to show the user what is available I need a way to grab a list of files. And so we come back to the original problem. Posted by cbagger01 on May 21 2005,04:09
You should be able to do an "ls" from an ftp command script, just like any other ftp command.Then you redirect the output from stdout into a file and then parse the file. Posted by clacker on May 21 2005,12:31
cbagger01 is right. I made the following script and it worked:
I saved it as a file called test and then ran ftp -n < test >> filelist and it worked. You can parse the file names out with: awk '{print $9}' filelist You can't get a directory listing from an http connection as far as I know. If there is an index page, that gets shown if no specific page is used as the target. Posted by mikshaw on May 21 2005,12:43
If this method will not get a directory listing through http then i can't use it. Half the DSL mirrors are http. I guess i'll stick with gnu-utils for now, or maybe make a wget extension.Thanks for your testing and suggestions guys! |