[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #YYI-682303]: [Datastream #QGV-275164 ] Downloading in bulk

Hi Taleena,

re:
> I am going to be downloading my files from
> 
> http://idd.ssec.wisc.edu/uniwisc/urequest/
> 
> and was wondering if there is a script or special way to download the
> data all at once.

The best way to transfer multiple files is to use utilities like 'wget' or 
'curl'.

re:
> I have tried to ssh into the server and there's a connection timeout.

We only provide a web-based access to the files in the Unidata-Wisconsin
archive (i.e., we do not allow SSH logins or run an FTP server due to
security implications).

Here is an example 'wget' invocation that should get all of the files
in a the 19910415 subdirectory of urequest:

wget --user=unidata3 --password=unirec7 -r -nd -e robots=off 
http://idd.ssec.wisc.edu/uniwisc/urequest/19910415

We tested this 'wget' example on a Linux machine as follows:

cd /tmp
mkdir scratch
cd scratch
wget --user=unidata3 --password=unirec7 -r -e robots=off -R index.html 
http://idd.ssec.wisc.edu/uniwisc/urequest/19910415

This got all of the image files (and a few extraneous ones that will have to be 
deleted)
into a directory hierarchy that looks like:

idd.ssec.wisc.edu/uniwisc/urequest/19910415

You could then run the command for each day of data you want, or you could 
create
a shell script that contained the set of 'wget' invocations and get them all.


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: YYI-682303
Department: Support Datastream
Priority: Normal
Status: Closed