Bryan,
So this *should* be as easy as:
from siphon.catalog import TDSCatalog
cat = TDSCatalog('
http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_onedeg/catalog.xml
')
cat.latest.download()
However, because I was foolish, if you try to do this with a big file (i.e.
more than your memory), you will run out of memory because it tries to
download all the bytes first and then writes them to the file. As a
workaround, you can use this function (from here:
https://stackoverflow.com/questions/16694907/download-large-file-in-python-with-requests
):
import requests
import shutil
def download_file(url):
local_filename = url.split('/')[-1]
with requests.get(url, stream=True) as r:
with open(local_filename, 'wb') as f:
shutil.copyfileobj(r.raw, f)
return local_filename
and do:
download_file(cat.latest.access_urls['HTTPServer'])
This should be fixed in the upcoming Siphon 0.9 release.
Hope this helps,
Ryan
On Wed, Oct 9, 2019 at 3:02 PM Bryan Guarente <guarente@xxxxxxxx> wrote:
> Ryan,
> I am looking for a way to grab all of today's model data as grb files. We
> had talked about grabbing the "latest" catalog and checking it every time a
> new model updates to download more data. I thought siphon or even trying
> python-awips would work to get some of this done. Pointing to an EDEX
> server seemed smart to me, but I am sure you have different ideas.
>
> Thanks.
> Bryan
> --
> Bryan Guarente
> Meteorologist/Instructional Designer
> The COMET Program
> Boulder, CO
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> python-users mailing list
> python-users@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, or change your membership options,
> visit: https://www.unidata.ucar.edu/mailing_lists/
>
--
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO
