[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #QZB-369254]: A catalog-friendly compressed data format?



Hi Brian,

A compressed format that can be used for both thredds catalogs and the
IDV is netcdf4. In fact, netcdf4 is one of the two distribution formats for
GOES-R products.

Cheers,

Sean

> Hello Unidata.
> 
> As one of the showcases of our new Unidata server, a scientist here
> wants to offer high resolution precipitation estimates
> (derived from satellite input sources).
> 
> These are huge datasets, but have a lot of zero values.
> That community uses compressed formats for that reason.
> But I would like it to be IDV-accessible.
> 
> Is there a compressed CDM format?
> Or is this what the "satellite image" formats are good for,
> and are quantitative data readily convertible to such formats?
> 
> Any advice welcomed.
> 
> Thanks,
> Brian Mapes
> 
> 
> Begin forwarded message:
> 
> From: Robert Joyce - NOAA Affiliate <address@hidden<mailto:address@hidden>>
> Date: November 22, 2013 3:07:48 PM GMT-05:00
> To: Brian Mapes <address@hidden<mailto:address@hidden>>
> Subject: Re: Collecting Datasets for the new Unidata server
> 
> Hi Brian;
> 
> In a week or two CPC will stage it's first gauge corrected CMORPH 
> precipitation, I think at least a year of that would be a good candidate.  I 
> need to find out from Pingping what the format will be, then I can get a 
> number on the disk space usage.  Unfortunately we store CMORPH online in 
> monthly tar files of bzipped binary files.
> 
> Bob
> 
> 
> On Sun, Nov 10, 2013 at 3:56 PM, Brian Mapes 
> <address@hidden<mailto:address@hidden>> wrote:
> Hello MPO big obs dataset users.
> 
> (please forward to others I forgot freely, I am reply-alling an old
> vis email, and adding a few folks randomly, incompletely).
> 
> 
> Our new Unidata server is here, getting set up this week, with
> many TB of space for an OpenDAP-accessible data repository.
> 
> 
> Overall, I want to really try to use this sytsem to unify analyses of
> 
> 1. Current weather data (1-few months rolling archive),
> (1.5 a Forecasts archive for verification and predictability work?)
> 2. Climate context (summary stats from historical archives).
> 3. Specialized obs datasets to put in all this context (DYNAMO, CAROB, ?).
> 
> Things that exist online elsewhere, we will add virtual links to.
> 
> Hosting model-generated simulation output, on non-obs-aligned
> model timelines, is not the main purpose, but ask me case by case.
> 
> 
> For 1., Unidata will soon help me set up the LDM for weather feeds,
> including some backlogged verbal requests from several of you.
> The interested should email me - let's meet, after we get an
> initial configuration in place, and brainstorm improvements from there.
> 
> A Webpage harvester can be easily set up for things not in the LDM
> (e.g. is there a TIGGE site we could grab some forecast data from?)
> 
> 
> For 2. I'm looking to build up a Web-served collection of
> climatology statistic files, for all the common obs/analyses sets.
> So I can make anomaly and standardized anomaly mask layers
> for weather display templates. For all our favorite datasets,
> I'll need
> 
> * mean climo (at monthly, daily-mean, 3hourly resolutions) to make anomalies
> * stdev climo (at ",",") for making standardized anomaly (Z-score, t-test) 
> overlays
> 
> ? (other moments, or extremal range for declaring "unprecedented" events,
> or ??, if you can dream it let's do it)
> 
> 
> 
> I don't mind a little redundancy with vis cluster datasets, since
> this will be Web-open, and vis is a walled garden where
> big computations need fast access... so,
> Vis users, can someone point me to what's available there?
> I could login and do ls commands but help is appreciated.
> 
> 
> 
> REANALYSIS AND SST:
> Some are already out there.
> 
> PSD has NCEP1 and 20Cv2, and I made .ncml variable aggregations,
> but I don't have the climatogical stdev files for normalizing anomalies.
> 
> SST sets are out on servers pretty well but again rarely the stdev 
> climatology.
> 
> MERRA is wonderfully aggregated on a GDS server
> (which can even be harnessed to do the climo calculations
> for us I think, if one of you knows GDS syntax).
> 
> CFSR is maddeningly non-aggregated in the NOMADS access.
> If any of you have reorganized sets of CFS data I'd be interested.
> 
> 
> 
> SATELLITE AND RADAR DIRECT OBS:
> For satellite texture, NOAA IR and COLA TRMM3B42 archives are good,
> but sometimes as much as a couple years behind current. Again no stdev climo.
> 
> Bob Joyce will make us some unique high-res satellite precip products. 
> (right?)
> 
> I'd be happy to set up a harvester from any realtime-data sites, if some of 
> you
> know such sites let me know. (for things not in the LDM stream). Brian McN,
> know any radar-imagery archives? (ash.mmm.ucar.edu<http://ash.mmm.ucar.edu/> 
> would be great to
> have digitally as an archive for IDV layering, but McIDAS type headers needed)
> 
> 
> 
> Other ideas welcome too.
> Please feel free to be ACTIVIST!
> I am busy too so things take awhile if left to me.
> 
> Please forward this freely, I'm replying to an old email
> plus a few add-ons, forgetting many I'm sure.
> 
> Brian
> 
> 
> 
> On Oct 15, 2010, at 10:54 AM, Pedro DiNezio wrote:
> 
> 
> i could help with CMIP3, though i'm across the street. Katinka could help me.
> 
> over these years i've been able to build a nice database with IPCC AR4 
> experiments, including some that are not at the CMIP3 database, such as CCSM3 
> and GFDL models and some other files that i've got directly from the 
> modelling centers.
> 
> my database is quite consistent. what i would do is copying my database from 
> AOML to the RSMAS machine on a disk. it's about 8 to 10 Tb. i have a 2 Tb 
> portable disk, does anybody have larger disk?
> 
> i also have a beautiful set of matlab functions that read the (CMIP3 
> CMORized) netcdf files into matrices. these functions take care of the 
> calendars and the fact that some variables could be located in more than one 
> netcdf file. all this stays under "the hood" for the user. i'm sure they will 
> work for CMIP5. i would like to pass them to RSMAS students before i leave.
> 
> are you including observational datasets, such as:
> 
> HadSLP
> HadSST
> EN3
> ERSST
> ERSLP
> 
> ?
> 
> 
> -------- Original Message --------
> Subject:        Fwd: data sets
> Date:   Thu, 14 Oct 2010 21:28:46 -0400
> From:   Amy Clement <address@hidden><mailto:address@hidden>
> To:     Pedro DiNezio <address@hidden><mailto:address@hidden>
> 
> 
> could you chime in on this?
> 
> -------- Original Message --------
> Subject:        data sets
> Date:   Thu, 14 Oct 2010 15:44:12 -0400
> From:   Brian Mapes <address@hidden><mailto:address@hidden>
> To:     address@hidden<mailto:address@hidden>
> CC:     Ben Kirtman <address@hidden><mailto:address@hidden>, shuyi chen 
> <address@hidden><mailto:address@hidden>, amy clement 
> <address@hidden><mailto:address@hidden>, Robert Burgman 
> <address@hidden><mailto:address@hidden>, chidong Zhang 
> <address@hidden><mailto:address@hidden>
> 
> 
> 
> Happy to coordinate.
> 
> Trimming the less likely suspects off the email traffic...
> 
> can I ask for an ls of what you all already have in the house?
> 
> May I (should I) make copies onto the new machine?
> Instead of starting a cold search and download task by Web.
> 
> Brian
> 
> 
> On Oct 14, 2010, at 3:23 PM, address@hidden<mailto:address@hidden> wrote:
> 
> > Great idea. Maybe individuals or groups could volunteer to be
> > sponsors of
> > particular data sets and take responsibility for archiving/updating
> > them?
> >
> >> Can we coordinate some baseline datasets on there?
> >>
> >> I bet some of you have much of this data lying around on
> >> existing machines I'm only dimly aware of.
> >>
> >>
> >> Suggestions
> >>
> >> Top level:
> >> monthlies/
> >> dailies/
> >> subdailies/
> >>
> >> Analyses:
> >> ERA40/
> >> ERAInt/
> >> NCEP/
> >> NCEP2/
> >> CFSRR/
> >> MERRA/
> >> (others?)
> >>
> >> Satellite & products:
> >> SSMI/
> >> TRMM/
> >> CMAP/
> >> OLR/
> >> CLAUS/
> >> ISCCP/
> >> ??more?
> >>
> >> Recent baseline model climatologies:
> >> AM2/
> >> CM2/
> >> CAM4/
> >> CAM5/
> >> CCSM3/
> >> CCSM4/
> >> CFS/
> >>
> >> Model archives:
> >> CMIP3//
> >> CMIP5/
> >>
> >> On Oct 13, 2010, at 1:59 PM, Ben Kirtman wrote:
> >>
> >>> Dear Co-Conspirators,
> >>>
> >>> The rsmas vis host is ready for business.  The access is
> >>> vis1.ccs.miami.edu<http://vis1.ccs.miami.edu/> and you login with your 
> >>> usual CCS userid and
> >>> password. Your data disk is /userid and group access is your
> >>> individual default group.  If folks want to allow more sharing we
> >>> can use other groups, if they exist,  or they can put in a request
> >>> for a new group by logging in to their account at
> >>> http://iweb.ccs.miami.edu:8080/account/login.htm
> >>>
> >>> Applications  are installed in /usr/local  or /usr/local/apps.
> >>>
> >>> If anyone needs assistance migrating data or has any questions or
> >>> problems please just let us know (Warner).
> >>>
> >>> I tried logging in and using matlab and grads and they worked fine,
> >>> although you will need to make sure the application you are
> >>> interested is in your path.
> >>>
> >>> Many thanks to the CCS team for getting this up and running!
> >>>
> >>> -Ben
> >>>
> >>>
> >>> --
> >>> Ben Kirtman
> >>> Division of Meteorology and Physical Oceanography
> >>> Rosenstiel School for Marine and Atmospheric Science
> >>>
> >>> and
> >>>
> >>> The Center for Computational Science
> >>> University of Miami
> >>>
> >>> 4600 Rickenbacker Causeway
> >>> Miami Fl 33149
> >>> Phone: 305-421-4046<tel:305-421-4046>
> >>> address@hidden<mailto:address@hidden>
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden> or address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 

Ticket Details
===================
Ticket ID: QZB-369254
Department: Support IDV
Priority: High
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.