Hi Terry,
I think that your suggestion is implementable, I’m just trying to work through
the potential issues so that a means can be found to work for the maximum
number of users without causing surprising behaviour or the need for
special-case coding.
I think the remaining issues would be:
1. Choice of the attribute names for the suggested scale range. Does anyone
else add this information to NetCDF files and are there any commonly-used
attribute names? I’ll ask the CF community this question.
2. Behaviour with packed variables. I would assume that the most useful thing
would be for the view range to be encoded with respect to unpacked variable
values. (Valid_min/max can be encoded with respect to either packed or unpacked
values.)
So I think we should probably do this, but it would be useful to have some
input from other users as they may raise things that we haven’t thought of.
Cheers,
Jon
--
Dr Jon Blower,
Technical Director, Reading e-Science Centre,
MELODIES project coordinator,
School of Mathematical and Physical Sciences,
University of Reading
Tel: +44 118 378 5213
Email: j.d.blower@xxxxxxxxxxxxx
On 22 Aug 2014, at 03:29, Terry.Rankine@xxxxxxxx wrote:
> No we don't use aggregations.
>
> I am an Australian, and all I really want: is to do less work!
> The ability for a server to do some of that work for me, is awesome! Your
> thoughts and help is appreciated.
>
> I am not suggesting that we change THREDDS defaults, I am suggesting that we
>
> have the option to allow an admin to delegate a default/preference to the
> data
> owner.
>
> The 'pseudo code' would allow arbitrary delegation using a selector to pull
> the data out of the netcdf header.
>> <default color range>
>> <min>$delegate[elevation.someVar]</min>
>> <max>$delegate[elevation.otherVar]</max>
>> </default color range>
> As an admin of multiple mixed systems, I would be free-ed, where
> appropriate,
> to allow the 'view range' to come from a dataset.
>
> Just to give you an example, we sent a boat out to do some bathymetry scans,
>
> they dumped 3 netcdf's in the 'please host me folder'.
> Three separate runs were done, our variable is 'elevation' in this context.
> The near shore one was (-200 -20), and the two offshore where (~-6500,-50),
> (-2000, -50).
>
> Are you suggesting that:
> * Because others use aggregations - you won't consider the ability
> delegate and to read from the file?
> * Or are you saying that actual_min/max sound good - lets hard code them
> in?
>
> I think neither of those are wonderful conclusions. I see value in the
> delegation pattern, with a 'configured' selector, today they can be read
> from a header (simple), tomorrow they could be an extra function like Mike
> McCann <mccann@xxxxxxxxx> uses manually now:
> So the default min (max would be the same) could end up looking like:
> * $select[file.layer.somvar]
> * $calculate[$math.percentile(file.layer, 2.75)]
> With the obvious wording - you can make assumptions on 'performance'
>
> Once again - "I am just lazy" and looking for a better integration between
> data-owner, and administrator.
>
> So for now - I will look at capturing the min/max (and hacking in a
> percentile variable too - good idea Mike), and educating my data providers
> to do the same from now on.
> What Thredds does with the data - well that's now up to you guys!
>
> Regards
> Terry
> "That lazy architect guy"
>
>
> -----Original Message-----
> From: Jon Blower [mailto:j.d.blower@xxxxxxxxxxxxx]
> Sent: Thursday, 21 August 2014 4:37 PM
> To: Rankine, Terry (Mineral Resources, Kensington)
> Cc: thredds@xxxxxxxxxxxxxxxx
> Subject: Re: NetCDF, WMS, and auto color scale (not set by admin)
>
> Hi Terry,
>
> Do you serve aggregations, or is each NetCDF file a separate dataset? With
> an
> aggregation, there isn't a one-to-one correspondence between a WMS layer and
> a
> NetCDF variable in a particular file. This is also true for things like
> velocity fields, which are derived on-the-fly from vector components. One of
>
> the THREDDS team could hopefully explain how variable attributes behave
> under
> aggregation (do they just take the values from the first file in the
> aggregation?)
>
> One problem with "baking" this information into the NetCDF file, is that
> people frequently change their mind about what the values should be. The
> concept of a "sensible view range" is not an intrinsic property of the data,
>
> hence we have always kept this as a WMS configuration issue.
>
> By the way, valid_min/max isn't the solution I'm afraid. These attributes
> define a range, outside of which data are considered to be missing. So for
> an
> ocean temperature field (with a sensible view range of [-2:35] degC perhaps)
>
> the valid_min/max could perfectly legitimately be set to [-100:100].
>
> There was a proposal (which I don't think is official CF) to use
> "actual_min/max" attributes to denote the actual range of data in a
> variable.
> There was reluctance to accept this into CF because it is redundant
> information, but I think it can be easily calculated and inserted into a
> NetCDF file using the NetCDF Operators (nco).
>
> However, if you think that something like actual_min/max would work for your
>
> use case, we could consider whether this would be implementable as a means
> of
> providing the "sensible view range".
>
> Cheers,
> Jon
>
> --
> Dr Jon Blower,
> Technical Director, Reading e-Science Centre, MELODIES project coordinator,
> School of Mathematical and Physical Sciences, University of Reading
> Tel: +44 118 378 5213
> Email: j.d.blower@xxxxxxxxxxxxx
>
> On 21 Aug 2014, at 03:45, <Terry.Rankine@xxxxxxxx> <Terry.Rankine@xxxxxxxx>
> wrote:
>
>> The two workflows we have are
>> 1. Dump the data in "this" folder heriachy/structure - it will
>> magically appear for you.
>> 2. Email us and we will host it.
>>
>> I cant see any issue with scripting it, but I was under the impression
>> that changes to the xml file would require TDS to be restarted, plus
>> its just another bit of data to keep in 'sync' somehow.
>>
>> My desire here would be that, as an admin, I would tell the
>> wmsConfig.xml <location /this/path/his-data-set-*.nc>
>> <layer name=elevation>
>> <default color range>
>> <min>$delegate[elevation.vmin]</min>
>> <max>$delegate[elevation.vmax]</max>
>> </default color range>
>> </layer>
>> </location>
>>
>> As an admin I would deliberately hand over the 'setting the right view
>> stuff' for those who could control the data production/attribution.
>> Freeing me from that role.
>>
>> I completely understand where it is not appropriate - but I don't
>> believe that makes it 'inappropriate' for the simple cases.
>>
>> Terry
>>
>> -----Original Message-----
>> From: thredds-bounces@xxxxxxxxxxxxxxxx
>> [mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jon Blower
>> Sent: Thursday, 21 August 2014 2:48 AM
>> To: thredds@xxxxxxxxxxxxxxxx
>> Subject: Re: [thredds] NetCDF, WMS, and auto color scale (not set by
>> admin)
>>
>> Hi Terry,
>>
>> As far as I remember THREDDS has always required setting of the
>> default colour scale range by the admin. (However the standalone ncWMS
>> could do this automatically, because it scans datasets during an
>> ingestion step, which THREDDS does not perform.)
>>
>> There is currently no way to read the default colour range from the
>> NetCDF files, although a means could theoretically be invented, with
>> "special"
>> attributes. One problem is that, in a multidimensional variable, the
>> most appropriate range for one z or t level may not be so appropriate
>> for another value of z or t. There are other interesting issues with
>> the behaviour of NetCDF attributes in aggregations (what happens if
>> different files in the same aggregation have different attributes?)
>>
>> How do the data providers communicate their preferences to you at the
>> moment? Could you, for example, craft a script that takes their values
>> (from a simple text file maybe) and inserts it appropriately into the
>> wmsConfig.xml automatically?
>>
>> Cheers,
>> Jon
>>
>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> -
>>>
>>> Message: 1
>>> Date: Wed, 20 Aug 2014 01:19:29 +0000
>>> From: <Terry.Rankine@xxxxxxxx>
>>> To: <thredds@xxxxxxxxxxxxxxxx>
>>> Subject: [thredds] NetCDF, WMS, and auto color scale (not set by
>>> admin)
>>> Message-ID:
>>>
>> <918EBF4F0E40D340B213B473CCAB6B126000492D@xxxxxxxxxxxxxxxxxxxxxxxxxx>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> Hi Guys
>>>
>>>
>>>
>>> In older versions of thredds I could leave the auto color scaling for
>>> a WMS to thredds to take care of for me. Can I have it back - but a
>>> bit
>> different.
>>>
>>>
>>>
>>> As a system admin - I am just 'serving the data' not really
>>> understanding what 'sensible viewing defaults' for
>>> defaultColorScaleRange for any file in the directory should be.
>>> However - the data owners I interact with all have a 'preference' on
>>> how to color/scale their visualisations for their data files.
>>>
>>>
>>>
>>> Is there a way to make thredds read the netCDF file - load the
>>> variable, and load the visualisation range FROM THE DATA FILE - not
>>> the wmsConfig.xml - where I would need to explicitly set it?
>>>
>>>
>>>
>>> IE - if on the elevation layer for this service:
>>>
>>> http://vhirl-dev.csiro.au/thredds/dodsC/dem/testconvert/bathymetry_su
>>> r
>>> veys_G
>>> A-2476_100m_version1_WGS84UTM49S_esri_grid_grid100m_GA-2476_Exmouth_1
>>> 0
>>> 0m_exm
>>> outh100m.nc4.html
>>>
>>>
>>>
>>> I set a netcdf parameter like defaultColorScaleRange or something
>>> less thredds specific, max/min or something which could guide the 'lazy'
>>> configuration option for the data?
>>>
>>>
>>>
>>> Regards
>>>
>>> Terry
>>>
>>>
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
