Unidata Developer's BlogUnidata Developer's Bloghttps://www.unidata.ucar.edu/blogs/developer/en/feed/entries/atom2024-03-05T10:00:34-07:00Apache Rollerhttps://www.unidata.ucar.edu/blogs/developer/entry/spring-2023-sbn-ground-stationSpring 2023 SBN Ground Station Repointing GuideUnidata News2023-01-30T14:01:39-07:002023-02-24T14:43:34-07:00<p>
On January 24, 2023, NOAA released Service Change Notice 22-77, describing the
change of relay satellite for all SBN services (NOAAPort and NWWS) from Intelsat’s
Galaxy 28 to the newly launched Galaxy 31. For ground station operators, this move
is non-trivial in that the location of Galaxy 28 is geostationary over 89°W, and
Galaxy 31 is geostationary over 121°W.
</p>
<p>
The following guide is intended to help operators who are not familiar with repointing a
satellite dish to provide the information to a professional satellite technician.
</p>
<p class="byline">
By Stonie Cooper, Unidata
</p>
<p>
On January 24, 2023, NOAA released Service Change Notice 22-77, describing the
change of relay satellite for all SBN services (NOAAPort and NWWS) from Intelsat’s
Galaxy 28 to the newly launched Galaxy 31. For ground station operators, this move
is non-trivial in that the location of Galaxy 28 is geostationary over 89°W, and
Galaxy 31 is geostationary over 121°W.
</p>
<p>
The current transition schedule requires ground stations to switch to the new
satellite on a relatively tight schedule:
</p>
<ul>
<li>5 Feb 2023: Galaxy 31 illuminated</li>
<li>10 Feb 2023: Ground Station Transition Begin</li>
<li>31 Mar 2023: Ground Station Transition End</li>
<li>3 Apr 2023: Galaxy 28 decommissioned</li>
</ul>
<p>
This guide is intended to help operators who are not familiar with repointing a
satellite dish to provide the information to a professional satellite technician.
</p>
<p> <p>
<strong>Note: </strong>This guide was prepared for SBN ground station operators to
be able to provide information to their professional satellite technician to
repoint a satellite reflector for SBN reception. UCAR is in no way responsible for
injuries, damages, loss of data, loss of income, loss of life, or other issues
that may arise from the use of the information herein. Lightning, wind, heavy
equipment, heights, and all other dangers associated with construction, satellite
communications, and satellite reflector maintenance are a serious threat to life
and potential injury. UCAR in no way indorses anyone not trained and experienced
with to perform any of the tasks outlined in this document, rather, this document
is meant to act as the starting point for a Statement of Work for an operator to
hire a professional to perform such work.
</p></p>
<h3>Assumptions</h3>
<p>Assumptions made in this guide include the following:</p>
<ul>
<li>The reflector is attached to the mount in an “Az/El” mount paradigm</li>
<li>The mount is anchored without the use of a motorized actuator</li>
<li>The reflector is pole-mounted</li>
<li>The Novra S300N has been configured for the new satellite
(see <a href="#novra">Novra S300N configuration</a>, below).</li>
</ul>
<h3>Example</h3>
<div class="img_r" style="width: 200px;">
<a class="lightbox" title="Figure 1: UCAR SBN ground station reflector." href="/blog_content/images/2023/20230130_sbn_fig01.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig01.png" alt="UCAR SBN reflector" />
</a>
<div class="caption">
Figure 1: UCAR SBN ground station reflector.<br>(Click to enlarge.)
</div>
</div>
<p>
In this example, we step through the process of repointing the current SBN reflector
at UCAR's Foothills Laboratory in Boulder, Colorado (Figure 1).
</p>
<p>
Using a GPS, cell phone, or Google Maps, determine the latitude and longitude of the
satellite reflector that will be repointed.
</p>
<p>
For our example, the latitude is found to be 40.03721°N and the longitude is -105.24192°E.
</p>
<p>
Searching the Internet will yield several on-line satellite pointing calculators.
For this example, we use the <a href="https://www.satsig.net/ssazelm.htm">Satellite
Signals home page</a>, which can calculate results for all aspects of the repointing.
</p>
<p>
Providing our latitude and longitude as inputs, and using the current satellite,
Galaxy 28 at 89°W, as our control, the current look angle and polarity tilt will be
used to make relative changes.
</p>
<pre>
<strong>Guide Location:</strong>
40.03721°N latitude.
-105.24192°E longitude.
<strong>Galaxy 28:</strong>
89°W longitude (geostationary).
Current Azimuth:
155.64° from true North.
150.19° from magnetic North.
Current Elevation:
40.73° above the horizon (or “level”).
Current Polarization tilt at peak signal:
-18.41° from LNB housing horizontal (clockwise) if viewed from the front.
<strong>Galaxy 31:</strong>
121°W longitude (geostationary).
Current Azimuth:
203.68° from true North.
198.24° from magnetic North.
Current Elevation:
40.9° above the horizon (or “level”).
Current Polarization tilt at peak signal:
17.91° from LNB housing horizontal (counterclockwise) if viewed from the front.
</pre>
<p>
<strong>Note:</strong> For determining the change in azimuth, it does not matter
if you use true North or magnetic North, but pick one and always use the North
reference selected.
</p>
<p>
In this example, the repositioning data above require the following changes:
</p>
<p> <pre>
Azimuth delta: <strong>48.04° to the west</strong>
Elevation delta: <strong>rise 0.17° above horizon</strong>
Polarization tilt delta: <strong>36.32°</strong>
(counterclockwise, viewed from facing the reflector)
</pre></p>
<p>
If there is more than 1° change in elevation, it is best to start by changing the
elevation first. Select a section of the mount or part of the dish that is flat,
and using a magnetic declinometer, note the starting angle before loosening any
hardware.
</p>
<div class="img_r" style="width: 200px;">
<a class="lightbox" title="Figure 2: Protractor set for elevation adjustment." href="/blog_content/images/2023/20230130_sbn_fig02.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig02.png" alt="Elevation adjustment" />
</a>
<div class="caption">
Figure 2: Protractor set for elevation adjustment.
</div>
</div>
<p>
On most pole mounted reflectors, the elevation is changed by loosening and rotating
nuts on an elevation rod. It may be necessary to loosen bolts and nuts very
slightly at pivot points, but only enough so the dish will move. It is important to
loosen the nut in the direction of the adjustment, first. In the case of raising
the dish, loosen the bottom nut, and then tighten the top nut to follow the bottom
nut. The goal is to change the elevation by the delta as calculated from the
current position and the position needed to point to Galaxy 31.
</p>
<div class="img_l" style="width: 200px;">
<a class="lightbox" title="Figure 3: Elevation adjustment key points." href="/blog_content/images/2023/20230130_sbn_fig03.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig03.png" alt="Elevation adjustment key points" />
</a>
<div class="caption">
Figure 3: Elevation adjustment key points.
</div>
</div>
<p>
Conversely, lowering the pointing angle entails loosening the top nut, and then
turning the bottom nut to follow the top nut. Once you have changed the angle as
seen on the protractor by the desired delta, snug, but not tighten, the adjustment
nuts and any loose pivot points.
</p>
<p>
Pole mounted reflectors normally adjust azimuth by rotating the entire
dish on the pole after loosening the collar bolts. Before committing to the azimuth
change, take the time to make witness marks of the current reflector position (figure 4).
</p>
<div class="img_r" style="width: 200px;">
<a class="lightbox" title="Figure 4: Make initial witness marks." href="/blog_content/images/2023/20230130_sbn_fig04.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig04.png" alt="Azimuth witness marks" />
</a>
<div class="caption">
Figure 4: Make initial witness marks.
</div>
</div>
<p>
Next, measure the circumference of the pole (not the collar). Most pole mounted
collars are 12” or 18” long, although the Comtech 3-piece reflector has a collar
that is 48” long. Regardless, the circumference needed is of the pole that the
collar sits on, which, on the Comtech, will be close to the ground.
</p>
<div class="img_l" style="width: 200px;">
<a class="lightbox" title="Figure 5: Measure pole circumference." href="/blog_content/images/2023/20230130_sbn_fig05.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig05.png" alt="Measure pole circumference" />
</a>
<div class="caption">
Figure 5: Measure pole circumference.
</div>
</div>
<p>
In this example, the pole has a circumference of 536mm. Knowing the angle of the
direction change needed to repoint to Galaxy 31, the arc distance change on the
pole can be calculated:
</p>
<p>
<img src="/blog_content/images/2023/20230130_sbn_eq.png">
</p>
<p>
Rotating the collar to repoint to Galaxy 31, west of the current satellite,
results in an arc distant change of 71.53mm.
</p>
<div class="img_l" style="width: 200px;">
<a class="lightbox" title="Figure 6: Measure to the new location." href="/blog_content/images/2023/20230130_sbn_fig06.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig06.png" alt="Measure new location" />
</a>
<div class="caption">
Figure 6: Measure to the new location.
</div>
</div>
<p>
Final azimuth preparation entails scribing a second witness mark on the pole,
to which the collar (and thus whole dish) will be rotated to acquire Galaxy 31.
</p>
<div class="img_r" style="width: 200px;">
<a class="lightbox" title="Figure 7: Apply target witness mark." href="/blog_content/images/2023/20230130_sbn_fig07.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig07.png" alt="Target witness mark" />
</a>
<div class="caption">
Figure 7: Apply target witness mark.
</div>
</div>
<p>
Mount collars will have three or four sets of bolt pairs equally spaced around the
collar that secure the dish at the correct azimuth. In this example, there are three sets of two
bolts. We suggested that only one set of bolts be loosened, and only loosened
enough to allow the dish to rotate, but not rotate freely.<br><br>
</p>
<div class="img_l" style="width: 200px;">
<a class="lightbox" title="Figure 8: Bolts for adjusting azimuth." href="/blog_content/images/2023/20230130_sbn_fig08.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig08.png" alt="Azimuth adjustment" />
</a>
<div class="caption">
Figure 8: Bolts for adjusting azimuth.
</div>
</div>
<p>
With assistance, push the reflector in the direction that will bring the collar mark
in line with the 121°W pole witness mark. Snug the bolt only slightly. With
assistance, use Novra monitoring software tool (see <a href="#novra">Novra S300N
configuration</a> below for details), and note the Carrier to Noise ratio (C/N).
Very slightly, push the reflector to the left (as viewed from behind the dish), and
note changes to the C/N. If the C/N decreases, push the reflector back to the right
in small increments. Allow the Novra statistics to sample over 10 seconds at a
minimum with each slight move before pushing the reflector again.
</p>
<p>
Continue moving the reflector in small motions, noting an increase/decrease in the
C/N, with the goal to move the reflector to achieve the highest C/N. Once the
highest C/N has been located, make witness marks on the collar and pole, separate
from other witness marks, as the final azimuth position for Galaxy 31.
</p>
<div class="img_r" style="width: 200px;">
<a class="lightbox" title="Figure 9: Nudge reflector to the right before tightening." href="/blog_content/images/2023/20230130_sbn_fig09.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig09.png" alt="Tightening tip" />
</a>
<div class="caption">
Figure 9: Nudge reflector to the right before tightening.
</div>
</div>
<p>
<strong>Stonie’s Tip:</strong> On pole mounted reflectors, tightening the collar
bolts to lock down the reflector will result in the reflector moving from the
intended “peaked” position. This is due to the bolts being tightened “biting” into
the pole and walking the collar away from the intended location. To overcome this
potentially frustrating aspect, once you achieve the peak C/N from the Novra
monitoring tool, push the reflector slightly to the right, viewed from behind the
reflector, before starting the tightening process. When the bolts are tightened,
they will bite and walk the collar to the left, and with practice, will put the
collar back to the optimal witness mark.
</p>
<p>
Once all collar bolts are confirmed to be tight, return to the elevation rod peaking
process. Lower the elevation angle very slightly, and again, note the C/N reading
from the Novra monitoring tool. Adjust the nuts on the elevation rod until the
highest C/N reading is achieved. At that point, tighten all nuts and bolts, both on
the elevator rod itself, as well as all pivot points. This concludes the base
reflector repointing and attention will now turn to the polarization tilt.
</p>
<p>
Not all reflector feedhorn/scalar ring assemblies allow for adjustment in polarity
other than horizontal to vertical, but in discrete relation to the reflector.
However, for those scalar ring/feedhorn assemblies that allow adjustment, additional
peaking should be considered.
</p>
<div class="img_l" style="width: 200px;">
<a class="lightbox" title="Figure 10: Measuring initial polarization tilt angle." href="/blog_content/images/2023/20230130_sbn_fig10.png">
<img width="200" src="/blog_content/images/2023/20230130_sbn_fig10.png" alt="Polarization tilt angle" />
</a>
<div class="caption">
Figure 10: Measuring initial polarization tilt angle.
</div>
</div>
<p>
As done previously, it is best to start with a reference of what the LNB’s
polarization tilt angle is initially. Use the bolt heads on the LNB as stops for
the edge of the protractor to insure an accurate reading of the angle.
</p>
<p>
Loosen the set screw(s) for the feedhorn and rotate counterclockwise (viewing from
within the reflector) the delta angle calculated earlier. For Boulder, this was
36.32° in total from the initial position. Snug the set screw(s), and with
assistance, record the C/N reading from the Novra monitoring tool. Note that while
a person is standing or in front of the reflector, the C/N will be lower than it was
when the azimuth and elevation were adjusted. Maintain the same physical location
of the operator adjusting the polarity until final peak.
</p>
<p>
Slightly rotate the feed to the right, note the increase or decrease in the C/N. If
increasing, continue to rotate, incrementally, in that direction until the C/N
decreases. Likewise, if the C/N decreases, go the opposite direction. Once the C/N
has reached the highest level, secure the set screw(s) <em>without overtightening</em>.
</p>
<p>
On aftermarket feedhorns, it is also possible to adjust the focal length of the feed
by sliding it forward or out of the scalar ring with the set screw(s) only snug (but
loose enough to allow feedhorn movement). This process is the same as the other
peaking processes and can help attain a further gain in signal strength or decrease
in noise.
</p>
<p>
Once satisfied that the peak C/N has been obtained, confirm the coaxial connections
are watertight, all items loosened during the repointing are retightened, and
replace zip ties with fresh zip ties or Velcro to prevent movement of cables on or
around the dish. Wind induced movement in cables produces inadvertent RF noise, and
cable hygiene goes a long way to further reduce noise sources.
</p>
<h3 id="novra">Novra S300N configuration</h3>
<p>
The following information for configuring the Novra S300N satellite receiver has
been provided by NOAA.
</p>
<table class="simple">
<tr>
<td style="vertical-align:top;">
<pre>
NOAAPort ingest:
-- Symbol Rate: 30 Msps
-- RF Frequency: 1130 MHz
-- PID(s): 101
102
103
104
105
106
107
108
150
151
-- FEC Type: DVB-S2
-- Modulation/Coding: 16PSK 2/3
-- ISI: 18
</pre>
</td>
<td style="vertical-align:top;">
<pre>
NWWS ingest:
-- Symbol Rate: 30 Msps
-- RF Frequency: 1130 MHz
-- PID(s): 201
-- FEC Type: DVB-S2
-- Modulation/Coding: QPSK 1/3
-- ISI: 2
</pre>
</td>
</tr>
</table>
<p>
One method to reconfigure the Novra is to save off the current Novra configuration,
edit the resulting xml file with the items that need changed, and then reload.
The Novra supplied Linux “cmcs” utility can be downloaded here:
</p>
<p>
<a href="https://novra.com/downloads?type=All&search=&nid=&_escaped_fragment_=&page=8">https://novra.com/downloads?type=All&search=&nid=&_escaped_fragment_=&page=8</a>
</p>
<p>
Using the "cmcs" utility:
</p>
<pre>
$ <strong>cmcs -list</strong>
S300N IP address: 192.168.0.1 MAC: jj-bb-aa-zz-yy-xx
$ <strong>cmcs -ip 192.168.0.1 -pw <password> -save mynovra.xml</strong>
</pre>
<p>
Edit the resulting <code>mynovra.xml</code> file for RF Frequency:
</p>
<pre>
<S300N_CONFIG>
<NETWORK ReceiverIP="192.168.0.1" SubnetMask="255.255.255.0" DefaultGateway="192.168.0.254" StatusDestinationIP="255.255.255.255" StatusDestinationPort="6516" IGMPFilter="OFF" />
<SATELLITE>
<SIGNAL RF="<span style="background-color:yellow;">1130</span>" SymbolRate="30000" AutoSR="OFF" GoldCode="0" SearchType="DVBS2" MODCOD="2/3 16APSK" ISI="18" />
<LNB>
<LNBSpec LOFrequency="0" PolaritySwitchingVoltages="13 &amp; 18" HiLowBandTone="22" />
<LNBControl Power="ON" Polarization="Horizontal/Left" Band="High" LongLineCompensation="ON" />
</LNB>
<CARRIER_TO_LO>Carrier Freq. &gt;= L.O. Freq.</CARRIER_TO_LO>
</SATELLITE>
<CONTENT>
<TRANSPORT_STREAM PIDS="Selected">
<PID Number="101" Processing="MPE" />
<PID Number="102" Processing="MPE" />
<PID Number="103" Processing="MPE" />
<PID Number="104" Processing="MPE" />
<PID Number="105" Processing="MPE" />
<PID Number="106" Processing="MPE" />
<PID Number="107" Processing="MPE" />
<PID Number="108" Processing="MPE" />
<PID Number="150" Processing="MPE" />
<PID Number="151" Processing="MPE" />
<PID Number="NULL" Processing="RAW" />
</TRANSPORT_STREAM>
<IP_REMAP_TABLE Enabled="false" RemapSourceIP="false" />
</CONTENT>
</S300N_CONFIG>
</pre>
<p>
Then reload the configuration:
</p>
<pre>
$ <strong>cmcs -ip 192.168.0.1 -pw <password> -load mynovra.xml</strong>
</pre>
<p>
Note that since the only parameter being changed here is the signal RF frequency, you
could skip editing the configuration XML file by hand and use the cmcs command to
change the parameter directly:
</p>
<pre>$ <strong>cmcs -ip 192.168.0.1 -pw <password> -rfreq 1130</strong></pre>
<p>(The method you choose to change this value is up to you. There is no need
to <em>both</em> edit the XML file and use the command with the <code>-rfreq</code>
parameter.)
</p>
<p>
Finally, to monitor the signal strength, use the cmcs command:
</p>
<pre>
$ <strong>cmcs -ip 192.168.0.1 -pw <password> -shsat</strong>
Satellite Interface Settings:
Receiver MAC Address: jj-bb-aa-zz-yy-xx
Receiver Mode: DVBS2
Frequency: 1110.0 MHz
Symbol Rate: 30.000 Msps
ModCod: 2/3 16APSK
Gold code: 0
Input Stream Filter: On
Input Stream ID: 18
Signal Lock: On
Data Lock: On
Uncorrectable Rate: 0/Second
Packet Error Rate: 0.0000e+00
Carrier to Noise C/N: 15.2dB
Signal Strength: -53 dBm
</pre>
<p>
If you have questions about this example satellite receiver repointing process, please
contact <a href="mailto:support@unidata.ucar.edu">support@unidata.ucar.edu</a>.
</p>
https://www.unidata.ucar.edu/blogs/developer/entry/contributor-license-agreement-for-unidataContributor License Agreement for Unidata ProjectsRyan May2017-09-28T15:18:03-06:002017-09-28T15:18:03-06:00<p>Unidata hosts a variety of Open Source software projects on GitHub. We use the Open Source model because we believe strongly that broad participation in all aspects of Unidata's work is essential to achieving the Unidata community's goals. Developing software that focuses on community needs is one of our main objectives, and participation by community members in all aspects of the development process — from coding to testing, documenting, and commenting — is incredibly valuable.</p>
<p>As community participation in Unidata's Open Source efforts grows, we are facing increasingly complex situations surrounding contributions made to Unidata-hosted projects. As a result, we have decided to begin requiring that community members who wish to contribute code to Unidata projects on GitHub agree to the Unidata Contributor License Agreement (CLA).</p>
<p>Unidata hosts a variety of Open Source software projects on GitHub. We use the Open Source model because we believe strongly that broad participation in all aspects of Unidata's work is essential to achieving the Unidata community's goals. Developing software that focuses on community needs is one of our main objectives, and participation by community members in all aspects of the development process — from coding to testing, documenting, and commenting — is incredibly valuable.</p>
<p>As community participation in Unidata's Open Source efforts grows, we are facing increasingly complex situations surrounding contributions made to Unidata-hosted projects. As a result, we have decided to begin requiring that community members who wish to contribute code to Unidata projects on GitHub agree to the <a href="https://gist.github.com/dopplershift/488c982201749423a52ca193f31a9515">Unidata Contributor License Agreement (CLA)</a>. This agreement is based on a template from the <a href="http://harmonyagreements.org">Harmony Agreements</a> project, whose goal is to standardize CLA's within the Open Source community.</p>
<h2>What is a Contributor License Agreement?</h2>
<p>A Contributor License Agreement (CLA) defines the terms under which intellectual property is contributed to a project. In Unidata's case, the CLA covers software code contributed to Unidata-hosted projects distributed under Open Source licenses. While the Unidata CLA is a legal document, and you should read it carefully before agreeing to it, the two main ideas are:</p>
<ul>
<li>You retain the ownership of the copyright to your contribution</li>
<li>You grant Unidata the right to use your contribution in perpetuity</li>
</ul>
<p>We feel it is important to use a CLA on Unidata-hosted projects for several reasons. Most important among them is the idea that once a Unidata software product has been released to users, the users can be confident that they have the right to continue using the software. Adding some formality to the software contribution process helps the Unidata community make well-informed choices about the benefits of using Open Source software.</p>
<p>The CLA process Unidata is adopting is becoming more and more common in the Open Source community. We don't think that asking contributors to sign the agreement will present a significant barrier to participation, and we believe that adding this layer of formality will benefit the Unidata community as a whole.</p>
<h2>How it Works</h2>
<p>When contributing using a Pull Request on GitHub, the following message will present itself, using <a href="https://cla-assistant.io">CLA Assistant</a>, as a comment on the pull request:</p>
<p><img src="/blog_content/images/2017/20170927_cla1.png" alt="Unsigned CLA" /></p>
<p>Contributors can click on the yellow "CLA not signed yet" badge, which will take them to a copy of the CLA. Contributors are asked to provide a little bit of information about themselves (for legal purposes):</p>
<p><img src="/blog_content/images/2017/20170927_cla2.png" alt="CLA page" /></p>
<p>Once the "I Agree" button is clicked, the browser will return to the original pull request page, but now the comment has been updated:</p>
<p><img src="/blog_content/images/2017/20170927_cla3.png" alt="Signed CLA" /></p>
<p>Contributors will only be asked to electronically sign once (unless the CLA is updated), and the agreement applies to all GitHub repositories hosted under the Unidata organization.</p>
<p>For more information about CLAs, see these resources:</p>
<ul>
<li><a href="http://harmonyagreements.org">Harmony Agreements</a></li>
<li>"Producing OSS" has a section on <a href="http://producingoss.com/en/contributor-agreements.html">Contributor Agreements</a></li>
<li>A more <a href="https://julien.ponge.org/blog/in-defense-of-contributor-license-agreements/">detailed blog post</a> by Julien Ponge about CLAs</li>
</ul>
<h2>About the Contributor License Agreement</h2>
<p>Unidata's CLA comes from
<a href="http://www.harmonyagreements.org">Project Harmony</a>, which is a
community-centered group focused on contributor agreements for free and open
source software.</p>
<p>The document you are reading now is not a legal analysis of the CLA. If you
want one of those, please talk to your lawyer. This is a description of the
purpose of the CLA.</p>
<h3>Why is a signed CLA required?</h3>
<p>The license agreement is a legal document in which you state you are entitled to
contribute the code/documentation to Unidata and are willing to have it used in
distributions and derivative works. This means that should there be any kind of
legal issue in the future as to the origins and ownership of any particular
piece of code, we have the necessary forms on file from the contributor(s)
saying they were permitted to make this contribution.</p>
<p>The CLA also ensures that once a contribution has been made, contributors cannot try
to withdraw permission for its use at a later date. People and companies can
therefore use Unidata open source projects, confident that they will not be asked
to stop using pieces of the code at a later date.</p>
<p>Lastly, the CLA gives Unidata permission to change the license under
which the project, including the various contributions from many developers, is
distributed in the future. The CLA states that this license needs to be one
that has been approved by the Open Source Initiative, including both copyleft
and permissive licenses. This gives Unidata the freedom to adjust licenses in the
future if needed (e.g. some clause of the current license is found to be invalid;
change to a standard license), so long as the license remains open source.</p>
<h3>Am I giving away the copyright to my contributions?</h3>
<p>No. This is a pure license agreement, not a copyright assignment. You still
maintain the full copyright for your contributions. You are only providing a
license to Unidata to distribute your code without further restrictions. This is
not the case for all CLA's, but it is the case for the one we are using.</p>
<h3>Can I withdraw permission to use my contributions at a later date?</h3>
<p>No. This is one of the reasons we require a CLA. No individual contributor can hold
such a threat over the entire community of users. Once you make a contribution, you
are saying we can use that piece of code forever.</p>
<h3>Can I submit patches without having signed the CLA?</h3>
<p>No. We will be asking all new contributors and patch submitters to sign before
they submit anything.</p>
<p class="quote">
This CLA explanation is based on <a href="https://www.djangoproject.com/foundation/cla/faq/">Django Contributor License Agreement Frequently Asked Questions</a> (copyright Django Software Foundation. <a href="http://creativecommons.org/licenses/by/3.0/us/">CC-BY</a>) The content has been modified slightly to reflect situations specific to Unidata.
</p>
https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_ddx_lexical_elementsDAP4 Commentary: DDX Lexical ElementsDennis Heimbigner 2012-03-27T18:00:50-06:002012-03-28T11:06:15-06:00This document describes the lexical elements that occur in
the DAP4 grammar.
<p>
Within the
<a href="http://dl.dropbox.com/u/53929684/xsd.rng">
Relax-NG (rng) DAP4 grammar</a>,
there are markers for occurrences of
primitive type such as integers, floats, or strings.
The markers typically look like this when defining
an attribute that can occur in the DAP4 DDX.
</p><pre><attribute name="namespace"><data type="string"/></attribute></pre>
The "<data type="string"/>"
specifies the lexical class for the values that this
attribute can have. In this case, the namespace attribute is
defined to have a String value. Similar notation is used
for values occurring as text within an xml element. The
lexical specification later in this document defines the
legal lexical structure for such lexical items.
Specifically, it defines the format of the following lexical
items.
<ol>
<li> Constants, namely: string, float, integer, and character.
</li><li> Identifiers
</li></ol>
<p>
The specification is written using the
<a href="http://www.iso.org/iso/iso_catalogue/catalogue_ics/catalogue_detail_ics.htm?csnumber=38790">
ISO/IEC 9945-2:2003 Information technology -- Portable Operating System Interface (POSIX) -- Part 2: System Interfaces</a>.
This is the extended Posix regular expression
specification.
</p><p>
I have augmented it in the following ways.
</p><ol>
<li>Names are assigned to regular expressions using the notation
<pre>name = regular-expression</pre>
<p>
</p></li><li>Named expressions can be used in subsequent regular
expressions by using the notation {name}. Such occurrences
are equivalent to textually substituting the expression
associated with name for the {name} occurrence: More or less
like a macro.
</li></ol>
<h4>DAP4 Lexical elements</h4>
Notes:
<ol>
<li>The definition of {UTF8} is deferred to the next section.
<p>
</p></li><li>Comments are indicated using the "//" notation.
<p>
</p></li><li>Standard xml escape formats (&xDD) are assumed to be allowed anywhere.
</li></ol>
<p>
<b>Basic character set definitions</b>
</p><pre>CONTROLS = [\x00-\x1F] // ASCII control characters<br />WHITESPACE = [ \r\t\f]+<br />HEXCHAR = [0-9a-zA-Z]<br />// ASCII printable characters<br />ASCII = [0-9a-zA-Z !"#$%&'()*+,-./:;<=>?@[\\\]\\^_`|{}~]<br /></pre>
<p>
<b>Ascii characters that may appear unescaped in Identifiers</b><br />
This is assumed to be basically all ASCII printable characters
except the characters ' ', '.', '/', '"', ''', and '&'.
Occurrences of these characters are assumed to be representable
using the standard xml '&xx;' notation.
</p><pre>IDASCII = [0-9a-zA-Z!#$%'()*+,-:;<=>?@[\\\]\\^_`|{}~]<br /></pre>
<p>
<b>The numeric classes: integer and float</b><br />
</p><pre>INTEGER = {INT}|{UINT}|{HEXINT}<br />INT = [+-][0-9]+{INTTYPE}?<br />UINT = [0-9]+{INTTYPE}?<br />HEXINT = {HEXSTRING}{INTTYPE}?<br />INTTYPE = ([BbSsLl]|"ll"|"LL")<br />HEXSTRING = (0[xX]{HEXCHAR}+)<br /><pre></pre><br />FLOAT = ({MANTISSA}{EXPONENT}?)|{NANINF}<br />EXPONENT = ([eE][+-]?[0-9]+)<br />MANTISSA = [+-]?[0-9]*\.[0-9]*<br />NANINF = (-?inf|nan|NaN)<br /></pre>
<p><b>The Character classes</b><br />
</p><pre>STRING = ([^"\&]|{XMLESCAPE})*<br />CHARACTER = ([^'\&]|{XMLESCAPE})<br /></pre>
<p>
Note that the character type only supports ASCII characters because
it can only hold a single 8-bit byte.
</p><p>
<b>The Identifier class</b><br />
</p><pre>ID = {IDCHAR}+<br />IDCHAR = ({IDASCII}|{XMLESCAPE}|{UTF8})<br />XMLESCAPE = &x{HEXCHAR}{HEXCHAR};<br /></pre>
<p>
Note that the above lexical element classes are not
disjoint. For example, the sequence of characters 1234 can
be either an identifer,a float, or an integer. So the order
of testing is assumed to be this.
</p><ol>
<li>INTEGER
</li><li>FLOAT
</li><li>ID
</li><li>STRING
</li></ol>
<h4>UTF-8 Character Encodings</h4>
We discuss UTF-8 character encoding in the context
of this document.
<a href="http://www.w3.org/2005/03/23-lex-U">
http://www.w3.org/2005/03/23-lex-U</a>.
<p>
The most correct (validating) version of UTF8 character set is as follows.
</p><pre>UTF8 = ([\xC2-\xDF][\x80-\xBF]) <br /> | (\xE0[\xA0-\xBF][\x80-\xBF]) <br /> | ([\xE1-\xEC][\x80-\xBF][\x80-\xBF]) <br /> | (\xED[\x80-\x9F][\x80-\xBF]) <br /> | ([\xEE-\xEF][\x80-\xBF][\x80-\xBF]) <br /> | (\xF0[\x90-\xBF][\x80-\xBF][\x80-\xBF]) <br /> | ([\xF1-\xF3][\x80-\xBF][\x80-\xBF][\x80-\xBF]) <br /> | (\xF4[\x80-\x8F][\x80-\xBF][\x80-\xBF])<br /></pre>
The lines of the expression cover the UTF8 characters as follows:
<ol>
<li> non-overlong 2-byte
</li><li> excluding overlongs
</li><li> straight 3-byte
</li><li> excluding surrogates
</li><li> straight 3-byte
</li><li> planes 1-3
</li><li> planes 4-15
</li><li> plane 16
</li></ol>
<p>
Note that ASCII and control characters are not included.
</p><p>
The above reference also defines some alternative regular expressions.
</p><p>
The most relaxed version of UTF8 is this.
</p><pre>UTF8 = ([\xC0-\xD6].)<br /> |([\xE0-\xEF]..)<br /> |([\xF0-\xF7]...)<br /></pre>
<p>
The partially relaxed version of UTF8 is this.
</p><pre>UTF8 = ([\xC0-\xD6][\x80-\xBF]) <br /> | ([\xE0-\xEF][\x80-\xBF][\x80-\xBF]) <br /> | ([\xF0-\xF7][\x80-\xBF][\x80-\xBF][\x80-\xBF])<br /></pre>
<p>
We deem it acceptable to use this last relaxed expression
for validating UTF-8 character strings.
</p>https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_dap4_grammarDAP4 Commentary: DAP4 GrammarDennis Heimbigner 2012-03-27T16:50:24-06:002012-03-27T16:50:24-06:00[Version: 1.0]
<p>
At the end of this document are instructions for accessing and
testing a formal grammar for the DAP4 DDX using the Relax-NG
schema language. I constructed it initially without any reference to any
other explicit or implicit grammars so I could record my ideas. I
have since modified it based on examining James'
<a href='%20http://docs.opendap.org/index.php/DAP4:_Data_Model'>
implied grammar</a>
and from comments from others and from
a comparison with the
<a href='http://scm.opendap.org/trac/browser/trunk/xml/dap/dap4.xsd'>
xsd grammar.</a>
</p><h4>Differences with DAP4 xsd Grammar</h4>
I converted the
<a href='http://scm.opendap.org/trac/browser/trunk/xml/dap/dap4.xsd'>
xsd grammar</a>
to an equivalent
<a href='http://dl.dropbox.com/u/53929684/xsd.rng'>
relax-ng (rng) grammar.</a>
<p>
One major difference I see is in dimension handling.
</p><ul>
<li> I just used the name "dimension" rather
than "shareddimension". For me, all dimensions (except anonymous
ones) are shared.
<p>
</p></li><li> The xsd separates out scalars from arrays. I always allowed
the dimensions for a variable to be optional to handle the scalar
case.
<p>
</p></li><li> I attempted to be as consistent as possible, so I allowed
any type including sequences and structures to be dimensioned.
(but see
<a href='https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_sequences_and_vlens'>previous commentary</a>).
<p>
</p></li><li> The dimensions of a variable are currently specified in the
rng grammar as a sequence of elements named "Dimension" contained
in the "variables" element type.
</li></ul>
Other differences:
<ul>
<li> The Dataset element in the xsd has a couple of extra
attributes. I added these.
<p>
</p></li><li> The xsd appears to allow attributes to themselves have
attributes. This needs discussion.
<p>
</p><p>
</p></li><li> The URL basetype is in the xsd. But I do not see the justification
for keeping it.
<p>
</p></li><li> It appears that the Dataset contains a top level <group>
declaration. I chose to treat the Dataset itself as the top-level
group.
<p>
</p></group></li><li> Attribute declarations appear to have their own "namespace"
attribute. Not sure why this is needed.
<p>
</p></li><li> I do not understand the purpose of the "NewAttribute" attribute.
<p>
</p></li><li> There may still be some minor differences in representing
coordinate variables.
<p>
</p></li><li>
The xsd represents attribute values thus:
<pre><attribute name="a"><value>...</value><value>...</value></attribute></pre>
I chose to use attributes in the multi-valued case because I
prefer not to use elements with content unless really
necessary. So I represented the above as this.
<pre><attribute name="a"><value value="..."/><value value="..."/></attribute></pre>
<p>
</p><p>
</p></li><li> There is an issue of interleaving of definitions, or
equivalently, what elements must occur in a fixed order.
<p>
</p></li><li> Where should attributes be legal? I think the rng grammar
and the xsd grammar agree on this: putting them almost
everywhere, but it needs discussion.
</li>
<p>
</p><li> I dropped Blobtype. I fail to see the need for this.
</li></ul>
<p> </p><h4> Testing the Relax-NG Grammar </h4>
You will need to copy three files:
<ol>
<li> dap4.rng - this is the grammar file. it uses the
<a href='http://relaxng.org'>Relax-NG schema language</a>
This grammar file can be obtained from
<a href='http://dl.dropbox.com/u/53929684/dap4.rng'>
http://dl.dropbox.com/u/53929684/dap4.rng</a>.
<p>
</p></li><li> test.xml - this is a test file that I am growing to cover the whole grammar.
This can be obtained from
<a href='http://dl.dropbox.com/u/53929684/test.xml'>
http://dl.dropbox.com/u/53929684/test.xml</a>.
<p>
</p></li><li> jing.jar - Jing is a validator that takes the grammar and a
test file and checks that the test file conforms to the
grammar. This can be obtained from
<a href='http://dl.dropbox.com/u/53929684/jing.jar'>
http://dl.dropbox.com/u/53929684/jing.jar</a>.
</li></ol>
To use this jar file, do the command:
<p> </p><pre>java -jar jing.jar dap4.rng test.xml</pre>No output is produced if the validation succeeds, otherwise,
error messages are produced.https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_possible_notation_forDAP4 Commentary: Possible Notation for Server CommandsDennis Heimbigner 2012-03-27T16:06:28-06:002012-03-27T16:06:28-06:00Looking to the future, it is clear that eventually
our query language, or more generically our previous discussion of
<a href='https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary1'>
URL Annotations</a> must encompass three classes of computations.
<ol>
<li> Queries in the DAP2 sense,
</li><li> Commands to control the client-side processing of requests on the server
(i.e. thing like caching),
</li><li> Server-side processing.
</li></ol>
I want to propose a notation for everything in the URL after
the "?". I think this notation has ability to represent a wide
variety of features without, I hope, being too generic.
<p>
The notation is basically nested functions combined with single
assignment variables. A semantically nonsensical, but
grammatical example would look something like this:
"?_x=f(17,g(h(12))),f2(_x,[0:3:10])".
</p><p>
Everything past the "?" is in the form of a comma separated
list of nested function invocations. Anything that begins with an
underscore is considered a local, temporary, variable, anything
that does not look like a function call (i.e. that is not a name followed
immediately by a left parenthesis) is assumed to be a string constant. Each
function has an arbitrary number of argument expressions
separated by commas.
</p><p>
There would be several semantic rules.
</p><ol>
<li> A variable may only be assigned to once (single assignment),
but may be referenced as many times as desired after that.
<p>
</p></li><li> All functions have a defined "return type", which looks like
a legal DDX minus certain things like groups, enumeration
declarations, and dimension declarations. In addition, a function
may be defined to have a "void" return type, which means it is
executed for its side-effects on the server.
<p>
</p></li><li> Any expression that is not assigned to a variable and does
not have a void return type will have its return value returned
to the caller as part of a DATADDX.
</li></ol>
<h4>Notes</h4>
My hypothesis is that this notation should also be able to handle
most kinds of server side processing by defining and composing
functions.
<p>
The standard projection+selection constraints of DAP2 can be
represented using a special query() function whose argument is
the standard DAP2 constraint, or alternatively, one could define
a collection of nested functions to do the same thing, or
alternatively, we could split the query part into two pieces
separated by a semicolon. The first piece would be a constraint
expression and the second piece (after the semicolon) would be in
the nest function call form defined above.
</p><p>
An important aspect has to do with the construction of what may
be referred to as a DATADDX. It defines the structure of a DDX
that is the composition of the return types of the invoked
functions that will return a (possibly structured) value. I need
to work this out. BUT, in any case, the resulting DATADDX may
have only have a loose relation to any DDX representing the raw
dataset. This is because server-side computations will not have
been represented in the original DDX, but only in the DATADDX.
</p><p>
I also hypothesize that Ferret notations
</p><pre> http://.../thredds/dodsC/hfrnet/agg/6km_expr_{}{let deq1ubar=u[d=1,l=1:24@ave]}<br /></pre>
could be represented in my proposed
function notation without having to clutter up the URL format.https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_sequences_and_vlensDAP4 Commentary: Sequences and VlensDennis Heimbigner 2012-03-27T15:52:23-06:002012-03-27T15:52:24-06:00<blockquote>
"Oh what a tangled web we weave
when first we practice to build a type system."
</blockquote>
Recently, I made the claim to James that the Sequence construct
could serve to represent CDM/HDF5 vlen constructs.
<p>
His reply was as follows:
</p><blockquote>
In my opinion we will want vlens to be in the data model, if not
as a type, then as a feature of arrays - see the schema and my
text on the Data Model page. The reason for that [is that] I have
already tried using Sequence for vlen (in hdf4) and it was a
failure. Not a failure from the technical POV but because neither
server writers nor client writers nor client users ever 'got it.'
I think one part of the problem for those people was that vlens
in HDF4 do not support relational operators via the API while
Sequences are supposed to. So the conceptual mismatch doomed the idea.
</blockquote>
<p>
This is a very important observation, and has caused me to
re-think how sequences and vlens should be handled in DAP4.
</p><h4>Vlens</h4>
Let us start by addressing the addition of vlens to the DAP4 data model.
<p>
Some possible ways to insert vlens into the dap4 data model
include the following.
</p><ol>
<li> In CDM, a vlen is marked by a "*" as a dimension name in the
set of dimensions associated with a variable. the list of
dimensions is allowed to have any number of occurrences
of "*" [Aside, I will note that "unlimited" is also an option
for CDM, but should not be needed for DAP4 because at the time of
a request for data, the size of the unlimited dimension is known].
<p>
</p></li><li> James has proposed something similar except that the "*" is
restricted to occurring as the last dimension.
<p>
</p></li><li> Another possibility is to create a new container object,
call it <i>Vlen</i>, that (like <i>Sequence</i>) is inherently of variable
length and is not dimensionable. [Aside: The term "vlen" is kind
of odd. the term "list" would actually make more sense]
</li></ol>
<p>
If we were to choose option 2 (terminal * only),
then we must address the question
of translation between CDM and DAP4. Going from DAP4 to CDM is
straightforward because having the last dimension be "*" is legal
in CDM. Going from CDM to DAP4 requires the introduction of a
number of Structure elements.
</p><p>
Consider the following CDM example (using a pseudo-syntax)
</p><pre> Int32 v[d1][*][d2][*][d3].<br /></pre>
This would have to be represented in DAP4 something like this.
<pre> Structure v_1 {<br /> Structure v_2 {<br /> Int32 v[d3];<br /> }[d2][*];<br /> }[d1][*];<br /></pre>
<p>
In the lucky case that the last dimension is a already a "*":
</p><pre> Int32 v[d1][*][d2][*];<br /></pre>
then we have a simpler representation.
<pre> Structure v_1 {<br /> Int32 v[d2][*];<br /> }[d1][*];<br /></pre>
<h4>Commentary</h4>
<ul>
<li> As a personal matter, I would prefer to use the CDM
representation. Adding a new container type (Vlen),while
appealing semantically, only complicates the model more. James'
approach requires the use of additional Structure definitions
which, in my opinion, obscures the underlying semantics.
<p>
</p></li><li> One thing that I need to check is how this affects the
proposed on the wire format.
</li></ul>
<h4>Sequences</h4>
Nathan Potter noted the following.
<blockquote>
At one time we considered using ''nested'' sequences as a
representation of an SQL database, where essentially the keys
that link the tables in the DB define the structure of some
nested sequence thing. I think it's a useful idea, but there is
no implementation of it to play with. [Aside: I could swear that
someone told me that they actually used a relational database as
the back end to a sequence]
</blockquote>
And later Nathan Potter noted the following:
<blockquote>
I mentioned (in the same email that contained the nested sequence
comment quoted above) that we wrote and released a server that
represents a single database table or view as a single
sequence. This is quite different from the point that I was
making in the section quoted above that there may be a use case
for nested sequences. (which was in response to your
question "Are there any other legitimate examples for using
NESTED sequences.") [Ndp 12:08, 26 February 2012 (PST)]
</blockquote>
<p>
It is this ability to have selection constraints applied that
separates sequences from vlens. It is clear that in the absence
of selection constraints, there is no essential difference
between sequences and vlens. Further, in the few examples I could
find or were sent to me, it seemed that nested sequences were
being used as, in effect, vlens.
</p><p>
So, we seem to have two very similar concepts (vlen and
sequence), which complicates the DAP4 model. The question for me is:
</p><blockquote>
Do we get rid of the Sequence concept, or at least define it as equivalent to the following?: Structure {...} [*].
</blockquote>
My current belief is that we should keep sequences but with the following restrictions:
<ol>
<li> Sequences can only occur as top-level, scalar, variables within a group.
</li><li> Sequences may not be nested in any other container (i.e. other sequences or structure)
</li></ol>
This keeps sequences for the original purpose of acting
as "relations". All other places where we might use a sequence
before will now use a vlen.
<p>
As with vlens, the translation between DAP4 and CDM needs to be addressed.
</p><ul>
<li> The conversion from DAP4 to CDM can be addressed using the
rule above, namely that a sequence is, in CDM, represented as
<pre>Structure {...} [*]</pre>.
<p>
</p></li><li> Translation from CDM to DAP4 allows for the option of never
using sequences, but always using vlens. An alternate translation
might be to say that if you have a top-level CDM structure whose
only dimension is a vlen, then translate that to a sequence (in
effect inverting the DAP4->CDM translation).
</li></ul>https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary1DAP4 Commentary: Characterization of URL AnnotationsDennis Heimbigner 2012-03-27T15:33:58-06:002012-03-27T15:54:42-06:00<h4>Characterization of URL Annotations</h4>
Requests for data using the DAP4 protocol will require a
significant number of annotations specifying what is to be
retrieved, commands to the server, and commands to the client.
<p>
This document is intended to just describe the information with
which URLs need to annotated based on past experience. It also
enumerates the possible URL components that can be used to encode
the annotations. I will consider a specific encoding in a
separate document.
</p><p>
Looking at the DAP2 URLs, we see three classes of annotations:
protocol, server commands, client commands, and queries (aka
constraints).
</p><h4>Protocol</h4>
For DAP2, the fact that the DAP2 protocol being used is inferred
from context. In netcdf-C, for example, the fact that the
dataset name is a URL is sufficient to indicate the use of the
DAP2 protocol (although that will change). For some servers,
such as TDS, the protocol is also inferrable from elements of the
URL path. For example, in the URL
<pre>http://.../thredds/dodsC/...<br /></pre>
the "dodsC" indicates the use of the DAP2 protocol. TDS also
supports a schema called "dods:" that also indicates the use of DAP2.
<h4>Server Commands</h4>
Server commands in DAP2 are appended to the dataset URL to
indicate attributes of the request. For example:
<pre>http://test.opendap.org/dataset.nc.dds<br /></pre>
The defined kinds of server commands for DAP2 are as follows:
<ul>
<li>Component requests: ".dds", ".das"
</li><li>Data requests with format: ".dods", ".asc"
</li><li>Miscellaneous: ".html", ".ver"
</li></ul>
<h4>Client commands</h4>
Client commands are interpreted by the client-side library to
specify actions to be performed by the library. The existence of
client commands is important because we want to communicate from
the user to the library without requiring any knowledge by
intermediate code layers. For example, netcdf C tools such as
ncdump send URLs to the underlying netcdf DAP2 library without
having to be cognizant of their structure.
Currently, the primary use for client commands is caching, to
indicate the degree of caching and prefetch to be used with a
given request to the server.
<p>
Currently, client commands are represented as "name=value" pairs
or just "name" enclosed in square braces: "[nocache]", for
example. These commands are prefixed to the URL such as this.
</p><pre>[show=fetch]http://test.opendap.org/dataset.nc<br /></pre>
The legal set of client commands is client library specific.
<p>
One notable problem with this form of client command is that it
prevents generic URL parsers from parsing the URL because, of
course, the square bracket notation is non-standard.
</p><p>
It should be noted that an alternative to using client commands
in the URL is to use a configuration file (often referred to as
the ".rc" file such as ".dodsrc"). This configuration file is
assumed to be either in the caller's home directory or in the
current working directory. It contains the necessary client
commands to be applied. It is mildly less convenient for the user
to use a .rc file than to embed a client command in the URL.
</p><h4>Queries</h4>
The third class of URL annotations specifies some form of query
to control the information to be extracted from a dataset on the
server. This information is then passed back to the client.
<p>
In DAP2, queries consisted of projections and selections
specifying a subset of the data in a dataset.
</p><p>
A projection represents a path through the DDS parse tree
annotated with constraints on the dimensions. For example, this query:
"?P1.P2[0:2:10].F[1:3][4:4]".
</p><p>
A selection represents a boolean expression to be applied to the
records of a sequence. Syntactically, a selection could cross
sequences, thus implying a join of the sequences, but in practice
this diss not allowed.
</p><p>
DAP2 queries also allowed the use of functions in the projections
and selections to compute, for example, sums or averages. But the
semantics was never very well defined. The set of allowable
functions is server dependent.
</p><h4>Annotation Mechanisms</h4>
DAP4 will need to support at least the three classes of
annotations described above. Whatever annotation mechanisms are
chosen, the following properties seem desirable.
<ul>
<li>The resulting URL should be parseable by generic URL parsers
=>Client commands should be embedded at the end of URLs, not the beginning.
</li><li>Whatever annotation encoding is used, it is desirable if it is as
uniform as possible.
</li></ul>
As mechanisms, we have the following available to us:
<ul>
<li> The URL schema -- "http:" for example, or the TDS "dods:"
schema. Using this is somewhat undesirable because it would need
to encode also an underlying encrypted protocol like
https: (versus http:).
<p>
</p></li><li> URL path elements such as the current use of
e.g. http://host/../dodsC/... by TDS.
<p>
</p></li><li> URL query -- everything after the first '?' in the URL. URL
queries technically have a defined form as name=value pairs, but
in practice are pretty much free form.
<p>
</p></li><li> URL fragment -- everything after the last '#'. Again these
are pretty much free form.
<p>
</p></li><li> Filename extensions -- everything between the data set name
in the path and the start of the query. The DAP2 ".dds"
and ".dods" are examples of this.
<p>
</p></li><li>Alternate extension formats. Ethan Davis has proposed
the use of a "+" notation instead of filename extensions:
"+ddx+ascii", for example. This has the advantage of clearly not
being confused with filename extensions while also making clear
the additive nature of such annotation.
</li></ul>
I should note that the Ferret server has taken to seriously
abusing the URL format with URLs like this.
<pre>http://.../thredds/dodsC/hfrnet/agg/6km_expr_{}{let deq1ubar=u[d=1,l=1:24@ave]}<br /></pre>
so we have much to aspire to :-)https://www.unidata.ucar.edu/blogs/developer/entry/dap4_commentary_the_on_theDAP4 Commentary: The on-the-wire formatDennis Heimbigner 2012-03-27T15:12:33-06:002012-03-27T15:12:33-06:00<h4>Background</h4>
The current DAP2 clients, use two different approaches to
managing the packet of data that is sent by the server.
<p>
The C++ libdap library uses what I will call an "eager"
evaluation method. By this I mean that the whole packet is
processed when received, is decomposed into its constituent
parts (e.g. data arrays, sequence records, etc) and those parts
are used to annotate the parsed DDS.
</p><p>
In contrast, the oc library uses a "lazy" evaluation method. That
is, the incoming packet is sent immediately into a file or into a
chunk of heap memory. Almost no preproccessing occurs. Data
extraction occurs only when requested by the user code through
the API.
</p><h4>Problem addressed</h4>
The relative merits and demerits of lazy versus eager are well
known and will not be repeated here.
Lazy evaluation of the DAP2 packet is hampered by the inlining of
variable length data: sequences and strings specifically. If it
were not for those, the lazy evaluator could compute directly the
location of the desired subset of data as requested by the user,
and do so without having to read any intermediate information.
But when, for example, Strings are inlined, then it is necessary
to walk the packet piece by piece to step over the strings.
I plan to use lazy evaluation for my implementations of DAP4, and
propose here the outline of a format for the on-the-wire data
packet that makes lazy operation fast and simple without, I
believe, interferring with eager evaluation.
<h4>Proposed solution</h4>
Since we have previously agreed on the use of multipart-mime, the incoming
data is presumed to be sequence of variable length packets with a
known length (for each packet) and a unique id for each packet.
<p>
Under these assumptions, I propose the following format.
</p><p>
</p><ol>
<li>The initial packet is of known computable length, aka "fixed
length" for short. That is, its size can be computed solely
knowing the DXD for the incoming data. This means that strings
and sequences are not represented inline, but instead are
represented by fixed-size "pointers" into other, following
packets that contain the sequence and/or string data.
<p>
</p></li><li>Each element in a string array in the initial packet is represented
by three pieces of fixed size info:
<ol>
<li>the unique id of the packet containing the contents of the string.
</li><li>the offset in the packet defined in (a).
</li><li>the length of the string in bytes (assuming utf-8 encoding).
</li></ol>
<p>
</p></li><li>As an optimization, the string packet can be directly
appended to the fixed size initial packet, in which case, the
first item is not strictly necessary.
<p>
</p></li><li>Given a sequence object either a scalar or as an array of
sequences, the sequence is replaced by the following fixed size item:
<ul>
<li>The unique id of the packet containing the sequence records
</li></ul>
<p>
</p></li><li>Further, each record of the sequence packet is assumed to
be "fixed length" by applying the rules above. This means that
knowing the total size of the packet containing the sequence
records, it is possible to know the exact number of records in
the packet without actually having to walk the sequence packet to
count them.
</li></ol>
<h4>Rationale for the solution</h4>
The above representation makes lazy evaluation very simple and a
given item in a packet can be reached in <i>o(1)</i> time. Even with the
case of nested sequences/vlens, the proper item can be reached in
<i>o(log n)</i> time where n is the depth of the nesting.
The cost is that a hash map is needed to map unique id's to
offsets in the file or heap memory.
The lazy versus eager cases also apply on the server
side. Currently, for example, the opendap code on the thredds
server takes the underlying data source (.nc file for example),
converts it to DAP2 and annotates the DDS with the data. Then as
a second pass, the annotations are converted as needed and sent out
over the wire.
A lazy version would associate elements of the underlying source
with the DDS. Transfer of the data to the wire would then occur
directly from the original source to the wire format a needed.
As an aside, I have a (untested and unverified)
hypothesis is that the proposed encoding will also simplify
the use of lazy evaluation on the server side.
<h4>Updates</h4>
<dl>
<dt>2012-02-20</dt><dd>The above encoding has as one consequence that
all embedded counts that currently exist in DAP2 are
superfluous. Ditto for the sequence record markers. It may still
be desirable to include the counts for purposes of error
checking, but they are not strictly necessary.
</dd></dl>https://www.unidata.ucar.edu/blogs/developer/entry/first_two_weeksFirst two weeks!Sean Arms2011-06-07T14:27:15-06:002011-06-09T09:06:16-06:00<p>Community Driven: <i>Sure, Unidata has Policy and Users
committees, so they must care about what the community thinks, right?
Right?!? I mean, I was there – I felt like they cared what the
Users Committee thought while we were there, in town, in person.</i></p><p>
<meta http-equiv='CONTENT-TYPE' content='text/html; charset=utf-8' />
<title></title>
<meta name='GENERATOR' content='OpenOffice.org 3.3 (Unix)' />
<style type='text/css'>
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
A:link { so-language: zxx }
--></style></p><p align='left'><table width='450' height='288'><tbody><tr align='center'><td> <img width='431' vspace='0' hspace='0' height='231' border='0' align='bottom' src='https://www.unidata.ucar.edu/blog_content/images/UnidataCommDriven.png' alt='Image from Jeff Weber's t-shirt today!' /><br /></td></tr><tr align='left'><td><p><sub><i>Image taken from Jeff Weber's t-shirt today! Talk the talk, walk the walk, and wear the schwag!</i></sub><br /></p></td></tr></tbody></table> </p><p>As I begin my third week at Unidata, I
thought I would take a moment to reflect on my experience. It would
be quite a task to try to summarize my experience in one post, so
I’ll start off simply reflecting on the phrase “community
driven”, as it’s a phrase I’ve heard for years but never fully
appreciated.</p><p style='margin-bottom: 0in;'>“Community driven” was a phrase I heard tossed around
several times during my stint on the Unidata Users Committee as the
student representative, but it is hard to actually grasp what that
means until you’ve been put into a position to see colleagues put
that phrase into action. <i>Sure, Unidata has Policy and Users
committees, so they must care about what the community thinks, right?
Right?!? I mean, I was there – I felt like they cared what the
Users Committee thought while we were there, in town, in person.</i>
During the Users Committee meetings, we would bring up issues and
needs that we, or others in the community, felt needed to be
addressed. Six months later we would return to find that those
concerns were not just being listened to, but being acted upon. But
what was going on behind the scenes during those six months? Did the
staff address our concerns in the one week following the meeting,
only to forget about the community until our next Users Committee
meeting six months later? I never really thought this was the case,
but the cynical side of me (thanks, grad school!) is always drawn to this possibility (with any
organization claiming to be community oriented, not just Undiata).
Well, now that I’ve toured backstage area of Foothills Lab 4, I
have an idea of what happens during those periods.</p><p style='margin-bottom: 0in;'><b>Observation 1:</b> One of my first tasks as
Unidata as a Software Engineer has been to dig into support. While I
always heard that support was a big deal within Unidata, now that I
am ‘on the other side’, so to say, I have to agree that not only
is support a big deal, it’s an <b>unbelievably huge deal!</b> It’s clear
that addressing community needs is not something that happens only a
few times a year during the governing committee meetings and then
gets put on the back burner, but it’s a DAILY thing. Quite
honestly, I think it’s fun (yeah, I’ve logged into support at
home during the weekends – it’s kind of addicting...I may need a
support group in a few months). I’ve had the opportunity to answer
questions regarding the specifics of operating systems I’ve never
used and compilers I’ve never touched. While Google has been my
friend, a even bigger help has been looking at the wealth contained
within previous support questions and answers (these are indexed by
Google, BTW, and are available to all). The support tickets I’ve
reviewed show that the Unidata staff puts quite a bit of thought and
time into fully answering the issues submitted to support. On one
hand, the detailed replies make support easier for us in the future (especially
since those replies show up on through internet searches), but it also helps educate
the users. On the other hand, however, it sets the bar high – a bar that I
can only hope to meet!</p><p><b>Observation 2:</b> Another task I’ve been
assigned is to go through the workshop / online training materials to
look for areas where steps that are obvious to the developers do not
quite translate to the end users. It was always annoying in school to
hear a prof say “it’s obvious that...”, and even more annoying
to read it in a book (at least I could question the prof in class!).
What was even more annoying, and infuriating, is when steps were
skipped and it was never even hinted that they were skipped (at least
the ‘it’s obvious’ statement would clue us students into the
fact that a leap of faith had occurred). Ok, so how does this relate
to the concept of community driven / community oriented? Well, I
could have been told to go through these materials on my own time at home, as it’s
part of my job to know this stuff (and not my job to learn the basics
at work). I could have been told that users who don’t understand
basic computing should not being using our packages. <i>I <u>could</u> have been told those things.</i> Instead, it was
clear that if the community cannot use our packages due to ‘leaps
of faith’ in the documentation, then what is the point? If we
aren’t serving our community in every area (support, documentation,
and software), then we might as well pack-up and go home. So, let’s
take the time to allow someone who knows a bit about computing and a
bit about meteorology (as well as someone who grasps, at some level,
the general computing abilities of those in an academic setting) to
take a look at our training material in hopes that these documents
can be further clarified! An added bonus for me – I may get to help
out during the training workshops! I always enjoyed teaching, so this
will be an opportunity to get back into a classroom-like setting.</p><p><b>Observation 3:</b> I’ve had the chance to
sit through a few staff meetings so far, and I’ve been impressed at
the number of times I’ve heard “what does the community think?”
or "this brought up at a Users or Policy committee meeting.”.
I could play devil's advocate and say “well, strategic planning for
the next proposal is spinning up, so maybe that is why there is so
much focus on the community”, but the feel of these conversations
has been very much “business as usual”. Bottom line – these
folks care about your thoughts, needs, concerns, etc.!</p><p>In the end, those of us at Unidata can
care all we want about your needs and concerns, but we cannot help
you if we don’t hear from you! If the lines of communication break
down, we all lose :-( So, how can you keep in touch with us? Well,
there are several ways, and here are a few:
</p><p>
</p><ol><li><p style='margin-bottom: 0in;'>Contact those awesome individuals
serving on our governing committees (maybe even consider serving!)
(<a href='../../../community/index.html#governance'>https://www.unidata.ucar.edu/community/index.html#governance</a>)!</p>
</li><li><p style='margin-bottom: 0in;'>Want to get involved with Unidata
but don’t have the financial resources in your department? Well
have we the deal for you: Unidata Equipment Awards!
(<a href='../../../community/equipaward/'>https://www.unidata.ucar.edu/community/equipaward/</a>)</p>
</li><li><p style='margin-bottom: 0in;'>Check us out on Twitter or
Facebook.
</p>
</li><li><p style='margin-bottom: 0in;'>Subscribe to our “News from
Unidata” blog
(<a href='../../news/feed/entries/atom'>https://www.unidata.ucar.edu/blogs/news/feed/entries/atom</a>)
or the “Developer’s Blog”
(<a href='../../developer/'>https://www.unidata.ucar.edu/blogs/developer/</a>)
(both are open for comments, and comments - we love em’).</p>
</li><li><p style='margin-bottom: 0in;'>Subscribe to our users mailing
lists and participate
(<a href='../../../support/#mailinglists'>https://www.unidata.ucar.edu/support/#mailinglists</a>)!
</p>
</li><li><p style='margin-bottom: 0in;'>If you’re in Boulder for a
workshop or meeting, come say hello
(<a href='../../../about/index.html#visit'>https://www.unidata.ucar.edu/about/index.html#visit</a>)!
</p>
</li><li><p style='margin-bottom: 0in;'>Visit us at the Fall AGU or Annual
AMS meetings
(<a href='../../../events/index.html#conferences'>https://www.unidata.ucar.edu/events/index.html#conferences</a>).</p>
</li><li><p style='margin-bottom: 0in;'>Hang out with us virtually and see
what’s going on in the community by checking out a seminar through
the Unidata Seminar Series
(<a href='../../../community/seminars/'>https://www.unidata.ucar.edu/community/seminars/</a>)</p>
</li><li><p style='margin-bottom: 0in;'>Consider attending a training
workshop (<a href='../../../events/2011TrainingWorkshop/'>https://www.unidata.ucar.edu/events/2011TrainingWorkshop/</a>).</p>
</li><li><p>Consider attending the 2012
Triennial Workshop (organized by the Users Committee and Unidata!).
More details to come, but see:
<a href='../../../events/index.html#triennial'>https://www.unidata.ucar.edu/events/index.html#triennial</a>. <br /></p></li></ol><p>Which of these do you use? Which are most appealing to you? What can we do to better keep those lines of communication open? <br /></p>