New TDS Cloud Architectures: Proposal 1

[First Draft: 9/15/2016]
[Last updated: 9/16/2016]

The Thredds Data server (TDS) was designed to operate in a client-server architecture. Recently, Unidata has moved TDS into the cloud using its existing architecture.

There seems to be agreement inside Unidata that we need to begin rethinking that architecture to adapt to the realities of the cloud.

Proposal 1

This (first) proposal makes an assumptions about the nature of the cloud, and especially as it is likely to be in the near future.

This Assumption is that rather than having large quantities of data behind a (TDS) server, all data will be stored in cloud storage such as Amazon S3 or Azure blobs.

Secondarily, in such an environment, TDS cannot be aware of all data because it the set of all data is likely to be growing at a fast rate and by organizations not known to a given TDS server.

In this environment, the role of TDS becomes more of a locator and transformer of data. That is, TDS is must be made aware of some datasets and then it must apply various computations on that data to produce new derived data and then publish it into cloud storage.

Some consequences:

  • Unidata may have to get into the data discovery business; somthing it has tended to avoid so far.
  • The new TDS must be organized so that others can extend its capabilities by providing new kinds of computation models.
  • It is not clear if protocols such as DAP2, DAP4, CDMremote, etc. will be needed any longer because clients will be able to access the computed products using the S3 or Blob interfaces. In effect, streaming becomes replaced with the reification of computations into a file in S3/Blob.
  • Asynchronous computations more or less fall out of this proposed architecture if it possible for a client to poll S3/Blob for some dataset or for getting an event notification from the cloud.
  • Standardized file formats now become important than ever. The primary such formats for atmospherics is, I believe, netcdf3 and netcdf4. The HDF5 format is likely to also become more important, although its complexity vis-a-vis netcdf-4 will IMO hold it back.

Some questions:

  • Is there room for another (or several) standard file formats?
  • Is it possible to define a wrapper API for S3 and Azure blobs and whatever google and other cloud companies provde? This API would help clients having to lock in on a single provider?
  • What is the relation between this proposal and, say, Amazon lambda, or microservices?

[9/16/2016]

Notes on Services to be Provided

Catalog

Our current catalog system assumes that there is some set of dataset over which we have control and knowledge. As a rule, that set is the set of datasets on the Thredds server machine.

Under this proposal, this becomes less true. There may be no such set. Let us propose instead that we provide an umbrella catalog for which others can ask to have their datasets included. Additionally, others might ask to have their catalogs grafted onto our catalog tree. In any case, we are effectively talking about a federated catalog.

The value added is that we become the place to go to locate datasets. A consequence is that it becomes incumbent on us to:

  1. Make searching our catalogs easy and support sophisticated searches.
  2. Provide our catalog in a variety of formats, such as in the form of a set of relational tables.
  3. Provide the ability to crack datasets to obtain additional information for our catalogs.

CDM

We also need to think about the role of CDM in this proposal. currently, CDM is our UNCOL (historical reference) in that CDM is the common model that allows us to separate the dataset format from the users of that dataset. That is, an IOSP maps some data format to CDM and then tools can be defined in terms of CDM to avoid having to know about all the actual data formats. This is a very powerful approach and we should not discard it.

Subset Services

Data subsetting services, in the form of NCSS and the dap(2,4) constraint languages is an additional service we provide that will continue to be important in any new architecture. In fact, I think that pulling this out as a set of services would be enhanced with this architecture. [Needs more thought].

[More thoughts will be added as they occur to me]

From GEMPAK to AWIPS: Building the NSHARP Dynamic Library on OS X

A little known fact in the world of AWIPS(II) is just how dependent the system still is on NAWIPS-GEMPAK. The entire National Centers Perspective is dependent on pre-built shared object files for 64-bit Linux, which means that all of the D2D plugins which extend NSHARP (for bufr obs, NPP profiles, forecast models, etc.) also depend on these libraries.

This dependency has prevented use of the NSHARP plugin in the first release (15.1.1) of the OS X CAVE client. These are the steps taken to build NSHARP and GEMPAK libraries for OS X AWIPS 16.2.2.

You will need the https://github.com/Unidata/awips2-gemlibs repository on your Mac, as well as gcc and gfortran (from XCode). Pay attention to any version-specific include path or linked files, such as /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/, always account for the correct versions and locations on your own system.

NSHARP pre-built libraries

libbignsharp.dylib

Using the script below, the NSHARP dynamic library is built from C and FORTRAN source files (and their required include files supplied by the awips2-gemlibs repository, and as linked against $GEMINC, meaning that GEMPAK for OS X must be built and installed).

git clone https://github.com/Unidata/awips2-gemlibs.git
cd awips2-gemlibs/nsharp/

An optional step, which can be performed in a separate script or within the build script below, is to create ld-style *.a files in $OS_LIB which can then be referenced with -l flags (e.g. -lgemlib):

libs=(snlist sflist nxmlib gemlib gplt cgemlib rsl device xwp xw ps gn nsharp netcdf textlib)
for file in ${libs[@]}
do
  if [ ! -f $OS_LIB/lib$file.a ]; then
    echo "$OS_LIB/lib$file.a does not exist"
    if [ -f $OS_LIB/$file.a ]; then
      cp $OS_LIB/$file.a $OS_LIB/lib$file.a
      echo "copied OS_LIB/$file.a to OS_LIB/lib$file.a for linking"
    fi
  fi
done

Build libbignsharp.dylib with the following script (Note the GEMPAK includes and links -I$NSHARP, -I$GEMPAK/include, -L$OS_LIB, etc.).

#!/bin/bash
cd ~/awips2-gemlibs/nsharp/
. $NAWIPS/Gemenviron.profile
CC=gcc
FC=gfortran

export NSHARP=$GEMPAK/source/programs/gui/nsharp
export NWX=$GEMPAK/source/programs/gui/nwx

myLibs="$OS_LIB/ginitp_alt.o $OS_LIB/gendp_alt.o"

myCflags="$CFLAGS -I. -I./Sndglib -I$NSHARP  -I$GEMPAK/include  -I$OS_INC -I$NWX \
-I/opt/X11/include/X11 -I/usr/include/Xm -I/opt/local/include -I/usr/include/malloc -Wcomment -Wno-return-type -Wincompatible-pointer-types -DUNDERSCORE -fPIC -DDEBUG -c"

myFflags="-I. -I$OS_INC -I$GEMPAK/include -I$NSHARP -fPIC -g -c -fno-second-underscore -fmax-errors=200 -std=f95"

myLinkflags="-L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/ -L/opt/local/lib -L$OS_LIB -L. -L./Sndglib -L/usr/X11R6/lib \
-shared -Wl -Wcomment -Wincompatible-pointer-types -Wimplicit-function-declaration -Wno-return-type,-install_name,libbignsharp.dylib -o libbignsharp.dylib"

myLibsInc="$OS_LIB/ginitp_alt.o $OS_LIB/gendp_alt.o $OS_LIB/libnxmlib.a $OS_LIB/libsnlist.a \
 $OS_LIB/libsflist.a $OS_LIB/libgemlib.a $OS_LIB/libcgemlib.a $OS_LIB/libgplt.a $OS_LIB/libdevice.a \
 $OS_LIB/libxwp.a $OS_LIB/libxw.a $OS_LIB/libps.a  $OS_LIB/libgn.a $OS_LIB/libcgemlib.a $OS_LIB/libgemlib.a \
 $OS_LIB/libnetcdf.a $OS_LIB/libtextlib.a $OS_LIB/libxml2.a $OS_LIB/libxslt.a \
 $OS_LIB/libgemlib.a $OS_LIB/libcgemlib.a $OS_LIB/librsl.a $OS_LIB/libbz2.a"

myLinktail="-I$OS_INC \
  -I$GEMPAK/include -I$NWX -I$NSHARP -I. -I./Sndglib  -I/opt/X11/include/X11 -I/usr/include -I/usr/include/Xm -I/opt/local/include/ -I/opt/local/include -lhdf5 -lgfortran -ljasper -lpng -liconv -lc -lXt -lX11 -lz -lm -lXm"

$CC $myCflags *.c Sndglib/*.c
$FC $myFflags *.f
$CC $myLinkflags *.o $myLibsInc $myLinktail

cp libbignsharp.dylib ~/awips2-ncep/viz/gov.noaa.nws.ncep.ui.nsharp.macosx/

GEMPAK pre-built libraries

libgempak.dylib

libgempak.dylib is built in a similar way as libbignsharp.dylib:

#!/bin/bash
cd ~/awips2-gemlibs/gempak/
. $NAWIPS/Gemenviron.profile
CC=gcc
FC=gfortran

myCflags="$CFLAGS -I. -I$GEMPAK/source/diaglib/dg -I$GEMPAK/source/gemlib/er \
-I/opt/X11/include/X11 -I/usr/include/Xm -I/opt/local/include -I/usr/include/malloc -fPIC -DDEBUG -c"

myFflags="-I. -I$OS_INC -I$GEMPAK/include -fPIC -g -c -Wtabs -fno-second-underscore"

myLinkflags="-L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/ -L/opt/local/lib -L$OS_LIB -L. \
-shared -Wl -Wno-return-type,-install_name,libgempak.dylib -o libgempak.dylib"

myLibs="$OS_LIB/ginitp_alt.o $OS_LIB/gendp_alt.o $OS_LIB/libcgemlib.a \
$OS_LIB/libsflist.a $OS_LIB/gdlist.a $OS_LIB/libcgemlib.a $OS_LIB/libgemlib.a \
$OS_LIB/libcgemlib.a $OS_LIB/libgplt.a $OS_LIB/libdevice.a $OS_LIB/libcgemlib.a \
$OS_LIB/libgn.a $OS_LIB/libgemlib.a $OS_LIB/libcgemlib.a $OS_LIB/libnetcdf.a \
$OS_LIB/libcgemlib.a $OS_LIB/libtextlib.a $OS_LIB/libxml2.a $OS_LIB/libxslt.a \
$OS_LIB/libcgemlib.a $OS_LIB/libgemlib.a $OS_LIB/libcgemlib.a $OS_LIB/libcgemlib.a \
$OS_LIB/librsl.a $OS_LIB/libcgemlib.a $OS_LIB/libbz2.a"

myLinktail="-I$OS_INC -I$GEMPAK/include -I. -I/opt/X11/include/X11 -I/usr/include \
-I/usr/include/Xm -I/opt/local/include/ -I/opt/local/include \
-lhdf5 -lgfortran -ljasper -lpng -liconv -lc -lXt -lX11 -lz -lm -lXm"

$CC $myCflags *.c
$FC $myFflags *.f
$CC $myLinkflags *.o $myLibs $myLinktail

cp libgempak.dylib ~/awips2-ncep/viz/gov.noaa.nws.ncep.viz.gempak.nativelib.macosx/

libcnflib.dylib

#!/bin/bash
cd ~/awips2-gemlibs/cnflib/
. $NAWIPS/Gemenviron.profile
CC=gcc
FC=gfortran

myCflags="$CFLAGS -I/opt/X11/include/X11 -I/usr/include/Xm -I/opt/local/include \
-I/usr/include/malloc -Wno-return-type -DUNDERSCORE  -fPIC -DDEBUG -g -c"

myLinkflags="-L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/ -L/opt/local/lib \
-shared -Wl -Wno-return-type,-install_name,libcnflib.dylib -o libcnflib.dylib"

myLinktail="-lgfortran -lc"

myLibs="$OS_LIB/ginitp_alt.o $OS_LIB/gendp_alt.o $OS_LIB/gdlist.a $OS_LIB/gdcfil.a \
$OS_LIB/libgemlib.a $OS_LIB/libgplt.a $OS_LIB/libdevice.a $OS_LIB/libgn.a \
$OS_LIB/libcgemlib.a $OS_LIB/libgemlib.a $OS_LIB/libnetcdf.a $OS_LIB/libtextlib.a \
$OS_LIB/libxslt.a $OS_LIB/libxml2.a -liconv \
$OS_LIB/libz.a $OS_LIB/librsl.a -lbz2"

$CC $myCflags *.c
$CC $myLinkflags *.o $myLibs $myLinktail

cp libcnflib.dylib ~/awips2-ncep/viz/gov.noaa.nws.ncep.viz.gempak.nativelib.macosx/

libaodtv64.dylib

#!/bin/bash
CC=gcc
FC=gfortran

cd ~/awips2-gemlibs/aodt/AODTLIB/

gcc -fPIC -g -c -Wall *.c *.h
gcc -shared -Wl,-Wno-return-type,-install_name,libaodtv64.dylib -o libaodtv64.dylib *.o -lc

cp libaodtv64.dylib ~/awips2-ncep/viz/gov.noaa.nws.ncep.viz.gempak.nativelib.macosx/

libg2g.dylib

#!/bin/bash
cd ~/awips2-gemlibs/g2g/
. $NAWIPS/Gemenviron.profile
CC=gcc
FC=gfortran

myCflags="$CFLAGS -I$GEMPAK/include -I. -I$GEMPAK/source/diaglib/dg \
-I$GEMPAK/source/gemlib/er -I/opt/X11/include/X11 -I/usr/include/Xm \
-I/opt/local/include -I/usr/include/malloc -Wno-return-type -DUNDERSCORE \
-fPIC -DDEBUG -c"

myFflags="-I. -I$OS_INC -I$GEMPAK/include -fPIC -g -c -Wtabs -fno-second-underscore"

myLinkflags="-L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/ -L/opt/local/lib \
-L/usr/X11R6/lib -shared -Wl -Wno-return-type,-install_name,libg2g.dylib -o libg2g.dylib"

myLinktail="-lgfortran $OS_LIB/libjasper.a -lpng  -lc"

myLibs="$OS_LIB/ginitp_alt.o $OS_LIB/gendp_alt.o $OS_LIB/gdlist.a \
$OS_LIB/gdcfil.a $OS_LIB/libgemlib.a $OS_LIB/libgplt.a $OS_LIB/libdevice.a \
$OS_LIB/libgn.a $OS_LIB/libcgemlib.a $OS_LIB/libgemlib.a $OS_LIB/libnetcdf.a \
$OS_LIB/libtextlib.a $OS_LIB/libxslt.a $OS_LIB/libxml2.a \
-liconv $OS_LIB/libz.a $OS_LIB/librsl.a -lbz2"

$CC $myCflags *.c
$FC $myFflags *.f
$CC $myLinkflags *.o $myLibs $myLinktail

cp libg2g.dylib ~/awips2-ncep/viz/gov.noaa.nws.ncep.viz.gempak.nativelib.macosx/

Colored Temperature Obs in CAVE D2D (NMAP2-style)

One of the new data visualizations available in the upcoming AWIPS 16.2.2 release is a recreation of the legacy GEMPAK/NAWIPS colored temperature ("color_temp") bundle.

AWIPS CAVE

GEMPAK NMAP2

This plugin is available as the first item in the Surface menu:

Upload and Download Support for TDS

For version 5.0.0, it is possible to configure TDS to support the uploading and downloading of files into the local file system using the "/thredds/download" url path. This is primarily intended to support local File materialization for server-side computing. The idea is that a component such asJupyter can materialize files from TDS to make them available to code being run in Jupyter. Additionally, any final output from the code execution can be uploaded to a specific location in the TDS catalog to make it available externally.

[Read More]

Implementing Multi-threaded IO for the NetCDF-3 format

The issue of multi-threaded read and write of NetCDF files has repeatedly arisen in the netcdf mailing list (netcdfgroup@unidata.ucar.edu). Read on for a discussion of some options.

[Read More]
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« September 2016
SunMonTueWedThuFriSat
    
1
2
3
4
5
6
7
8
10
11
12
13
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
       
Today