Update on 13.2.1 Grib Decoder Threads
29 April 2013
Since the last update, which involved testing only the ingest and decoding of CONDUIT 0.5/2.5 degree GFS, I've opened up the NGRID and NEXRAD3 feeds, as well as text and satellite products from the WMO and NIMAGE feeds, respectively.
The goal is to compare the speed of the grid decoder on high-resolution CONDUIT GFS runs alone versus running in parallel with the full nationwide NEXRAD3 feed and other products.
[Read More]Testing 13.2.1 Unified Grib Decoder on CONDUIT GFS
22 April 2013
Last month we received the first version of AWIPS II which included the new unified grib decoder (13.1.2). The install procedure for 13.1.2 was more complicated than usual - we needed the full 13.1.1 installation plus a 13.1.2 "update" - so around 8 GB of RPMs to manage.
01 April 2013
Chunking Data: Choosing Shapes
28 March 2013
In part 1, we explained what data chunking is about in the context of scientific data access libraries such as netCDF-4 and HDF5, presented a 38 GB 3-dimensional dataset as a motivating example, discussed benefits of chunking, and showed with some benchmarks what a huge difference chunk shapes can make in balancing read times for data that will be accessed in multiple ways.
In this post, I'll continue looking at that example dataset to see how we can derive good chunk shapes, generalize to other datasets, look at how long it can take to rechunk a multidimensional dataset, and look at the use of Solid State Disk (SSD) for both accessing and rechunking data.
[Read More]29 January 2013
What is data chunking? How can chunking help to organize large multidimensional datasets for both fast and flexible data access? How should chunk shapes and sizes be chosen? Can software such as netCDF-4 or HDF5 provide better defaults for chunking? If you're interested in those questions and some of the issues they raise, read on ...
[Read More]
All Entries (90)
GEMPAK (0)
NetCDF Java (24)
NetCDF (74)
Unidata (7)
LDM (0)
McIDAS (0)
UDUNITS (1)
TDS (2)
IDV (1)
AWIPS II (5)
