Populating your RAMADDA server

7 Populating your RAMADDA server

So, you have your RAMADDA server installed, now you may ask how do I get go about populating it with data and other content. We covered how to make groups and upload files (or archives of files) through the web interface. However, RAMADDA also supports ways of providing access to the files that are already on the server machine.

7.0 Harvesting files

RAMADDA provides a harvester facility (see harvesters.html for more details) that allows an administrator to scan the local file system on the server and add entries into the database for those files. Here is an example of a harvester that walks a directory tree (/data/ldm/pub/casestudies/july18_2002) add adds group and file entries under the main RAMADDA group RAMADDA/Case Studies/Test.

Image 1: Harvesting files

Simply create the harvester, specify the file path to look under, specify a pattern to match (.* is match all files) and specify the RAMADDA group to add the new group under. See the above link for more details.

7.1 Serving up local files

If the server machine you have RAMADDA running on has a large collection of data files then you can very easily provide RAMADDA access to those files with the "Local File View" entry type. This provides a view of the file system. The Groups and Entries it creates are not in the database but they provide a way to access the data through all of the RAMADDA services.

For example, on Unidata's motherlode server we have a large archive (10's of millions of files) of real time data from Unidata's Internet Data Distribution system (IDD). This would be too large to harvest into the database. Instead, on our RAMADDA server we have a Local File View entry: IDD Data that provides a synthetic view of this large directory tree.

You create one of these entry types through the File->New Entry menu. The create form looks like:

Image 2: Creating a Local File View Entry

You specify a directory path on the server machine (e.g,. /data/ldm/pub) and other configuration options (e.g., what to show, what not to show, etc.)

Note: for security reasons the local file path that is specified has to be a subdirectory of one of the directory paths specified in the Site Administration page under Settings->Access->File System Access

You end up then with a view of that directory that looks like:

Image 3: Local File View Example