Re: How to save a ZIP file within a NetCDF file

• To: Ed Hill <eh3@xxxxxxx>
• Subject: Re: How to save a ZIP file within a NetCDF file
• From: "Richard P Signell" <rsignell@xxxxxxxx>
• Date: Tue, 23 Nov 2004 18:58:45 -0500

<FONT face="Default Sans Serif, Verdana, Arial, Helvetica, sans-serif" 
size=2><DIV>Ed,</DIV><DIV>&nbsp;</DIV><DIV>The code hash idea sounds useful for 
huge&nbsp;codes.&nbsp;&nbsp;&nbsp;</DIV><DIV>&nbsp;</DIV><DIV>But I still think 
the idea of stuffing the whole code into the</DIV><DIV>netcdf file will work 
for many folks and be the most desirable</DIV><DIV>route.&nbsp; I 
guess&nbsp;by&nbsp;your&nbsp;definition of "medium",&nbsp;the code I'm 
using</DIV><DIV>(ROMS) is "small", for though it is about 100,000 lines of 
FORTRAN code, </DIV><DIV>it compresses to less than 1 Mb!&nbsp;&nbsp;Since my 
netcdf output </DIV><DIV>files are&nbsp;typically&nbsp;300 -3000 Mb, I won't 
even notice the difference.</DIV><DIV>How big is your 
code?!</DIV><DIV>&nbsp;</DIV><DIV>For all the guys who are faithful to&nbsp;CVS 
and the like, this </DIV><DIV>code embedding stuff&nbsp;is certainly not 
necessary, but most of </DIV><DIV>the ocean modelers I know have 15 different 
unofficial versions </DIV><DIV>of any particular official version number, and 
they are&nbsp;are not keen </DIV><DIV>on giving any of them a finer grained 
version number for some</DIV><DIV>pretty good reasons (can't put a fine grained 
version number on</DIV><DIV>a 
band-aid)!</DIV><DIV>&nbsp;</DIV><DIV>-Rich</DIV><DIV><BR>Dr.&nbsp;Richard&nbsp;P.&nbsp;Signell&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rsignell@xxxxxxxx<BR>U.S.&nbsp;Geological&nbsp;Survey&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(508)&nbsp;457-2229&nbsp;<BR>384&nbsp;Woods&nbsp;Hole&nbsp;Road&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Fax:&nbsp;(508)&nbsp;457-2310<BR>Woods&nbsp;Hole,&nbsp;MA&nbsp;&nbsp;02543-1598&nbsp;&nbsp;&nbsp;<BR></DIV><DIV><DIV><BR></DIV><FONT
 color=#990099>-----owner-netcdfgroup@xxxxxxxxxxxxxxxx wrote: 
-----<BR><BR></FONT>To: Rich Signell &lt;rsignell@xxxxxxxx&gt;, 
netcdfgroup@xxxxxxxxxxxxxxxx<BR>From: Ed Hill &lt;eh3@xxxxxxx&gt;<BR>Sent by: 
owner-netcdfgroup@xxxxxxxxxxxxxxxx<BR>Date: 11/23/2004 05:41PM<BR>Subject: Re: 
How to save a ZIP file within a NetCDF file<BR><BR><font face="monospace" 
size=2>On&nbsp;Tue,&nbsp;2004-11-23&nbsp;at&nbsp;14:08&nbsp;-0500,&nbsp;Rich&nbsp;Signell&nbsp;wrote:<BR>&gt;&nbsp;Folks,<BR>&gt;&nbsp;<BR>&gt;&nbsp;Here's&nbsp;a&nbsp;problem&nbsp;we&nbsp;encounter&nbsp;all&nbsp;the&nbsp;time&nbsp;in&nbsp;the<BR>&gt;&nbsp;modeling&nbsp;business:&nbsp;&nbsp;we&nbsp;have&nbsp;a&nbsp;model&nbsp;run&nbsp;(netcdf&nbsp;output,<BR>&gt;&nbsp;of&nbsp;course),&nbsp;and&nbsp;we&nbsp;can't&nbsp;quite&nbsp;figure&nbsp;out&nbsp;what&nbsp;version&nbsp;of<BR>&gt;&nbsp;the&nbsp;code&nbsp;actually&nbsp;was&nbsp;used&nbsp;to&nbsp;produce&nbsp;it.&nbsp;&nbsp;Oh&nbsp;sure,<BR>&gt;&nbsp;it&nbsp;says&nbsp;"version&nbsp;2.1"&nbsp;but&nbsp;that&nbsp;was&nbsp;before&nbsp;we&nbsp;hacked<BR>&gt;&nbsp;the&nbsp;code&nbsp;to&nbsp;fix&nbsp;X,&nbsp;or&nbsp;put&nbsp;in&nbsp;that&nbsp;special&nbsp;little&nbsp;patch<BR>&gt;&nbsp;just&nbsp;for&nbsp;this&nbsp;run&nbsp;(or&nbsp;was&nbsp;it&nbsp;taken&nbsp;out)?&nbsp;&nbsp;Etc.<BR><BR><BR>Hi&nbsp;Rich,<BR><BR>It&nbsp;sounds&nbsp;like&nbsp;you&nbsp;could&nbsp;use&nbsp;a&nbsp;finer-grained&nbsp;version&nbsp;number.&nbsp;&nbsp;And/or&nbsp;a<BR>few&nbsp;more&nbsp;descriptive&nbsp;attributes&nbsp;that&nbsp;mention&nbsp;who&nbsp;is&nbsp;running&nbsp;the&nbsp;code&nbsp;and<BR>what&nbsp;they&nbsp;have&nbsp;or&nbsp;have&nbsp;not&nbsp;have&nbsp;done&nbsp;with&nbsp;it.<BR><BR>But&nbsp;getting&nbsp;back&nbsp;to&nbsp;your&nbsp;idea:&nbsp;for&nbsp;"medium"&nbsp;[1]&nbsp;and&nbsp;larger&nbsp;code&nbsp;bases,<BR>embedding&nbsp;the&nbsp;entire&nbsp;source&nbsp;code&nbsp;into&nbsp;the&nbsp;model&nbsp;output&nbsp;isn't&nbsp;necessarily<BR>a&nbsp;great&nbsp;idea.&nbsp;&nbsp;It&nbsp;doesn't&nbsp;scale.<BR><BR>Instead,&nbsp;have&nbsp;you&nbsp;considered&nbsp;embedding&nbsp;a&nbsp;hash&nbsp;of&nbsp;your&nbsp;model?&nbsp;&nbsp;For<BR>instance,&nbsp;during&nbsp;the&nbsp;model&nbsp;build&nbsp;stage&nbsp;you&nbsp;could&nbsp;easily&nbsp;create&nbsp;a&nbsp;list&nbsp;of<BR>the&nbsp;source&nbsp;files&nbsp;and&nbsp;their&nbsp;MD5SUMs&nbsp;and&nbsp;then&nbsp;embed&nbsp;that&nbsp;list&nbsp;into&nbsp;one&nbsp;or<BR>more&nbsp;output&nbsp;files&nbsp;(which&nbsp;may&nbsp;or&nbsp;may&nbsp;not&nbsp;be&nbsp;in&nbsp;a&nbsp;NetCDF&nbsp;format).<BR>Admittedly,&nbsp;it&nbsp;wouldn't&nbsp;tell&nbsp;you&nbsp;exactly&nbsp;what&nbsp;modifications&nbsp;were&nbsp;made&nbsp;to<BR>the&nbsp;code.&nbsp;&nbsp;But&nbsp;it&nbsp;would&nbsp;tell&nbsp;you&nbsp;which&nbsp;files&nbsp;had&nbsp;been&nbsp;modified&nbsp;and&nbsp;it<BR>would&nbsp;scale&nbsp;to&nbsp;work&nbsp;with&nbsp;much&nbsp;larger&nbsp;projects.&nbsp;&nbsp;And&nbsp;any&nbsp;hash(es)&nbsp;can&nbsp;be<BR>though&nbsp;of&nbsp;as&nbsp;a&nbsp;sort&nbsp;of&nbsp;"extended&nbsp;version&nbsp;number".<BR><BR>If&nbsp;you'd&nbsp;like,&nbsp;I'd&nbsp;be&nbsp;happy&nbsp;to&nbsp;work&nbsp;with&nbsp;you&nbsp;on&nbsp;writing&nbsp;up&nbsp;a&nbsp;quick&nbsp;pair<BR>of&nbsp;hash-list-embed&nbsp;and&nbsp;hash-list-extract&nbsp;utilities.<BR><BR>Ed<BR><BR>[1]&nbsp;lets&nbsp;define&nbsp;"medium"&nbsp;as&nbsp;"big&nbsp;enough&nbsp;to&nbsp;make&nbsp;the&nbsp;output<BR>&nbsp;&nbsp;&nbsp;&nbsp;file(s)&nbsp;unwieldy&nbsp;even&nbsp;when&nbsp;the&nbsp;source&nbsp;code&nbsp;has&nbsp;been<BR>&nbsp;&nbsp;&nbsp;&nbsp;compressed"<BR><BR>--&nbsp;<BR>Edward&nbsp;H.&nbsp;Hill&nbsp;III,&nbsp;PhD<BR>office:&nbsp;&nbsp;MIT&nbsp;Dept.&nbsp;of&nbsp;EAPS;&nbsp;&nbsp;Rm&nbsp;54-1424;&nbsp;&nbsp;77&nbsp;Massachusetts&nbsp;Ave.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Cambridge,&nbsp;MA&nbsp;02139-4307<BR>emails:&nbsp;&nbsp;eh3@xxxxxxx&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ed@xxxxxxx<BR>URLs:&nbsp;&nbsp;&nbsp;&nbsp;<A&NBSP;HREF="HTTP:
 designtimesp="7808" ? eh3 
web.mit.edu>http://web.mit.edu/eh3/</A>&nbsp;&nbsp;&nbsp;&nbsp;<A&NBSP;HREF="HTTP:
 designtimesp="7809" ? 
eh3.com>http://eh3.com/</A><BR>phone:&nbsp;&nbsp;&nbsp;617-253-0098<BR>fax:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;617-253-4464<BR><BR><BR><BR>***PRIVILEGED&nbsp;AND&nbsp;CONFIDENTIAL***&nbsp;<BR>The&nbsp;information&nbsp;contained&nbsp;in&nbsp;this&nbsp;e-mail&nbsp;message&nbsp;(including&nbsp;any&nbsp;attached&nbsp;files)&nbsp;is&nbsp;intended&nbsp;for&nbsp;the&nbsp;use&nbsp;of&nbsp;the&nbsp;addressee(s)&nbsp;only&nbsp;and&nbsp;is&nbsp;privileged&nbsp;information.&nbsp;The&nbsp;information&nbsp;should&nbsp;neither&nbsp;be&nbsp;posted&nbsp;to&nbsp;the&nbsp;Internet,&nbsp;nor&nbsp;published&nbsp;in&nbsp;any&nbsp;other&nbsp;public&nbsp;domain,&nbsp;without&nbsp;the&nbsp;express&nbsp;permission&nbsp;of&nbsp;the&nbsp;sender.&nbsp;If&nbsp;you&nbsp;are&nbsp;not&nbsp;the&nbsp;intended&nbsp;recipient(s)&nbsp;or&nbsp;the&nbsp;recipient's&nbsp;representative,&nbsp;you&nbsp;are&nbsp;hereby&nbsp;notified&nbsp;that&nbsp;any&nbsp;use,&nbsp;disclosure,&nbsp;copying&nbsp;or&nbsp;distribution&nbsp;of&nbsp;this&nbsp;communication&nbsp;is&nbsp;prohibited.&nbsp;If&nbsp;you&nbsp;have&nbsp;received&nbsp;this&nbsp;communication&nbsp;in&nbsp;error&nbsp;please&nbsp;notify&nbsp;us&nbsp;immediately&nbsp;at&nbsp;postmaster@xxxxxxxxxxxxxxxxx,&nbsp;and&nbsp;remove&nbsp;this&nbsp;message&nbsp;from&nbsp;your&nbsp;system.<BR><BR><BR></font></DIV></FONT>