[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #NWH-913724]: open_mfdataset doubt

Hi Maria,

The size of the data your working on isn't an issue (necessarily) for the core 
libnetcdf library, which is primarily what we deal with.  What I suspect is 
happening is an issue related to the Python xarray code.  I'm not familiar 
enough with Python, or xarray, to be able to confirm that, or to know if there 
is a workaround for loading large datasets (as you are doing).  The quickest 
way to find a workable solution would probably be to ask the xarray developers 
if there is anything that can be done at the xarray level to work with this 
data.  

I'm sorry I can't provide immediately actionable advice, but hopefully this 
will get you started in the right direction.

Have a great day!

-Ward

> Hi,
> 
> I need some help with NetCDF. I am trying to open several NetCDF4 using the
> xarray.open_mfdataset function. In my folder I have 62 files and their size
> together is like 500MB, but when I combine them in python the size become
> huge (18GB). So, I don't know if I am doing something wrong or it is normal
> or if there is a way to reduce this size.
> 
> I really appreciate if you could help me with this. My laptop cannot ingest
> this size of data so I am a little concerned, specially when I have to
> compute all the years...
> 
> Thanks in advance.
> Maria
> 
> This is my code (if it could help):
> 
> year = 2011
> path_save =
> 'C:/Users/maria/Documents/Arctic/data_original/cdm_globcolour/'+str(year)+'/'
> 
> cdm_files = []
> months = ['06', '07', '08', '09']
> 
> for month in months:
> cdm_files_month =
> glob.glob(str(year)+'/'+month+'/*cmems_obs-oc_arc_bgc-optics_my_l3-multi-1km_P1D.nc')
> cdm_files = cdm_files + cdm_files_month
> 
> cdm_files.sort()
> print(cdm_files)
> dset_year = xr.open_mfdataset(cdm_files, concat_dim='time',
> combine='nested')
> 
> 


Ticket Details
===================
Ticket ID: NWH-913724
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.