Hi Maria, The size of the data your working on isn't an issue (necessarily) for the core libnetcdf library, which is primarily what we deal with. What I suspect is happening is an issue related to the Python xarray code. I'm not familiar enough with Python, or xarray, to be able to confirm that, or to know if there is a workaround for loading large datasets (as you are doing). The quickest way to find a workable solution would probably be to ask the xarray developers if there is anything that can be done at the xarray level to work with this data. I'm sorry I can't provide immediately actionable advice, but hopefully this will get you started in the right direction. Have a great day! -Ward > Hi, > > I need some help with NetCDF. I am trying to open several NetCDF4 using the > xarray.open_mfdataset function. In my folder I have 62 files and their size > together is like 500MB, but when I combine them in python the size become > huge (18GB). So, I don't know if I am doing something wrong or it is normal > or if there is a way to reduce this size. > > I really appreciate if you could help me with this. My laptop cannot ingest > this size of data so I am a little concerned, specially when I have to > compute all the years... > > Thanks in advance. > Maria > > This is my code (if it could help): > > year = 2011 > path_save = > 'C:/Users/maria/Documents/Arctic/data_original/cdm_globcolour/'+str(year)+'/' > > cdm_files = [] > months = ['06', '07', '08', '09'] > > for month in months: > cdm_files_month = > glob.glob(str(year)+'/'+month+'/*cmems_obs-oc_arc_bgc-optics_my_l3-multi-1km_P1D.nc') > cdm_files = cdm_files + cdm_files_month > > cdm_files.sort() > print(cdm_files) > dset_year = xr.open_mfdataset(cdm_files, concat_dim='time', > combine='nested') > > Ticket Details =================== Ticket ID: NWH-913724 Department: Support netCDF Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.