Using NCML to reduce dimensions in netCDF file

StackOverflow https://stackoverflow.com/questions/18857163

  •  28-06-2022
  •  | 
  •  

Question

I'm trying to use NCML to 'convert' a CF-1.4 file to CF-1.6. Of particular interest is how to 1) remove dimensions and then 2) change the dimensions of variables. For example, below are the top portions (ncdump) of two netCDF files. The first is CF-1.4, with dimensions time, z, lat and lon. In this file, variables (e.g., temp) are functions of these four: temp(time,z,lat,lon). I would like, via NCML, to convert this to a CF-1.6 file as shown in the second file, where z/lat/lon are no longer dimensions and variables are functions of time only. Thanks,

File 1:

netcdf wqb_1.4 {
dimensions:
        time = UNLIMITED ; // (109008 currently)
        z = 1 ;
        lat = 1 ;
        lon = 1 ;
variables:
        float time(time) ;
                time:long_name = "Time" ;
                time:standard_name = "time" ;
                time:short_name = "time" ;
                time:axis = "T" ;
                time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
        float z(z) ;
                z:long_name = "depth below mean sea level" ;
                z:standard_name = "depth" ;
                z:short_name = "depth" ;
                z:axis = "z" ;
                z:units = "meters" ;
        float lat(lat) ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
                lat:short_name = "lat" ;
                lat:axis = "Y" ;
                lat:units = "degrees_north" ;
        float lon(lon) ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
                lon:short_name = "lon" ;
                lon:axis = "X" ;
                lon:units = "degrees_east" ;
        float temp(time, z, lat, lon) ;
                temp:long_name = "Temperature" ;
                temp:standard_name = "sea_water_temperature" ;
                temp:short_name = "temp" ;
                temp:units = "Celsius" ;
                temp:coordinates = "time lat lon alt" ;
                temp:valid_range = 10., 35. ;
                temp:_FillValue = -999.f ;
                temp:observation_type = "measured" ;

File 2:

netcdf wqb_1.6 {
dimensions:
        time = UNLIMITED ; // (109008 currently)
        name_strlen = 5 ;
variables:
        char station_name(name_strlen) ;
                station_name:long_name = "wqbaw" ;
                station_name:cf_role = "timeseries_id" ;
        float time(time) ;
                time:long_name = "Time" ;
                time:standard_name = "time" ;
                time:short_name = "time" ;
                time:axis = "T" ;
                time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
        float z ;
                z:long_name = "depth below mean sea level" ;
                z:standard_name = "depth" ;
                z:short_name = "depth" ;
                z:axis = "z" ;
                z:units = "meters" ;
        float lat ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
                lat:short_name = "lat" ;
                lat:axis = "Y" ;
                lat:units = "degrees_north" ;
        float lon ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
                lon:short_name = "lon" ;
                lon:axis = "X" ;
                lon:units = "degrees_east" ;
        float temp(time) ;
                temp:long_name = "Temperature" ;
                temp:standard_name = "sea_water_temperature" ;
                temp:short_name = "temp" ;
                temp:units = "Celsius" ;
                temp:coordinates = "time lat lon alt" ;
                temp:valid_range = 10., 35. ;
                temp:_FillValue = -999.f ;
                temp:observation_type = "measured" ;
Was it helpful?

Solution

Update: The solution below appears to work, but it doesn't: extracting data from it fails, as John M. found out (see other answers). We thought we had figured out that maintaining a singleton dimension was the solution, but going from four dimensions to one dimension ultimately leads to errors. As Sean A. pointed out, you cannot change the shape of variables using NcML.

original "solution" (doesn't actually work):

If your goal was to make your data CF-1.6 compliant, you could make that dimension be station with a value of one. So you could do this:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/usgs/data/file1.nc">
  <remove type="dimension" name="lon"/>
  <remove type="dimension" name="lat"/>
  <remove type="dimension" name="z"/>
  <dimension name="station" length="1"/>
  <dimension name="name_strlen" length="20" />
  <variable name="lat" shape="station"/>
  <variable name="lon" shape="station"/>
  <variable name="z" shape="station"/>
  <variable name="temp" shape="time station"/>
  <variable name="site" shape="station name_strlen" type="char">
    <attribute name="standard_name" value="station_id" />
    <attribute name="cf_role" value="timeseries_id" />
    <values> my_station_001 </values>
  </variable>
  <attribute name="Conventions" value="CF-1.6" />
  <attribute name="featureType" value="timeSeries" />
</netcdf>

OTHER TIPS

Rich's solution will sort of work for this very specific case, but for the wrong reasons. In NcML, you can remove a dimension object, but you cannot reshape your data variables. For this specific case, where you are trying to remove singleton dimensions (size of 1), things will appear to work out because it does not really change the way the data are laid out on disk. For example, if you use Unidata's toolsUI to do an ncdump for the temp variable using the NcML in Rich's answer, you will see that the singleton dimensions are still there and have not really be removed. I'm not sure how this affects the reading of your file - I think it will depend on the client. However, if you try to remove non-singelton dimensions, then this will fail epically.

If you really want to reshape your data correctly, you will have to rewrite your netCDF file. Unfortunately, there are not any "short cuts" to this that I know of. For example, if you use the NcML from Rich in Unidata's toolsUI and try to write out a new file based on it, you will get an error, such as "ERROR: Number of ranges in section (1) must be = 0 for Variable z." This is because the singleton dimensions still exist in the netCDF file, but the NcML file tries to force the range to 0. However, if you know python, then writing a script to rewrite your netCDF file should be pretty straight forward.

Note that the ability to reshape variables using NcML is a feature request that we hear about periodically - it would not hurt to send a feature request to support-netcdf-java@unidata.ucar.edu. Also note that Unidata is a community driven organization, and Rich is on our Users Committee which will be meeting next month. I'd suggest he mention the feature request at the meeting as well.

Cheers!

Sean

Rich is correct, the point of doing this was to try and bring our data up to CF-1.6, and the point of doing that was so we can serve our data via SOS. More specifically, we wanted to use ncSOS (built on TDS), and this particular flavor of SOS requires CF-1.6. In this regard, the modifications via NCML seem to work (plus some additional ones; see below).

I'd prefer not to have to modify datasets, some of which extend back several years. Sean's point about client tools is also pertinent in that many of our use cases involve tools that require variables to have dimensions of lat/lon. Our solution is thus to have one data set served via TDS via two NCML 'wrappers', one for ncSOS the other for those specific clients accessing via OPeNDAP that require lat/lon.

In addition to Rich's suggestion above, in order to work in ncSOS, we had to:

  1. add CF-1.6 'coordinates' attributes ("time lat lon z")
  2. add global attribute "featureType = timeSeries"
  3. add station_name variable
  4. change data type from point to station

The result is given here:

    <attribute name="featureType" value="timeSeries" />
    <remove type="dimension" name="lon"/>
    <remove type="dimension" name="lat"/>
    <remove type="dimension" name="z"/>
    <dimension name="name_strlen" length="4"/>
    <variable name="lat" shape=""/>
    <variable name="lon" shape=""/>
    <variable name="z" shape=""/>
    <variable name="station_name" shape="name_strlen" type="char">
      <attribute name="long_name" value="NS01" />
      <attribute name="cf_role" value="timeseries_id" />
      <values>NS01</values>
    </variable>
    <variable name="temp" shape="time">
      <attribute name="coordinates" value="time lat lon z" />
    </variable>

To follow up on Jim's post above, while Rich's NcML solution appeared to work initially, attempts to grab data via OPeNDAP or ncSOS were unsuccessful, corroborating Sean's skepticism above.

The catalog appears successfully and the OPeNDAP form shows the new dimensions and reshaped variables for CF-1.6. Additionally, the ncSOS GetCapabilities document appears successfully as well.

However, attempting to download some data using the OPeNDAP form has problems. I cannot get a subset of a variable on the OPeNDAP form. For example:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.html

If I try to get the first temp value with this URL:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp[0:1:0]

It gives me this error:

Error {
    code = 500;
    message = "NcSDArray InvalidRangeException=Number of ranges in section (1) must be = 4";
};

The only thing that succeeds is getting all values:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp[0:1:359]

In addition, attempting to grab data via ncSOS GetObservation fails as well. The following URL was attempted:

http://oos.soest.hawaii.edu/thredds-test/sos/hioos/nss/ns01agg.ncml?service=SOS&version=1.0.0&request=GetObservation&responseFormat=text%2Fxml%3Bsubtype%3D%22om%2F1.0.0%22&offering=NS01&observedProperty=temp&procedure=urn:ioos:station:org.pacioos:NS01

This results in the following error message in threddsServlet.log:

2013-10-02T09:03:44.844 -1000 [1288472818][    2602] INFO  - threddsServlet - Remote host: 128.171.151.240 - Request: "GET /thredds-test/sos/hioos/nss/ns01agg.ncml?service=SOS&version=1.0.0&request=GetObs
ervation&responseFormat=text%2Fxml%3Bsubtype%3D%22om%2F1.0.0%22&offering=NS01&observedProperty=temp&procedure=urn:ioos:station:org.pacioos:NS01 HTTP/1.1"
2013-10-02T09:03:44.845 -1000 [1288472819][    2602] INFO  - com.asascience.ncsos.controller.SosController - Handling SOS metadata request.
2013-10-02T09:03:45.614 -1000 [1288473588][    2602] ERROR - ucar.nc2.Structure - Structure.IteratorRank1.readNext()
ucar.ma2.InvalidRangeException: Number of ranges in section (1) must be = 4
    at ucar.ma2.Section.fill(Section.java:144)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:647)
    at ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.read(AggregationOuterDimension.java:774)
    at ucar.nc2.ncml.AggregationOuterDimension.reallyRead(AggregationOuterDimension.java:293)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:553)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:647)
    at ucar.nc2.dataset.StructurePseudoDS.reallyRead(StructurePseudoDS.java:193)
    at ucar.nc2.Variable._read(Variable.java:861)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:619)
    at ucar.nc2.Structure.readStructure(Structure.java:378)
    at ucar.nc2.Structure$IteratorRank1.readNext(Structure.java:464)
    at ucar.nc2.Structure$IteratorRank1.next(Structure.java:447)
    at ucar.nc2.ft.point.PointIteratorFromStructureData.nextStructureData(PointIteratorFromStructureData.java:103)
    at ucar.nc2.ft.point.PointIteratorFromStructureData.hasNext(PointIteratorFromStructureData.java:68)
    at ucar.nc2.ft.point.PointCollectionImpl.calcBounds(PointCollectionImpl.java:128)
    at com.asascience.ncsos.util.DatasetHandlerAdapter.calcBounds(DatasetHandlerAdapter.java:122)
    at com.asascience.ncsos.cdmclasses.TimeSeries.setData(TimeSeries.java:243)
    at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.setCDMDatasetForStations(SOSGetObservationRequestHandler.java:193)
    at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.<init>(SOSGetObservationRequestHandler.java:138)
    at com.asascience.ncsos.service.SOSParser.enhanceGETRequest(SOSParser.java:197)
    at com.asascience.ncsos.controller.SosController.handleSOSRequest(SosController.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:48)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.util.DatasetHandlerAdapter - Could not calculate the bounds of the PointFeatureCollection NS01
Structure.Iterator.readNext()
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.cdmclasses.baseCDMClass - TimeSeries - setData; exception:
java.lang.NullPointerException
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.service.SOSParser - java.lang.NullPointerException
2013-10-02T09:03:45.617 -1000 [1288473591][    2602] ERROR - com.asascience.ncsos.controller.SosController -
2013-10-02T09:03:45.817 -1000 [1288473791][    2602] INFO  - threddsServlet - Request Completed - 200 - -1 - 973:1

NcML now (since version 4.4) has an operation for removing dimensions of length 1, eg:

<variable name="temp">
  <logicalReduce dimNames="lat lon" />
</variable>

see

http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/AnnotatedSchema4.html#logicalReduce

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top