How to handle NULLS: C #, Microsoft SDS 1.3 and NetCDF files

Question

I am writing a C # program that uses the Microsoft Scientific Data-Set to read NetCDF files.

using System; using System.IO; using sds = Microsoft.Research.Science.Data; using Microsoft.Research.Science.Data.Imperative; namespace NetCDFConsoleApp { class Program { static void Main(string[] args) { // Gets dataset from file. var dataset = sds.DataSet.Open("E:\\Temp\\test.nc?openMode=readOnly"); // Get the starting DateTime from the meta data. string dt = (string)dataset.Metadata["START_DATE"]; //load dataset into array Single[,,] dataValues = dataset.GetData<float[,,]>("ACPR"); //Get DateTime from Metadata fields. DateTime dt2 = DateTime.ParseExact(dt, "yyyy-MM-dd_HH:mm:ss", null); // Latitude grid ranges from = 0 to 215; East Cape is ~ 125-144 for (int iLatitude = 137; iLatitude < 138; iLatitude++) { //Longitude ranges from 0 to 165; East Cape is ~ 125-150 for (int iLongitude = 133; iLongitude < 134; iLongitude++) { //There is normally 85 hours worth of data in a file. But not always... for (int iTime = 0; iTime < 65; iTime++) { // Get each data point float? thisValue = dataValues[iTime,iLatitude,iLongitude]; //Burp it out to the Console. Increment the datetime while im at it. Console.WriteLine(dt.ToString() + ',' + dt2.ToString() + ',' + iTime.ToString() + ',' + dt2.AddHours(iTime) ); } } } Console.ReadLine(); } } } 

Files contain predicted precipitation data on a map grid (X, Y). Each grid link must contain data for 85 hours.

 E:\temp>sds list test.nc [2] ACPR of type Single (Time:85) (south_north:213) (west_east:165) [1] Times of type SByte (Time:85) (DateStrLen:19) 

But sometimes they may have less (say 60-70 hours). When this happens, my C # programs do not work when importing data.

 var dataset = sds.DataSet.Open("test.nc?openMode=readOnly"); Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR"); 

I can reproduce the error on the command line.

Here I can successfully extract the clock 60-65 for Grid XY: 125 130. The last value that I have in this file is Time = 69.

 E:\temp>sds data test.nc ACPR[60:65,125:125,130:130] [2] ACPR of type Single (Time:85) (south_north:213) (west_east:165) Name = ACPR description = ACCUMULATED TOTAL GRID SCALE PRECIPITATION MemoryOrder = XY coordinates = XLONG XLAT XTIME stagger = FieldType = 104 units = mm [60,125,130] 13.4926 [61,125,130] 15.24556 [62,125,130] 16.3638 [63,125,130] 17.39618 [64,125,130] 20.00507 [65,125,130] 23.57192 

If I try to read past hour 69, I get the following error.

 E:\temp>sds data test.nc ACPR[60:70,125:125,130:130] [2] ACPR of type Single (Time:85) (south_north:213) (west_east:165) Name = ACPR description = ACCUMULATED TOTAL GRID SCALE PRECIPITATION MemoryOrder = XY coordinates = XLONG XLAT XTIME stagger = FieldType = 104 units = mm Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at nc_get_vara_float(Int32 , Int32 , UInt64* , UInt64* , Single* ) at NetCDFInterop.NetCDF.nc_get_vara_float(Int32 ncid, Int32 varid, IntPtr[] start, IntPtr[] count, Single[] data) at Microsoft.Research.Science.Data.NetCDF4.NetCdfVariable`1.ReadData(Int32[] origin, Int32[] shape) at sdsutil.Program.PrintData(Variable v, String range, String format) at sdsutil.Program.DoData(String uri, String[] args) at sdsutil.Program.Main(String[] args) E:\temp> 

If the file contains all 85 hours, I can request Time 0-100, and it still gives me 85 values ​​without errors.

I am convinced that this problem is NULL / missing data. Is there a way I can specify when importing data where the variable is not null? or use some type of try / catch?

 Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR")>> where it not blank thanks. ; 

Edit: I am starting to suspect that the file is not formed correctly. Using the SDS Viewer The meta data for a good file and a bad one look like this:

However, on the command line, metadata is the same for both.

 E:\temp>sds good.nc [2] ACPR of type Single (Time:85) (south_north:213) (west_east:165) [1] Times of type SByte (Time:85) (DateStrLen:19) E:\temp>sds bad.nc [2] ACPR of type Single (Time:85) (south_north:213) (west_east:165) [1] Times of type SByte (Time:85) (DateStrLen:19) E:\temp> 

Answer 1

Peter

Since the error is indicated in ReadData (Int32 [] origin, Int32 [] shape) (you specified the same thing); I see two possible solutions:

Before delving into the solution, you need to decide whether the missing data can be treated as 0.0 or whether it should be considered as missing. If a value other than 0.0 is missing, then a potentially missing value may be encoded as -1.0 if null is unacceptable. Presenting a value of -1.0 for missing data suggests that a negative rainfall value is not possible.

If the result of the dataValues contains zeros, is potentially all you have to do is replace float with float? in line:

 float thisValue = dataValues[iTime,iLatitude,iLongitude]; 

:

 float? thisValue = dataValues[iTime,iLatitude,iLongitude]; 

And if you are free at home with a float? then it was a happy decision. (You still need to decide how to handle the null values.)

Otherwise, a possible solution is 1)

After calling Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR"); make sure that the last index size of the dataValues array is 85. Potentially, GetData (..) does not populate all 85 fields, especially if the data in the first row contains less than 85 fields. Then, if necessary, manually replace the zeros with 0 or -1.0.

Then, when you retrieve the data, you process zeros, 0 or -1.0 respectively:

 float? thisValue = dataValues[iTime,iLatitude,iLongitude]; // determine what to do with a null/0.0/-1.0 as a thisValue[..] value, // .. potentially continue with the next iteration 

Possible solution 2)

If you have a GetData (..) method in Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR"); , then you will see that it GetData(..) sets all 85 values ​​and missing values ​​as nulls / 0 / -1.0. Then, when you retrieve the data, you process zeros, 0 or -1.0 respectively.

Greetings

Avi

Answer 2

I recommend that you try this, since you do not know the data type that it is trying to return:

 Object[,,] dataValues = dataset.GetData<object[,,]>("ACPR"); 

Then you can check if you have a valid float in the loop.

 if ( dataValues[iTime,iLatitude,iLongitude] == null ) { float floatValue = 0; if (Single.TryParse(dataValues[iTime,iLatitude,iLongitude].ToString(), out floatValue) { Console.WriteLine(dt.ToString() + ',' + dt2.ToString() + ',' + iTime.ToString() + ',' + dt2.AddHours(iTime) ); } }