VAR Accuracy Measurement Using Forecast Accuracy ()

Question

I am trying to learn a vector autoregressive model using the vars package in R. This package has no way to measure the accuracy of the returned model.

In particular, I want to use MASE, as defined in the accuracy function, from the forecast package in R, to compare forecasting with VAR with forecasting using Arima models for each time series of components (I use 4, possibly correlated series time). accuracy does not recognize the varest object returned by vars . How can I get a MASE for each predicted component? I want to calculate both inside the sample and outside the sample

Code example:

 library(vars) library(forecast) data(Canada) v<- VAR(window(Canada, end=c(1998,4)), p=2) accuracy(v$varresult[[1]]) 

The accuracy argument is an lm object and returns training accuracy for series 1 as:

  ME RMSE MAE MPE MAPE MASE Training set 1.536303e-15 0.3346096 0.2653946 -1.288309e-05 0.0281736 0.03914555 

I want to get the accuracy of the test outside the sample, using something like (not quite so, since it is necessary to indicate the forecast period):

  accuracy(v$varresult[[1]], window(Canada[,1], start=c(1999,1))) 

but this is not supported for lm objects and returns an error

  Error in testaccuracy(f, x, test) : Unknown list structure 

And if I use the values ​​directly as follows, I do not get the MASE, which needs information about the training set. It is also prone to "one by one" errors, because values ​​are used, not ts objects, for which accuracy will correspond to stored time points:

  p<-predict(v, n.ahead=8) accuracy(p$fcst[[1]][,"fcst"],window(Canada[,1], start=c(1999,1))) ME RMSE MAE MPE MAPE ACF1 Theil U Test set -0.1058358 0.8585455 0.7385238 -0.01114099 0.07694492 0.5655117 1.359761 

Ideally, I would like to predict this as:

 fr<-forecast(v$varresult[[1]], h=8) 

but this cannot work, because he needs another series for prediction and gives:

 Error in eval(expr, envir, enclos) : object 'datamat' not found 

I could try to copy the forecast.Arima functions, etc. and try writing the forecast.varresult package, but is there an easier way out?

Answer 1

Why don't you try reading the documentation. Here is what he says about the first argument, f :

An object of the “forecast” class or a numerical vector containing forecasts. It will also work with Arima, ets, and lm objects; if x is omitted, then the accuracy measures of the sample are returned.

VAR does not return an object of the "forecast" class, but you can calculate a numerical vector containing forecasts.

Now we read about the second argument x .

An additional numerical vector containing actual values ​​of the same length as the object, or time series that overlap with time f.

Well, that’s pretty simple. Just give it the actual values ​​in x and the predicted values ​​in f .

But this will not give you a MASE, since the further on the help page explains that "the MASE calculation is scaled using MAE from naive forecasts in the sample for off-season time series, seasonal naive seasonality forecasts for seasonal time series and average forecasts in the sample for data, non-time series. " Therefore, he cannot perform this calculation without historical data, and if you do not pass an object of the “forecast” class, he will not know about them.

However, it is not difficult to deceive in order to give what you want. Here is the code he makes:

 trainingdata <- window(Canada, end=c(1998,4)) testdata <- window(Canada, start=c(1999,1)) v <- VAR(trainingdata, p=2) p <- predict(v, n.ahead=8) res <- residuals(v) fits <- fitted(v) for(i in 1:4) { fc <- structure(list(mean=p$fcst[[i]][,"fcst"], x=trainingdata[,i], fitted=c(NA,NA,fits[,i])),class="forecast") print(accuracy(fc,testdata[,i])) }