I am comparing models on a time series: Exponential Smoothing (Simple, Additive and Multiplicative Trend, Additive and Multiplicative Seasonal, and other combinations), ARIMA (with Python Auto_ARIMA), and a Simple Moving Average.
My concern is auto_arima optimizes parameters by minimizing the AIC (can be switched to BIC, hqic, oob). Exponential Smoothing minimizes the SSE (Sum of Squared Errors).
With them minimizing different measures, what should I use to select the lowest forecast error in model selection?
I was initially choosing the best model based on MAPE, but then I took a look at several simple exponential smoothing outputs between the optimized value for alpha/smoothing_level, and other manually inputted values like 0.4, 0.6, 0.8. What I found was the 'Optimized value' based on minimzing SSE had higher MAPE than the model instances that used alpha values I had specifically defined, whereas other error measures like the MAE, MSE, RMSE of the optimized alpha were lower. Thus, seeing that the optimized alpha produced lower of the other error measures but higher MAPE, it made me want to look for other measures.
I tried the MASE (Mean Absolute Scaled Error) (Hyndman, 2006) which was described to be appropriate against the limitations of scale-dependent errors like MAE, and percentage errors like MAPE, mainly on time series with intermittent demand or having 0 values. But I was confused because what was initially the 'best model' from my runs where I select based on lowest MAPE, would come to have a MASE > 0.90. According to Hyndman from the same article, below 1 would mean that it is better than the Naive one-step forecasts, and higher than that would worse than the naive forecasts. But also, one-step forecasts would usually have MASE < 1.0, and "Multistep MASE values are often larger than one, as it becomes more
difficult to forecast as the horizon increases." I am performing multi-step forecasts, so do I assume that 0.90 is an alright error on the best model?
I may also consider the MAE, since I am only forecasting one series at a time, and not comparing across multiple series so it does not fall under the limitation of scale-dependent errors mentioned in (Hyndman, 2006).
Nagusameta t1_ittdqfy wrote
Reply to [D] Simple Questions Thread by AutoModerator
I am comparing models on a time series: Exponential Smoothing (Simple, Additive and Multiplicative Trend, Additive and Multiplicative Seasonal, and other combinations), ARIMA (with Python Auto_ARIMA), and a Simple Moving Average.
My concern is auto_arima optimizes parameters by minimizing the AIC (can be switched to BIC, hqic, oob). Exponential Smoothing minimizes the SSE (Sum of Squared Errors).
With them minimizing different measures, what should I use to select the lowest forecast error in model selection?
I was initially choosing the best model based on MAPE, but then I took a look at several simple exponential smoothing outputs between the optimized value for alpha/smoothing_level, and other manually inputted values like 0.4, 0.6, 0.8. What I found was the 'Optimized value' based on minimzing SSE had higher MAPE than the model instances that used alpha values I had specifically defined, whereas other error measures like the MAE, MSE, RMSE of the optimized alpha were lower. Thus, seeing that the optimized alpha produced lower of the other error measures but higher MAPE, it made me want to look for other measures.
I tried the MASE (Mean Absolute Scaled Error) (Hyndman, 2006) which was described to be appropriate against the limitations of scale-dependent errors like MAE, and percentage errors like MAPE, mainly on time series with intermittent demand or having 0 values. But I was confused because what was initially the 'best model' from my runs where I select based on lowest MAPE, would come to have a MASE > 0.90. According to Hyndman from the same article, below 1 would mean that it is better than the Naive one-step forecasts, and higher than that would worse than the naive forecasts. But also, one-step forecasts would usually have MASE < 1.0, and "Multistep MASE values are often larger than one, as it becomes more
difficult to forecast as the horizon increases." I am performing multi-step forecasts, so do I assume that 0.90 is an alright error on the best model?
I may also consider the MAE, since I am only forecasting one series at a time, and not comparing across multiple series so it does not fall under the limitation of scale-dependent errors mentioned in (Hyndman, 2006).