KIEAE Journal
[ Research Articles ]
Journal of the Korea Institute of Ecological Architecture and Environment - Vol. 16, No. 2, pp.17-22
ISSN: 2288-968X (Print) 2288-9698 (Online)
Print publication date Apr 2016
Received 15 Feb 2016 Revised 23 Mar 2016 Accepted 25 Mar 2016
DOI: https://doi.org/10.12813/kieae.2016.16.2.017

Comparison of Sensitivity Analysis Methods for Building Energy Simulations in Early Design Phases: Once-at-a-time (OAT) vs. Variance-based Methods

Kim, Sean Hay*
*Architectural Engineering Program, Seoul National University of Science and Technology, South Korea seanhay.kim@seoultech.ac.kr


© Copyright Korea Institute of Ecological Architecture and Environment

Abstract

Purpose

Sensitivity analysis offers a good guideline for designing energy conscious buildings, which is fitted to a specific building configuration. Sensitivity analysis is, however, still too expensive to be a part of regular design process. The One-at-a-time (OAT) is the most common and simplest sensitivity analysis method. This study aims to propose a reasonable ground that the OAT can be an alternative method for the variance-based method in some early design scenarios, while the variance-based method is known adequate for dealing with nonlinear response and the effect of interactions between input variables, which are most cases in building energy simulations.

Method

A test model representing the early design phase is built in the DOE2 energy simulations. Then sensitivity ranks between the OAT and the Variance-based methods are compared at three U.S. sites.

Result

Parameters in the upper rank by the OAT do not much differ from those by the Main effect index. Considering design practices that designers would chose the most energy saving design option first, this rank similarity between two methods seems to be acceptable in the early design phase.

Keywords:

Sensitivity Analysis, Building Energy Simulation, Once-at- a-time, OAT, Variance based method, Main effect index

1. Introduction

It is well known that design decisions made during early building design phase greatly impact on the energy performance of the designed building. For more responsive and reliable decisions, using building energy simulation becomes an industry de-facto standard design process; after initial mass study, designers try to find the most energy saving design options over continuous simulation experiments by various design options. Often selecting model parameters and their design options follow the design guidelines [1]. However, there are no universal priority of energy sensitive parameters; a set of energy sensitive parameters varies upon climate, building type, building shape, range of design alternatives, etc.

Sensitivity analysis offers a good guideline that is fitted to a specific building configuration. Although designers know the merit of sensitivity analysis, they don't have enough time for doing sensitivity analysis in every design situation, as well as only simulation experts trained with statistical background can produce reliable sensitivity analysis. Therefore sensitivity analysis is still too expensive to be a part of regular design process.


2. Sensitivity analysis for building energy simulations

Sensitivity analysis is to identify how different types and degrees of model parameters (i.e., input variables) affect the output responses of simulation model. Also it sets up an order by the strength and relevance of the model parameters in determining the variation of output response. Sensitivity analysis is performed by many reasons when mathematical (simulation) model is initially created or it needs to be refined. Concerning building energy simulation model, purposes of sensitivity analysis include:

  • 1) Better understanding of the relationship between model parameters and output responses of the simulation model,
  • 2) Testing the model robustness with presence of diverse sources of uncertainty in the model parameter, and
  • 3) Selecting model parameters with greater impacts or screening model parameters with least impacts and thus simplifying the simulation model or the scope of considerations.

When a modeler creates a new simulation model, sensitivity analysis is useful by the first two reasons. When a designer uses the created model, sensitivity analysis offers a clue on which model parameters the designer needs to first focus.

Prevalent methods of sensitivity analysis are largely classified into five categories as listed as in Table 1. The first three methods are relatively known and have been often referred in literature concerning sensitivity analysis of building energy model. For local derivative method, however, it is quite hard to approximate building energy models by the Taylor series; the regression (not necessarily linear) appears to be more doable and useful for an alternative of the Taylor series.

Frequently employed sensitivity analysis methods in building performance simulations

When the model response tends to be nonlinear and/or strong correlation between model parameters and model interactions are observed, the first four methods are known resulting in inaccurate sensitivity measure. As this limitation is often the case of building energy models, the Variance-based method, a global sensitivity algorithm, is known better suitable for sensitivity analysis of building energy model.

Basically the variance-based method assumes a full exploration of input variable space accounting for interactions and nonlinear responses, which means theoretically it requires a full factorial analysis. It is a big computational burden as well as a huge effort for the evaluator who prepares input data. Since sensitivity analysis of building energy model is most useful at early design phases when a majority of design information is still undecided or not known, evaluators encounter a practical barrier against continuing sensitivity analysis due to a difficulty of data collection and preparation. Although the Variance-based method is known performing better than the first four methods, therefore, in some cases evaluators choose the most simple algorithm such as the OAT (Once-at-a-time) because it could be only available solution for them.

Initiated by this practice often observed in the field, this study aims to propose a reasonable ground that the OAT (the simplest) can be an alternative method for the Variance-based method (the most complex) in some sensitivity analysis scenarios by means of comparing results by the two in the context of practical design case.


3. OAT method

As its name indicates, the model response is measured for each change of each model parameter while the other parameters keep hold, i.e., there is no simultaneous change of multiple parameters, thus no detection of the presence of interactions between input. The OAT typically proceeds as follows:

  • Step 1: Change the value of a model parameter from its base value while keeping the others at their base value.
  • Step 2: When exploring the space of the first model parameter is complete, it is set to its base value back. And then repeat for each of the other model parameters in the same manner.
  • Step 3: Calculate the degree of model response displacement by each model parameter (Equation 1) and then compare the magnitude for all the evaluated model parameters using plots.
    YXi=YXi-YX0Xi-X0(Eq.1) 

    where the model is a function Y = f(Xo, X1, …, Xn), and X0 indicates the vector of base values for X.

Since moving one value of model parameter at a time saves a lot of computational effort and also gives very intrinsic perception of “how it goes” to the evaluator, the OAT is often preferred by evaluators because of its high practicality.

In many cases of sensitivity analysis of building energy model in the design context, values of model parameters tend to be discrete. For instance, U-value of single pane of window glazing does not always smoothly vary when the glazing pane becomes double. From time to time this discrete feature as well as a limited knowledge about value of model parameters in the early design phase makes the exhausted exploration over all the input variables harder. To perform a global sensitivity analysis, therefore, strong simulation expertise and experience in data preparation for compensating this discrete feature are required. In this case practitioners tend to (or are forced to) choose the OAT, rather than the global sensitivity analysis, despite its comprehensive merits.

Design options for each model parameter and their simulated Energy Use Intensity (EUI: MJ/m2/yr) at three sites


4. Variance-based method

The Variance-based method measures sensitivity across the whole input space dealing with nonlinear response, and the effect of interactions between input variables, thus it is a global sensitivity analysis; the “sensitivity” is calculated by the normalized variance of the model response for a specific input variable and “effect” is calculated by decomposing the variance into fractions which can be attributed to inputs or a certain set of inputs.

Decomposing the variance assumes the model is a function Y = f(X), where X is a set of input variables {X1, X2, …, Xn} and Y is the model response. Also the inputs are assumed to be independent and uniformly distributed within the input space, which is to ensure the generality. Then f(X) can be decomposed such as:

fX=f0+f1++fn+f12++fn-1n+f12n(Eq.2) 

where f0 is a constant, fi is a function of Xi, and fij is a function of Xi and Xj.

Equation 2 enables the following functional decompositions with respect to the conditional expected value for each Xi.

f0=EY(Eq.3) 
fiXi=EY|Xi-f0(Eq.4) 
fijXiXj=EY|XiXj-f0-fi-fj(Eq.5) 

Along with Equation 3 to 5, the following decomposition of variance is also enabled.

VarY=V1+···+Vn+V12+···+Vn-1n+V12···n(Eq.6) 
Vi=VarXiEX~iY|Xi(Eq.7) 
Vij=VarXijEX~ijY|Xi,Xi-Vi-Vj(Eq.8) 

where X~i indicates the set of varying all the input variables except Xi.

Eventually Equation 6 demonstrates how the variance of the model response can be decomposed into the terms with respect to each input variable Xi (the first parenthesis) as well as the interaction between input variables (the second parenthesis). It also demonstrates how much Xi contributes “alone” to the total variance of the model output and how much Xi and its interactions with other input variables contributes to the total variance of the model output. The former is called the Main effect index and the latter is called the Total effect index of Xi. Both indexes are typically standardized by dividing by the total variance, i.e., Var(Y).


5. Comparison of the OAT and the Main effect index

While the OAT could be the simplest method of sensitivity analysis, the Variance-based method requires a quite large effort for data preparation and computational expense. This study aims at finding out an opportunity that the OAT can be used alternatively in some design situations such as early design phases. For this experiment, the test set is configured as follows.

5.1. Test configuration

An ideal five story office building with 1500 m2 (50m x 30m) of each floor area is selected for simulations as depicted as in Figure 1. Initially the 50m long side is set to face the South (thus the other side faces the North). It is a regular office operating from 8am to 5pm weekday. It is built in the DOE2 building energy simulation via Autodesk Green Building Studio [4]; the base value includes 40% of the window-wall ratio, R15 roof deck, R13+3.8 metal frame exterior wall, double clear U-SI 0.56, SHGC 0.69, VLT 0.78 window glazing, VAV (Variable Air Volume) with COP 5.0 chiller, 82% gas boiler and VSD (Variable Speed Drive) fans, and no lighting controls. Also its built environment includes 300 occupants, 9.69 W/m2 of LPD (Lighting Power Density), 14.42 W/m2 of EPD (Electric Power Density), and 0.4 ACH of infiltration for perimeter zones.

Additionally to observe sensitivity variations depending on locations, three sites - Miami, FL, San Francisco, CA, and Chicago, IL, which represent the ASHRAE climate zone 1, 3, and 5, respectively - are chosen.

Fig. 1.

The test building faces the South.

Fig. 2.

ΔEUI of the test model at Miami

Fig. 3.

ΔEUI of the test model at San Francisco

Fig. 4.

ΔEUI of the test model at Chicago

5.2. Comparison of the OAT and Main effect indices

Model parameters that are known as “building energy contributors” and typically being decided in the pre-schematic and schematic design phase are selected, and also their design options considering actual industry practices in the U.S. are chosen. These parameters and options are listed in Table 2.

With the EUIs calcuated using the base model and its variants in Table 2, the delta EUIs, i.e., the difference between the base EUI and the alternative EUI, for three site cases are calculated and depicted in Figure 2, 3, and 4. The x axis of charts indicates the design option label for each model parameter (Table 2) while “0” indicates the delta EUIs of the base model is zero.

Comparison of the OAT and the Main effect ranks of the test model at three sites

Only the Main effect index for each model parameter is considered for this time. Since a full factorial analysis requires practically impossible simulation runs (4x3x9x8x5x7x9x6x7x2x2), the Latin hypercube sampling [5] is employed. Sensitivity ranks by the Main effect index for each model parameter are shown in Table 3 and then compared to the OAT ranks.

In Table 3, it is observed that parameters in the upper rank (from 1st to 5th) by the OAT do not much differ from those by the Main effect index. As expected, reducing both internal and external heat gains seems to be the most urgent design action in hot climate (represented by Miami, FL). HVAC performance and reducing heat loss seems to be the most urgent design action in cold climate (represented by Chicago, IL). For San Francisco, CA with mild winter, thus almost no heating required, reducing both internal and external heat gains seems to be the most urgent to reduce cooling energy of an office building in summer.

Considering the design practice that designers would choose the most energy saving design option first and/or would more concern the most energy sensitive design parameter, this rank similarity between two sensitivity analysis methods seems to be acceptable in the early design phase.


6. Discussion and Conclusion

This paper presents a pilot study to demonstrate the OAT can be an alternative for the Variance-based method in the early design phase. As the study is still in its initial stage, some limitations still need to be mentioned;

1) The Total Effect index that takes into account of a serious interaction between model parameters need to be tested against the OAT,

2) More complex base model and/or more design options varying upon design context need to be tested. In particular, the range of design options should be extended as well as the design options resulting in more discrete input value variation should be heavily tested, and

3) Other Variance-based methods such as Metamodeling [6] and FAST (Fourier Amplitude Sensitivity Test) [3] need to be tested.

Sensitivity analysis is a useful tool in making design decisions for energy conscious building design; it produces configuration-specific ranks of design variables than design guidelines, such that designers can more focus on a smaller number of design variables. Unfortunately it is still difficult for most building design practitioners and also still expensive in terms of computation cost and analysis expertise. If a simpler and cheaper method such as the OAT would not seriously distort the result by more accurate, but more expensive sensitivity analysis methods, suggesting usecases and scenarios for this alternative method would enhance the productivity of design practitioners.

Announcement

This study was supported by the Research Program funded by the Seoul National University of Science and Technology.

References

  • ASHRAE Advanced Energy Design Guides, ASHRAE, https://www.ashrae.org/standards-research--technology/advanced-energy-design-guides.
  • Morris, M. D., (1991), Factorial sampling plans for preliminary computational experiments, Technometrics, 33, p161-174. [https://doi.org/10.1080/00401706.1991.10484804]
  • Saltelli, A., K. Chan, and M. Scott (Eds.), (2000), Sensitivity Analysis. Wiley Series in Probability and Statistics, New York, John Wiley and Sons.
  • Autodesk Green Building Stuido, Autodesk, http://gbs.autodesk.com.
  • Iman, R.L., Davenport, J.M., and Zeigler, D.K., (1980), Latin hypercube sampling (program user's guide).
  • Storlie, C.B., Swiler, L.P., Helton, J.C., and Sallaberry, C.J., (2009), Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models, Reliability Engineering & System Safety, 94(11), p1735-1763. [https://doi.org/10.1016/j.ress.2009.05.007]

Fig. 1.

Fig. 1.
The test building faces the South.

Fig. 2.

Fig. 2.
ΔEUI of the test model at Miami

Fig. 3.

Fig. 3.
ΔEUI of the test model at San Francisco

Fig. 4.

Fig. 4.
ΔEUI of the test model at Chicago

Table 1.

Frequently employed sensitivity analysis methods in building performance simulations

Sensitivity Analysis Method Description Measure of sensitivity
One-at-a-time (OAT) • The most common and simplest method
• Simply observing how much the model response changes by the value change of a model parameter.
Displacement from nominal output by each movement of a model parameter
Screening • Identify which model parameters contribute on the variation of the model response by how much degree.
• Typically used in a preliminary analysis to filter out uninfluential model parameters for shake of analysis simplification.
Elementary effect [2]
Regression • Fitting a linear regression to the model response using standardized regression coefficients.
• Multiple linear regression is often employed.
• Goodness of fit is measured by R2 and the pattern of residuals.
R2 and regression coefficient such as P or F value
Local derivative method • Using a Taylor series to approximate the model, then compute partial derivatives of the output variable with respect to the change of the input variables.
• The derivative is taken at a certain fixed point in the space of the input variable, thus it is called local model.
Normalized partial derivative of each model parameter
Variance-based method • Decomposing the variance of the output variable into parts attributable to model parameters and combinations of model parameters. Main effect index [3]
Total effect index [3]

Table 2.

Design options for each model parameter and their simulated Energy Use Intensity (EUI: MJ/m2/yr) at three sites

Model parameters Design options with label EUI at Miami EUI at San Francisco EUI at Chicago
Base BA0 687.0 506.9 730.2
Building orientation BO1 45° clockwise 728.7 534.4 759.4
BO2 90° clockwise 721.6 534.6 758.9
BO3 135° clockwise 731.9 536.8 761.7
Size of non-North facing windows SW1 50% of the base size 647.2 468.9 680.7
SW2 150% of the base size 727.4 553.4 810.0
Exterior wall construction EW1 Uninsulated 688.3 529.3 798.3
EW2 R13 Metal frame 688.8 515.0 754.2
EW3 R13 Wood frame 694.2 513.3 732.3
EW4 R10+R13 Metal frame 681.7 492.2 698.8
EW5 R38 Wood frame 684.6 494.0 692.1
EW6 R2 CMU (Concrete Masonry Unit) 684.3 515.6 769.0
EW7 14” ICF (Insulated Concrete Form) 684.5 496.4 698.1
EW8 12.25” SIP (Structurally Insulated Panel) 682.9 492.5 689.6
Roof construction RC1 Uninsulated 738.4 635.2 884.9
RC2 R10 Wood frame 688.7 511.9 738.9
RC3 R15 Wood frame 687.3 507.2 730.3
RC4 R19 Wood frame 688.8 510.8 734.4
RC5 R38 Wood frame 686.9 503.7 718.4
RC6 R60 Wood frame 686.3 502.2 714.8
RC7 10.25” SIP 685.6 502.4 717
Window glazing WG1 Single clear low iron 740.5 566.7 853.6
WG2 PPG double pane 575.9 413.8 587
WG3 Bronze low e triple pane 562.6 411.3 584
WG4 Low e quadruple pane 629.6 423.8 569.7
Air tightness (Infiltration) AT1 0.17 ACH 685.6 502.2 705.2
AT2 0.4 ACH 687 506.9 730.2
AT3 0.8 ACH 689.4 517 772.1
AT4 1.2 ACH 692.3 528.9 816.4
AT5 1.6 ACH 694.2 544.4 864.5
AT6 2 ACH 696.9 560.5 919.3
HVAC HV1 PSZ (Packaged Single Zone): EER 11; AFUE 78%; economizer; VSD fan 587 433.1 554.2
HV2 PSZ: EER 20; FUE 85%; cycling fan 532.9 440.7 536.4
HV3 HP (Heat Pump): EER 9.5; Heating COP 3.2; economizer; VSD fan 621.4 427.8 510.2
HV4 HP: SEER 17; 9.6 HSPF; high eff. Fan 535.5 417.9 476.9
HV5 VAV: COP 6.1; 82% gas boiler; economizer; VSD fan 670.7 509.7 818.4
HV6 UFAD (UnderFloor Air Distribution): COP 7.5;95% gas boiler; economizer; VSD fan&pump 543.4 537.3 808.3
HV7 PTHP (Packaged Heat Pump): EER 11.9 459.7 390.6 453.7
HV8 PTAC (Packaged Air Conditioner): EER 12.7; 94% gas boiler 486.3 454.9 558.1
Lighting Power Density LP1 0.3 W/ft2 (3.2 W/m2) 578.3 438.1 675.5
LP2 0.7 W/ft2 (7.5 W/m2) 651.6 482.1 709.6
LP3 1.1 W/ft2 (11.8 W/m2) 723.3 531.9 751.7
LP4 1.5 W/ft2 (16.1 W/m2) 795.4 583.8 799.1
LP5 1.9 W/ft2 (20.5 W/m2) 868.9 637 848.5
Electric Power Density EP10.6 W/ft2 (6.5 W/m2) 593.9 453.7 688.9
EP2 1 W/ft2 (10.8 W/m2) 644.3 480.2 708.9
EP3 1.3 W/ft2 (14.0 W/m2) 682 503.7 727.7
EP4 1.6 W/ft2 (17.2 W/m2) 720.3 528.6 747.8
EP5 2 W/ft2 (21.5 W/m2) 772.3 564 777.5
EP6 2.6 W/ft2 (28.0 W/m2) 851.2 618.1 824.2
Occupancy Sensor OS1 On 668.5 493.9 720.0
Daylight Controls DC1 On 675.9 500.8 726.0

Table 3.

Comparison of the OAT and the Main effect ranks of the test model at three sites

Model parameters Miami San Francisco Chicago
Main effect rank OAT Main effect rank OAT Main effect rank OAT
Building orientation 8 (3%) 7 9 (2%) 9 9 (2%) 9
Size of non-North facing windows 6 (4%) 5 6 (6%) 6 7 (5%) 7
Exterior wall construction 6 (4%) 9 7 (5%) 8 8 (4%) 8
Roof construction 5 (7%) 6 3 (15%) 5 3 (12%) 4
Window glazing 4 (9%) 4 2 (18%) 3 2 (20%) 2
Air tightness 9 (2%) 10 8 (3%) 7 5 (7%) 5
HVAC 3 (19%) 3 3 (15%) 4 1 (31%) 1
LPD 1 (26%) 1 1 (19%) 1 4 (11%) 3
EPD 2 (21%) 2 3 (15%) 2 6 (6%) 6
Occupancy Sensor 8 (3%) 8 9 (2%) 10 10 (1%) 10
Daylight Controls 9 (2%) 11 11 (1%) 11 11 (1%) 11