Energy consultants and auditors use energy modeling software for a variety of purposes, including rating the performance of an existing house, calculating the effect of energy retrofit measures, estimating the energy use of a new home, and determining the size of new heating and cooling equipment. According to most experts, the time and expense spent on energy modeling is an excellent investment, because it leads to better decisions than those made by contractors who use rules of thumb.
Yet Michael Blasnik, an energy consultant in Boston, has a surprisingly different take on energy modeling. According to Blasnik, most modeling programs aren’t very accurate, especially for older buildings. Unfortunately, existing models usually aren’t revised or improved, even when utility bills from existing houses reveal systematic errors in the models.
Most energy models require too many inputs, many of which don’t improve the accuracy of the model, and energy modeling often takes up time that would be better spent on more worthwhile activities. Blasnik presented data to support these conclusions on March 8, 2012, at the NESEA-sponsored Building Energy 12 conference in Boston.
Blasnik sees more data in a day than most raters do in a lifetime
Blasnik has worked as a consultant for utilities and energy-efficiency programs all over the country. “I bought one of the first blower doors on the market,” Blasnik said. “I’ve been trying to find out how to save energy in houses for about 30 years. I’ve spent a lot of time looking at energy bills, and comparing bills before and after retrofit work is done. I’ve looked at a lot of data. Retrofit programs are instructive, because they show how the models perform.”
According to Blasnik, most energy models do a poor job of predicting actual energy use, especially for older houses. And since large datasets show that the differences between the models and actual energy use are systematic, we can’t really blame the occupants; we have to blame the models.
Blasnik isn’t the only researcher to note that most energy models do a poor job with existing houses. Blasnik cited several other researchers who have reached the same conclusion, including Scott Pigg, whose found that REM/Rate energy-use predictions are, on average, 22% higher than the energy use shown on actual energy bills.
Retrofit studies are consistent: projected savings are overestimated
Blasnik cited five studies that found that the measured savings from retrofit work equal 50% to 70% of projected savings. “The projected savings are always higher than the actual savings,” said Blasnik, “whether you are talking about insulation retrofit work, air sealing, or lightbulb swaps.”
So why do energy-efficiency programs almost always overestimate anticipated savings? The main culprit, Blasnik said, is not the takeback (or rebound) effect. Citing data from researchers who looked into the question, Blasnik noted, “People don’t turn up the thermostat after weatherization work. References to the takeback effect are mostly attempts to scapegoat the occupants for the energy model deficiencies.”
Many assumptions, inputs, simplifications, and algorithms are bad
The biggest errors occur in modeling estimates of energy use in older homes. “Post-retrofit energy use is pretty close to modeled estimates,” said Blasnik, “but pre-retrofit use is dramatically overestimated because of poor assumptions, biased inputs, and bad algorithms.”
Poor assumptions. “Models and auditors underestimate the efficiency of existing heating equipment,” said Blasnik. “They often assume 60% efficiency for old furnaces.”
Low R-value estimates for existing walls (R-3.5) and attics. “They also use lots of biased defaults,” said Blasnik. “They assume R-3.5 for an old wall, when many old walls actually perform at R-5 or R-6.” Energy models often underestimate the effects of a high framing factor, thick sheathing, and multiple layers of old siding, all of which improve a wall’s R-value.
Low R-value estimates for existing single-pane windows. “They assume that old single-pane windows are R-1, when they are probably closer to R-1.35 or R-1.4. When calculating the outside surface film coefficient, they assume worst-case conditions — in other words, that the wind is always blowing away heat from the window. They do it that way because the design load is always calculated for the coldest, windiest day of the year (even though the coldest day usually isn’t windy). If an auditor calculates single-pane windows at R-1, he’s assuming that the wind is blowing continuously nonstop all winter long. But in a real house, the wind speed is often close to zero up against the window.”
Low or absent estimates for thermal regain. Blasnik explained that energy models underestimate thermal regain from basements and crawlspaces. “Most models get big things wrong, like how basements and crawlspaces work,” he said. “Vented crawl spaces usually aren’t at the outdoor temperature. When the outdoor temperature is 10 degrees, a vented crawl space can be at 50 degrees. Why is it that when you insulate a basement ceiling, you get very little savings — maybe zero savings, or maybe $20 a year? Well, if you have a furnace and ductwork in the basement, you are regaining a lot of the heat given off by the furnace and ducts, due to the directional nature of air leakage in the wintertime. The stack effect brings basement air upstairs. The basement is pretty warm, so the air leaking into the house is warmer than the models predict. A similar effect happens in attics: because of the stack effect, most of the air leaving the house leaves through the attic. In a leaky house, you might have 200 cfm of air flow being dumped into the attic. That makes the attic warmer than the models predict. If the attic is 50 degrees, the heat loss through the ceiling insulation is less than the model assumes.”
Models also ignore interactions between air flow and conduction. “Every single house acts like an HRV, since outdoor air flowing through walls is picking up some of the heat that is leaving the house,” said Blasnik. “The heat exchange is always going on, but it’s not being quantified or accounted for. Complicated models use algorithms for air infiltration that aren’t very good — the infiltration and conduction interactions aren’t modeled.”
Too many inputs
Anyone designing a computer model has to decide which inputs to require. “The trouble with the complicated models is that they ask for inputs that you can’t measure well,” said Blasnik. “After all, a lot of people don’t even know which orientation is south. Unfortunately, many existing models ask for inputs that are difficult to assess — for example, window shading percentages, wind exposure ratings, and soil conditions. What’s the water table height? What’s the flow rate of the water? Who knows?”
As Blasnik noted, “It’s hard enough to get auditors to agree on the area of a house.”
Many models ask for inputs which are open to interpretation. Blasnik asked, “How do you decide if a basement is conditioned or unconditioned? Perhaps it’s semi-conditioned? Or unintentionally conditioned? Or maybe unintentionally semi-unconditioned?”
When making these types of assessments, it’s hard for technicians to avoid unintentional bias. Technicians entering pre-retrofit information on an older home often come up with pessimistic R-value estimates for existing insulation levels, leading to overestimated savings projections.
Because of these problems with input accuracy, default assumptions are often more accurate than data collection. But even when using a model with the best possible default assumptions, there are limitations to accuracy. “Houses are complicated, and that’s a problem,” said Blasnik. “Lots of factors are difficult to model: foundation heat loss, infiltration, wall heat loss, attic heat loss, framing factors, edge effects, window heat loss, window heat gain, exterior shading, interior shading, the effect of insect screens, air films, HVAC equipment performance, duct efficiency and regain, AC refrigerant charge, and air flows over HVAC coils. There are many unknowns: soil conductivity and ground temperatures are unknown. Wind speed is unknown. Leak locations are unknown.”
The good news: energy models do a better job with newer homes
Because newer homes tend to have lower rates of air leakage and higher R-values than older homes, energy models usually do a better job of predicting energy use in newer homes.
A study of 10,258 recently built Energy Star homes in Houston showed that the median discrepancy between the REM/Rate prediction and the actual energy use in the homes was 17%. In other words, in half of the homes the discrepancy between the modeled and actual energy use was 17% or less; in the rest of the homes, the discrepancy was greater.
Is energy modeling cost-effective?
Blasnik noted the irony that energy experts who analyze the cost-effectiveness of window replacement or refrigerator swaps haven’t bothered to calculate the cost-effectiveness of energy modeling.
“How do the time, effort, and costs of collecting detailed data and using complicated models compare to the benefits?” Blasnik asked. “For most residential retrofits, it is hard to justify the cost of a detailed model that takes more than a few minutes to fill out. It makes more sense to just fix the obvious problems instead of doing a detailed modeling exercise. Data collection work distracts you from other tasks. Often raters spend so much time filling out the audit software that they never talk to the occupants — the homeowner is just sitting there. So here’s an idea: maybe you could talk to the homeowners.”
Blasnik even questions the wisdom of modeling new homes. “If you are building super-efficient homes, the heating usage will be dominated by hard-to-model factors, including internal gains like light bulbs and plug loads,” said Blasnik. “Small changes make a significant difference. Do the owners have a few big dogs? How long does the bathtub water sit before it drains down the pipes? Are the shading calculations accurate? What about internal shading by the occupants? How clean are the windows? How big a swing in indoor temperature will the occupants accept? Most models pay close attention to heating use, but in a super-efficient home, the hot water load and plug loads are bigger than the heating load — these other loads dominate. One large-screen plasma TV may matter more than the thickness of the foam insulation under the slab.”
A study compares energy models
Energy Trust of Oregon is an independent nonprofit organization that sponsors a variety of energy-efficiency programs; its work is funded by public benefit charges tacked onto ratepayers’ electric bills. “In 2008, the Energy Trust of Oregon was aiming to come up with a low-cost energy rating for homes,” said Blasnik. “The question was, is there a low-cost alternative to paying $600 for a HERS rating of a house? Is there such a thing as a $100 energy rating — a ‘light’ energy rating?”
To help answer the question, the Energy Trust hired Blasnik to give advice on which energy models to test. When the team couldn’t identify a promising simplified model, Blasnik offered to develop a spreadsheet that would be easier to use than existing energy models. Dubbed the Simple spreadsheet, Blasnik’s creation required only 32 inputs and less operator knowledge than other energy models. Blasnik explained, “The spreadsheet was quickly designed to see if a simpler tool could work OK. The model asks for the conditioned floor area and number of stories, but it doesn’t ask you the area of the windows, walls, or attic. The model doesn’t want to know R-values for the walls or attic, or what kind of windows you have. No blower door or duct leakage numbers are necessary.” Instead of requiring R-value inputs, the Simple spreadsheet asks a technician to choose from a limited menu of options — for example, options like “some insulation,” “standard insulation,” or “average airtightness.” (See Image 3, below, for a list of the Simple spreadsheet inputs.)
A research project called the compared Blasnik’s Simple spreadsheet to three well-established energy models: REM/Rate and two versions of Home Energy Saver, dubbed Home Energy Saver (full) and Home Energy Saver (mid). The Home Energy Saver models were developed by the Department of Energy and Lawrence Berkeley National Laboratory. “The Simple spreadsheet has 32 data inputs,” said Blasnik. “This compares to 185 data inputs required for the full Home Energy Saver model.”
The three energy models made energy use projections for 300 existing houses, and these projections were then compared to actual energy bills. The Simple spreadsheet performed better in most situations; it had the smallest average error and far fewer cases with large errors. The mean absolute percentage error for the four energy models were:
- Simple, 25.1%
- Home Energy Saver (full), 33.4%
- REM/Rate, 43.7%
- Home Energy Saver (mid), 96.6%.
“My dumb spreadsheet does better than REM/Rate and the other models because the other models are horrible. For predicting gas use in older homes, REM/Rate had a median error of 85%. Two-thirds of the REM/Rate houses had huge errors. The mean actual use of gas was 617 therms a year, but REM/Rate predicted 1,089 therms. My Simple spreadsheet overpredicted by only 27 therms.”
Blasnik said, “The other models are very sophisticated, but they focus on the wrong areas. The moral is to get the big stuff right, and don’t waste your time with the other stuff. You can get worse answers if you collect more data than if you just make reasonable default assumptions. These detailed models are precise but not accurate — so they miss the target. The simplified models are accurate but not precise. It is better to be approximately right than precisely wrong.”
Unfortunately, Blasnik’s Simple spreadsheet is not available. However, an energy modeling tool based on Blasnik’s Simple algorithms has been developed; it just isn’t particularly easy to purchase. The software, , is available from Earth Advantage Institute in Portland, Oregon; that catch is that in order to be eligible to purchase the software, you must be a certified BPI analyst. Once you’ve obtained your BPI certification, you still can’t get the EPS software until you complete an additional multistage training program that includes a 5-hour online class, a 3-hour Webinar, and a final exam. The cost for the whole EPS package (training and software) is $199 for individual users.
Remember, it’s a house, not a science project
Blasnik reminds energy nerds that not every house needs to be a science project. “For energy retrofits, don’t waste your time doing simulations with dozens of inputs,” he said. “Do the obvious stuff. Just fix the leaky uninsulated house — don’t model it. If you need a computer to find out what work you need to do, then you don’t know the answer — no matter what the computer says. There are more important issues that come up in a retrofit project, like: Do we have people who know how to do the work? Will they do the work well?”
Energy nerds can get distracted by modeling and testing. “Bruce Manclark, an energy consultant working with Puget Sound Energy, realized that their duct-sealing program would have been cost-effective if only they didn’t have to do Duct Blaster testing before and after the sealing,” said Blasnik. “So Bruce said, ‘Let’s not test them.’ He called it the ‘Duct Ninja’ program. He recommended that workers just start sealing — seal the air handler and then seal every single duct connection you can access, without any testing. That way you don’t need testing equipment or training in using testing equipment, and you don’t need to spend hours testing. A lot of us are getting distracted by tests and computer software. What we really need are efficient processes to improve homes.”
Experienced energy retrofit workers rarely rely on models. “When we make retrofit decisions, other factors like experience are more important than modeling,” said Blasnik. “Even if you need modeling to make design decisions, you don’t have to model every house. Model something well just once, and then apply the lesson to lots of buildings. If a house isn’t unique, modeling is a waste of time.”
What about PHPP?
Blasnik’s analysis raises important questions about the need for fine details in residential energy models. Passivhaus designers are on the opposite end of the spectrum from Blasnik; the software used by Passivhaus designers (PHPP) is so complicated that most energy consultants don’t attempt to use it without first taking nine days of classroom training.
For example, consider this formula used to calculate window heat losses in Passivhaus buildings:
This level of detail raises several questions, including:
- Do most PHPP users supply accurate inputs?
- Is the PHPP model accurate?
- How much do the small differences that PHPP users sweat over really matter?
The Ja/Nein Fallacy
At the Building Energy 12 conference in Boston, Matthew O’Malia, an architect at GO Logic in Belfast, Maine, explained how Passivhaus designers approach their work. “PHPP is a massive spreadsheet,” said O’Malia. “It’s the mother of all spreadsheets. Here’s what I like about the Passivhaus approach: You either achieve the standard or you don’t. At the end of the spreadsheet, your answer appears in this box. The answer is either ‘Ja’ or ‘Nein.’ There is no ‘maybe’ in German.”
Some Passivhaus designers go further than O’Malia, implying that a building that falls short of the magic 15 kWh per square meter is at risk of failure. To these designers, the Passivhaus standard represents an important threshold for performance and moisture control. The implication is that designers who aren’t conversant with WUFI or THERM can end up designing buildings that encourage condensation and mold.
I propose a name for this mistake — the “Ja/Nein Fallacy.”
In fact, there is no evidence that superinsulated buildings that fall on the “Nein” side of the Passivhaus divide are experiencing moisture or performance problems. Moreover, as Blasnik pointed out, once the homeowners move into their new Passivhaus abode, variations in plug loads can overwhelm the small envelope issues that Passivhaus designers lose hours of sleep over.
Don’t throw your energy models out the window
Good energy models, including PHPP, can be very instructive for new-home designers. The best models clearly reveal the importance of choosing a compact shape, avoiding bump-outs, installing orientation-specific glazing, and addressing thermal bridges. Once learned, however, these valuable lessons do not need to be rediscovered for every new house.
Of course, designers of custom superinsulated homes are likely to continue using energy modeling programs, and their designs — resulting from an iterative process of continual refinement — help instruct designers and builders of simpler homes who may choose to avoid the expense of energy modeling.
Last week’s blog: “Solar Thermal is Dead.”