Engine
Engine copied to clipboard
OpenSourceRisk
Help debug error please: Error in ORE analytics: strike + displacement must be non-negative
Wonder if you can help.
Context: I am backtesting different portfolios and analytics in ORE every quarter across the period 2019-2023 with mostly real market data as part of an extended POC for a client. All interest rates (plus zero rates) are real historical data.
I am running a variety of analytics and instruments, the majority run OK.
Error: For a few different dates, all in 2020 when interest rates were cut around the world due to COVID, the risk job fails with "Error in ORE analytics: strike + displacement (-0.00393379 + 0) must be non-negative". (The value is different for each portfolio/as_of_date combination).
This message comes from an error handler in blackformula.cpp(47) (in Quantlib).
Relative to jobs that run ok, there does not appear to be any additional warnings or errors in the logs.
Why Only Three Points? Even if the floating rate (like EUR-EURIBOR-6M) went negative at certain points, it's possible that only on specific valuation dates (given the strike of the swap and the particular market conditions) the combined value became negative enough to trigger the error. Could this by why the error doesn't appear consistently across all dates?
Portfolio: All of the failed examples relate to Swaps (for example, a portfolio with a single IR Swap with EUR-EURIBOR-6M as the index).
Request:
Please suggest some ideas for debugging / fixing the error.
Could an avenue be to set displacement? I searched the user guide for 'displacement' but it isn't referenced. I can see displacement being defaulted to 0.0 in various places in the code and test cases that use small values for displacement. Can displacement be set through a parameter (e.g. for implied-volatility calculations or to override the engine function)? I looked at inputparameters.hpp and the answer seems to be no.
Or is there an option to adjust the volatility term structure through a parameter to take the shift into account?
Note, it could be that one of the synthetic rates is causing a problem, but I though this less likely as I have used them with 20+ different portfolios utilising most of the instruments that ORE covers.
Hi, there can be a variety of reasons why the strike would be negative. Generally, you need to reduce the scope to isolate the issue, ideally to one analytic/date/trade. You can do this by debugging the source code to add a conditional break point when this situation occurs, then figure out from the call stack the context that leads to this situation.
OK thanks @noonediesalone - I was hoping for a silver bullet, but I will try your suggestion. I had already reduced the scope to a single vanilla swap on one date. I will also try adding a single analytic at a time. I have seen this once before and it was do with some vol market data points that were not specified correctly (I used the wrong RIC). I may come back with other questions.
Figuring out the trade and date is obviously the hardest part if you have a sizable portfolio, so looks like you've made progress. Should be super simple to identify the analytic (just activate one at a time). But since it's a vanilla swap and you have issue with BS formula it's probable this is from simulation, maybe model calibration. If so, then you need to look on the ir model from the simulation file to see how you've defined the calibration instruments. Also, maybe activate the trace level for ORE logs to get better context surrounding the emitted exception.
Thanks for that.
As you guessed, the job fails in the simulation analytics. As mentioned, this job fails just for this single swap portfolio on three as_of_dates but otherwise runs successfully over three years. The market data and fixings files are common to 25 different portfolios for any specific as_of_date.
The job seems to fail on creating a swaption helper for the index=GBPLibor6M for expiry 10Y and term 10Y. Prior to this, it successfully creates a swaption helper for the index=USDLibor3M.
Anyway, the strike turns negative at that point and the error is caught in the Black formula.
ALERT [2023-Sep-15 08:06:06.611308] (OREAnalytics/orea/app/oreapp.cpp:384) : StructuredMessage { "category": "Warning", "group": "Analytics", "message": "Error in ORE analytics: strike + displacement (-0.00393379 + 0) must be non-negative", "sub_fields": [ { "name": "analyticType", "value": "OREApp::run()" }, { "name": "warningType", "value": "Error" } ] }
I note from the logs, in the line before it fails, that there is a reference to shift (=0):
DEBUG [2023-Sep-15 08:06:06.609947] (OREData/ored/model/lgmbuilder.cpp:155) : Created swaption helper with expiry 8Y and term 12Y: vol=0.1, index=GBPLibor6M Actual/365 (Fixed), strike=3.40282e+38, shift=0
Back to one of my original questions: Could a solution for negative rates / strike to set shift / displacement (as that's what it was designed for)? Displacement was added to the Black model in QuantLib some time ago. I searched the ORE user guide for 'displacement' but it isn't referenced. In QuantLib I can see displacement being defaulted to 0.0 in various places and test cases that use small values for displacement. Can displacement be set through a parameter in ORE (e.g. for implied-volatility calculations or to override the engine function)? I looked at inputparameters.hpp and the answer seems to be no. Or is there an option to adjust the volatility term structure through a parameter to take the shift into account?
I 'solved' the last issue like this by utilising the AllowNegativeRates in DefaultCurve. But that is not an option here.
To your question, here is the IRmodels section from the simulation file. I use these across several portfolios so they are not specific to this one. I don't see a problem here.
<InterestRateModels>
<LGM ccy="default">
<CalibrationType>Bootstrap</CalibrationType>
<Volatility>
<Calibrate>Y</Calibrate>
<VolatilityType>Hagan</VolatilityType>
<ParamType>Piecewise</ParamType>
<TimeGrid>1.0, 2.0, 3.0, 4.0, 5.0, 7.0, 10.0</TimeGrid>
<InitialValue>0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01</InitialValue>
</Volatility>
<Reversion>
<Calibrate>N</Calibrate>
<ReversionType>HullWhite</ReversionType>
<ParamType>Constant</ParamType>
<TimeGrid/>
<InitialValue>0.03</InitialValue>
</Reversion>
<CalibrationSwaptions>
<Expiries> 1Y, 2Y, 4Y, 6Y, 8Y, 10Y, 12Y, 14Y, 16Y, 18Y, 19Y</Expiries>
<Terms> 19Y, 18Y, 16Y, 14Y, 12Y, 10Y, 8Y, 6Y, 4Y, 2Y, 1Y</Terms>
<Strikes/>
</CalibrationSwaptions>
<ParameterTransformation>
<ShiftHorizon>0.0</ShiftHorizon>
<Scaling>1.0</Scaling>
</ParameterTransformation>
</LGM>
<LGM ccy="EUR">
<CalibrationType>Bootstrap</CalibrationType>
<Volatility>
<Calibrate>Y</Calibrate>
<VolatilityType>Hagan</VolatilityType>
<ParamType>Piecewise</ParamType>
<TimeGrid>1.0, 2.0, 3.0, 4.0, 5.0, 7.0, 10.0</TimeGrid>
<InitialValue>0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01</InitialValue>
</Volatility>
<Reversion>
<Calibrate>N</Calibrate>
<ReversionType>HullWhite</ReversionType>
<ParamType>Constant</ParamType>
<TimeGrid/>
<InitialValue>0.03</InitialValue>
</Reversion>
<CalibrationSwaptions>
<Expiries> 1Y, 2Y, 4Y, 6Y, 8Y, 10Y, 12Y, 14Y, 16Y, 18Y, 19Y</Expiries>
<Terms> 19Y, 18Y, 16Y, 14Y, 12Y, 10Y, 8Y, 6Y, 4Y, 2Y, 1Y</Terms>
<Strikes/>
</CalibrationSwaptions>
<ParameterTransformation>
<ShiftHorizon>0.0</ShiftHorizon>
<Scaling>1.0</Scaling>
</ParameterTransformation>
</LGM>
<LGM ccy="CHF">
<CalibrationType>Bootstrap</CalibrationType>
<Volatility>
<Calibrate>Y</Calibrate>
<VolatilityType>Hagan</VolatilityType>
<ParamType>Piecewise</ParamType>
<TimeGrid>1.0, 2.0, 3.0, 4.0, 5.0, 7.0, 10.0</TimeGrid>
<InitialValue>0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01</InitialValue>
</Volatility>
<Reversion>
<Calibrate>N</Calibrate>
<ReversionType>HullWhite</ReversionType>
<ParamType>Constant</ParamType>
<TimeGrid/>
<InitialValue>0.03</InitialValue>
</Reversion>
<CalibrationSwaptions>
<Expiries> 1Y, 2Y, 4Y, 6Y, 8Y, 10Y, 12Y, 14Y, 16Y, 18Y, 19Y</Expiries>
<Terms> 19Y, 18Y, 16Y, 14Y, 12Y, 10Y, 8Y, 6Y, 4Y, 2Y, 1Y</Terms>
<Strikes/>
</CalibrationSwaptions>
<ParameterTransformation>
<ShiftHorizon>0.0</ShiftHorizon>
<Scaling>1.0</Scaling>
</ParameterTransformation>
</LGM>
</InterestRateModels>
One more question: has anyone implemented a traceback in ORE (e.g. backwards-cpp)?
Any help on this would be useful. Thanks.
Hi, is this happening for swaption or for cap vols, do you know? I am asking because we convert Cap vols to normal vols always (during the bootstrap in the T0 market) and Swaption vols under certain circumstances, see OREAnalytics/orea/scenario/scenariosimmarket.cpp:
// convert to normal if
// a) we have a swaption (i.e. not a yield) volatility and
// b) the T0 term structure is not normal
// c) we are not in the situation of simulating ATM only and having a non-normal cube in T0,
// since in this case the T0 structure is dynamically used to determine the sim market
// vols
// d) we do not use spreaded term structures, in which case we keep the original T0
// term structure in any case
With normal vols you won't see the "displacement" error.
hi @pcaspers @noonediesalone - thanks for your help.
@pcaspers - The error happens with caps and swaptions as well.
In retrospect, it perhaps should have been obvious that as the error occurred in building helper swaptions during an exposure simulation, that one cause could have been the specification of the CalibrationSwaptions within the InterestRateModels section of the simulation.xml specification file.
As the issue occurs when building the swaption helper for expiry 10Y and term 10Y: vol=0.1, index=GBPLibor6M, I can cludge-solve the issue by amending the 10Y from the Expiries from CalibrationSwaptions for the default profile:
From this:
<Expiries> 1Y, 2Y, 4Y, 6Y, 8Y, 10Y, 12Y, 14Y, 16Y, 18Y, 19Y</Expiries>
<Terms> 19Y, 18Y, 16Y, 14Y, 12Y, 10Y, 8Y, 6Y, 4Y, 2Y, 1Y</Terms>
...to this:
<Expiries> 1Y, 2Y, 3Y, 4Y, 6Y, 9Y, 12Y, 14Y, 16Y, 18Y, 19Y</Expiries>
<Terms> 20Y, 17Y, 15Y, 12Y, 10Y, 8Y, 6Y, 4Y, 3Y, 2Y, 1Y</Terms>
As this is a temporary solution based on trial and error, could you offer any thoughts on how to avoid these issues? Or is it just a matter of the interest rate changes at that unusual point in history?
For example, would it make sense for there be a 'retry with displacement' error handling function if this strike + displacement non-negative error occurs, where it sets displacement = -strike?
For displacement distribution of vols, please check ShiftedLognormal in the user guide.
In simulation setup, for practical reasons, is probably best to have an IR model for each ccy, so that you can specifically select calibration set that applies to one ccy (e.g. in some cases you want to calibrate with longer expiry/terms, but the yield term structure doesn't go as long; or in some cases you want a different layout of calibration instruments, be that co-terminal / diagonal / or something else; or you might want to pre-compute the mean reversion; or ..).
In your example, seems the strike explodes even here expiry 8Y and term 12Y: vol=0.1, index=GBPLibor6M Actual/365 (Fixed), strike=3.40282e+38, shift=0. Maybe try to replicate this behavior by pricing a stand alone Swaption trade, that you construct with same properties as this calibration helper.
@noonediesalone Thanks again.
Could you please help me map out where/how I need to specify ShiftedLognormal vols? Here is my best guess:
1. Register the SwaptionVolatilities id in todaysmarket.xml e.g.
<SwaptionVolatilities id="SHLOG">
<SwaptionVolatility currency="GBP">SwaptionVolatility/GBP/GBP_SW_SHLOG</SwaptionVolatility>
</SwaptionVolatilities>
_Q: I have only used <SwaptionVolatilities id="default"> - can / should I specify a id? Can I mix normal and shiftedlog vols for different currencies or do I need to be consistent for all? _
2. Define the curve in curveconfig.xml
<SwaptionVolatility>
<CurveId>GBP_SW_SHLOG</CurveId>
<CurveDescription>GBP shiftedlognormal swaption volatilities</CurveDescription>
<Dimension>ATM</Dimension>
<VolatilityType>ShiftedLognormal</VolatilityType>
<Extrapolation>Flat</Extrapolation>
<DayCounter>Actual/365 (Fixed)</DayCounter>
<Calendar>TARGET</Calendar>
<BusinessDayConvention>Following</BusinessDayConvention>
<OptionTenors>
1M,3M,6M,1Y,2Y,3Y,4Y,5Y,7Y,10Y,15Y,20Y,25Y,30Y
</OptionTenors>
<SwapTenors>
1Y,2Y,3Y,4Y,5Y,7Y,10Y,15Y,20Y,25Y,30Y
</SwapTenors>
<ShortSwapIndexBase>GBP-CMS-1Y</ShortSwapIndexBase>
<SwapIndexBase>GBP-CMS-30Y</SwapIndexBase>
</SwaptionVolatility>
3. Define the shifts/displacement in marketdata.txt (assuming as a rate not bp) with a matrix of shifts:
YYYYMMDD SWAPTION/SHIFT/GBP/1M
YYYYMMDD SWAPTION/SHIFT/GBP/3M
YYYYMMDD SWAPTION/SHIFT/GBP/6M
YYYYMMDD SWAPTION/SHIFT/GBP/1Y
YYYYMMDD SWAPTION/SHIFT/GBP/2Y
YYYYMMDD SWAPTION/SHIFT/GBP/3Y
...etc...covering both option and swap tenors.
4. Specify the SwaptionVolatilities in simulation.xml - Q: how / do I need to link to the vol /curve id?
I look forward to hearing from you.
p.s. it runs OK with the cludge-fix. expiry 8Y and term 12Y: is the point before the strike turns negative.
Yes, something like this should work.
In addition, you need in todaysmarket.xml definition for configuration id:
<Configuration id="SHLOG">
<SwaptionVolatilitiesId>SHLOG</SwaptionVolatilitiesId>
</Configuration>
Then you can specify in ore.xml that you want this configuration to be used for simulation:
<Markets>
...
<Parameter name="simulation">SHLOG</Parameter>
...
</Markets>
Configuration id fallbacks on default if what is requested is not defined specifically under same id.
I'm not sure though if this goes on recursively, e.g. fallback on a fallback.