diff options
| author | Xusong Wang <xusongw@google.com> | 2018-08-28 16:50:01 -0700 |
|---|---|---|
| committer | Xusong Wang <xusongw@google.com> | 2018-08-28 16:53:23 -0700 |
| commit | 10d77e463ca1e844dbdf1b47add76990a560d2c0 (patch) | |
| tree | 414623bff17fdefe2cebcd908ca36cd587703526 /identity/support/src/cppbor_parse.cpp | |
| parent | d8a9538c5341c335793f435f94509227b1811daf (diff) | |
Fix logics for floating-point comparision in VTS test.
Set the acceptable error range based on both absolute tolerance and
relative tolerance.
Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon
(~5e-3) for FP16 relaxed computation. The relative tolerance is set to
5ULP of the corresponding precision. Add a TODO mark for potential
future adjustment on error limit based on testing.
Bug: 111768023
Test: none
Change-Id: Idedcec3e09fd7de9696811b93c81d0f180e896ef
Diffstat (limited to 'identity/support/src/cppbor_parse.cpp')
0 files changed, 0 insertions, 0 deletions