Hello everyone,
I have a question about the rounding results.
Why does rounding to 3 decimal places on the value 0.3301 give only 0.33 and not 0.330?
Thank you in advance for your help.
Ludivine
Find maintenance, best practice, and troubleshooting articles.
Hello everyone,
I have a question about the rounding results.
Why does rounding to 3 decimal places on the value 0.3301 give only 0.33 and not 0.330?
Thank you in advance for your help.
Ludivine
Hi ludivineb
I've had a look at some data where this should be replicated and I see the value displayed in a report with all three decimal places. Where are you seeing this? Can you share a screenshot?
/Andy
Hello Andy,
Thank you for your reply. Here is a screenshot to visualize the rounding problem:
When "Signal" is selected in the "Scope" column, the rounding is wrong while when "Peak and groups" is selected, the rounding is right.
Thank you in advance for your answer.
Ludivine
Hi ludivineb
Ah, I see. I had only tested it in a table, and not in a custom calculation with a scope like signal. The rounding, I would argue, is correct in both cases, but the formatting of the results is different when the value is displayed in a Results Table.
Change the format of the result for the calculation with the scope Signal:
These are the results for three different calculations:
The calculations being
Min("Compound_Concentration",CurrentInjection.AllIdentifiedPeaks) - in my data this gives me a number like yours with the third decimal place being zero
Round(CC1,3) - CC1 is the result of the calculation above, and shows the behaviour that you described
Format(CC2,"F3") - CC2 is the result of the rounding calculation, and, IF TYPE ASSOCIATED WITH THIS CALCULATION IS STRING, gives you the behaviour you want to see
Hope that answers your question.
Regards
Andy
Instead of round(x,y) use formatnumber(x,y) or formatnumber(x,y,true,true,false) It will give you the results as you want, but they will be in String format, not numeric, so further calculation won't be easy :)
Hello,
I think this has more to do with the display controls in injection results between peaks and the summary values. In the peak results you can force numeric precision on a column which will give you the remain zeros. This control is not available in in the summary results which may be why you see the difference.
Marty Adams