No statistical significance but observable trends

Question

I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.

For example, treatment A and treatment B are applied to 2 independent populations. Using a ttest to analyze the resulting data (lets say the data is total revenue), the p value == .2, so the effect of treatment on revenue was not statistically significant. However, the total revenue from treatment A was observably higher in treatment B. What can I say in this regard?

I've had academic advisors recommend saying 'While the effect of treatment was not significant, a trend was observed', and then one would go on describing the trend. Is this an adequate viewpoint, or a statistical folly? What conclusions would a data scientist in industry draw and present to stakeholders from a scenario like this?

Answer 1

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B.

From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference.

Due to the amount of randomness observed - if we did ran the experiment again, treatment B mean could be higher than treatment A mean.