Why and How We Calculate p‐value

Why and How We Calculate p-value

1. Why p-value is Important

Simply looking at total returns is not sufficient to evaluate the performance of a trading model.
Financial markets, especially equity indices like ETFs, often exhibit a long-term upward trend.
This means that even random trading strategies can sometimes produce positive returns.

To distinguish between:

• Gains caused by the general market trend
• Gains achieved through the intelligence of the model

we must compare the model's performance against random strategies.
This is why we calculate a p-value:
it quantifies the probability that the observed performance could have occurred by random chance.

A lower p-value (e.g., below 0.05) suggests that the model's behavior is statistically significant and not easily replicable by random trading.

---

2. How p-value is Currently Calculated

• Perform up to 1000 random trading simulations.
• Each simulation randomly selects buy and sell timings.
• The number of trades approximately matches the original model using the hold rate (the proportion of time the model is holding a position).
• After each simulation, the total return is calculated.
• If a random simulation achieves a higher total return than the model, it is counted as a success.

Finally, the p-value is calculated as:

$$ p_{value} = \frac{\text{number of better random simulations}}{\text{total simulations}} $$

This method provides a reasonable estimation, but can be computationally expensive because 1000 simulations are performed for each evaluation.

---

3. Possible Improvements: Adaptive Simulation Count

To optimize computation time, an adaptive stopping strategy can be considered:

• Monitor the estimated p-value and its standard error during the simulations.
• If the standard error becomes sufficiently small (e.g., below 0.01), stop early without reaching the maximum number of 1000 trials.
• This allows for faster evaluation when the p-value is either clearly low or clearly high.

This would greatly reduce unnecessary computation, especially when the model is clearly good or clearly bad.

---

Notes

Evaluating statistical significance (p-value) is crucial for avoiding false confidence in trading strategies,
especially in markets with strong directional biases.