compare_and_rank_results#

cinnabar.compare.compare_and_rank_results(femap: FEMap, prediction_type: Literal['nodewise', 'edgewise'] = 'edgewise', rank_metric: Literal['MUE', 'RMSE', 'RAE', 'R2', 'rho', 'KTAU', 'PI'] = 'MUE', metrics_to_compute: list[Literal['MUE', 'RMSE', 'RAE', 'R2', 'rho', 'KTAU', 'PI']] | None = None, num_bootstraps: int = 1000, confidence_level: float = 0.95, alpha: float = 0.05) → tuple[DataFrame, DataFrame][source]#

Compare and rank multiple result sources on a single FEMap based on the chosen performance metric and return an ordered table of results using the compact letter display (CLD) system.

Parameters:

  • femap (FEMap) – An FEMap instance with results from multiple sources to compare.

  • prediction_type ({“nodewise”, “edgewise”}, default “edgewise”) – The type of prediction in the FEMap to evaluate.

  • rank_metric ({“MUE”, “RMSE”, “RAE”, “R2”, “rho”, “KTAU”, “PI”}, default “MUE”) – The metric used to rank the models.

  • metrics_to_compute (list[{“MUE”, “RMSE”, “RAE”, “R2”, “rho”, “KTAU”, “PI”}] | None, default None) – A list of metrics to compute for each model. If None, all metrics appropriate for the prediction_type will be computed.

  • num_bootstraps (int, default 1000) – The number of bootstrap samples to use for estimating confidence intervals.

  • confidence_level (float, default 0.95) – The confidence level for the intervals.

  • alpha (float, default 0.05) – The significance level (Type I error probability) for determining statistical significance in pairwise comparisons. Lower values are more conservative and require stronger evidence for significance.

Note

  • The comparison method uses a joint bootstrapping procedure that generates a distribution of differences in the rank metric and checks for significant differences using a method inspired by. [1]

  • Each source must be evaluated on the same set of edges.

  • Prediction types “nodewise” and “edgewise” correspond to DGs and edgewise DDGs respectively.

  • When we have more than 2 models, we apply multiple testing correction to the pairwise comparisons using the Holm

    method to control the family-wise error rate in a low number of comparisons. For more information see https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method.

  • In cases where one method clearly outperforms another the bootstrap p-value may be reported as 0.0 because none

    of the bootstrap differences cross zero. This is a limitation of the bootstrap testing method. In such cases the confidence interval around the difference metric should be used to interpret the significance of the difference instead and a p-value of 0.0 should not be reported as the level of significance.

Returns:

A tuple containing two DataFrames: - The first DataFrame contains the computed metrics for each model. - The second DataFrame contains the pairwise comparison results between models.

Return type:

tuple[pd.DataFrame, pd.DataFrame]

References

On this page

This Page