How to evaluate the effectiveness of an algorithm that predicts probabilities?

Question

I need to evaluate the effectiveness of algorithms that predict the likelihood of something happening.

My current approach is to use the "standard error", i.e. the square root of the mean square of errors, where the error is a 1.0 prediction if an event occurred, or a prediction if an event did not occur.

Algorithms do not have specific applications, but it will be common to develop a prediction of the event that occurs for each of the many options, and then select an option that maximizes this probability. The advantage for us is directly proportional to the speed with which the desired event occurs among the options with the highest predicted probabilities.

It has been suggested that RMSE may not be the best option for this, and I'm interested in the opinions of others.

Answer 1

A Chi-square test is a widespread spreadability test:

Σ (O _i - E _i ) ² / E _i

O _i - i, E _i - . - (~ 5 10, , , ) . , Yates:

Σ (| O _i - E _i | - 0.5) ²/E _i

: . , , . , - RMSE, , .

-, .

Answer 2

ROC, .

:

(ROC), ROC-, (1 - ) . ROC (TPR = ) (FPR = false ). , (TPR & FPR) . [1]

ROC , ( ) . / . - , , . ROC , , , .

, , - "" ROC .

R .

Answer 3

, , .

:
, , , .

, , , , .

, .

:
- , -.

, . . , , , , .

.

, . , - .

:

, , . , .

, , , . (, .)

, . . , , .

Answer 4

, -, 0 1, ? , . "" . , - , , - .

- -, -. " Logit" .