Best way to automate testing AI algorithms?

Question

I wonder how people automatically test artificial intelligence algorithms.

One example for a Turing test - say, there were several applications for the competition. Is there any conceivable way to score candidates in automatic mode - except that people check them.

I also saw some data sets (hidden images of numbers / letters, groups of photographs, etc.) that can be downloaded and explored over time. What good resources exist for this.

One problem that I see: you do not want the algorithm to adapt to the test data over time, because you are trying to understand how good this is in the general case. Are there any methods to ensure that this is not the case? For example, each time he randomly tests or averages his results over a bunch of random tests.

In principle, given a bunch of algorithms, I want some kind of automated process to feed this data and see how well he "learned" or predicted new things that he had not seen.

Answer 1

, , - " ", , " ", .

Answer 2

- AI, , , "" . , : , . , , "" , 3 , , . , , , , (, ), . (, , ), , .

, , , , . , , , ( , /).

.

Answer 3

@ - , . , , , :

• : - , , . , , (, ).
• : , , Netflix, (, , , ..), , .

Weka . , mloss.org . , data.gov , .

Answer 4

K- , ( ) "", :

: .

AI- ( ) , "" , . . .

, S- ( , ), s, , , .

, . , , , " ". , , "" . , .

: +10 : -10

, , , , , , , .

, , , , . "" . .

, , , , "" Q-Learning.

- . , , . , , , , .

, , .

, , , - , , "" " " , , , , . ( ).

, " ", , - , "" . (). ,

"turn left every half mile"

"turn left every half mile if I have turned right 3 times and then turned left 2 times and had a quarter in my left pocket to pay fare... etc etc"  

.

, , A.I. . - , , , , , . ( !), , , RAM. , , , memristor - -

Neural Networks, . "" ... , "", .

ai ... research is mostly probabilistic because we don’t know how to make “thinking”, we just know how to imitate our internal learning process, try again.