1st International Workshop on Evaluating General-Purpose AI (EGPAI 2016)
Views: 759
When : 2016-08-30
Where : The Netherlands
Submission Deadline : 2016-06-01
Categories : Machine Learning , Artificial Intelligence
1st International Workshop on Evaluating General-Purpose AI (EGPAI 2016)
August 30, 2016
The Netherlands
Call For Papers
The aim of this workshop is to analyse all aspects of the evaluation of general AI systems. Most AI systems are tested on specific tasks. However, to be considered truly intelligent, a system must be flexible enough to be able to learn how to perform a wide variety of tasks, some of which may not be known until the system is deployed. This workshop will examine formalisations, methodologies and test benches for evaluating the numerous aspects of this type of general AI systems. We are interested in theoretical or experimental research focused on the development of concepts, tools and clear metrics to characterise and measure the intelligence, and other cognitive abilities, of general AI agents.
Topics
- Analysis and comparisons of AI benchmarks and competitions. Lessons learnt.
- Proposals for new general tasks, evaluation environments, workbenches and general AI development platforms.
- Theoretical or experimental accounts of the space of tasks, abilities and their dependencies.
- Evaluation of development in robotics and other autonomous agents, and cumulative learning in general learning systems.
- Tasks and methods for evaluating: transfer learning, cognitive growth, structural self-modification and self-programming.
- Evaluation of social, verbal and other general abilities in multi-agent systems, video games and artificial social ecosystems.
- Evaluation of autonomous systems: cognitive architectures and multi-agent systems versus general components: machine learning techniques, SAT solvers, planners, etc.
- Unified theories for evaluating intelligence and other cognitive abilities, independently of the kind of subject (humans, animals or machines): universal psychometrics.
- Analysis of reward aggregation and utility functions, environment properties (Markov, ergodic, etc.) in the characterisation of reinforcement learning tasks.
- Methods supporting automatic generation of tasks and problems with systematically introduced variations.
- Better understanding of the characterisation of task requirements and difficulty (energy, time, trials needed..), beyond algorithmic complexity.
- Evaluation of AI systems using generalised cognitive tests for humans. Computer models taking IQ tests. Psychometric AI.
- Application of (algorithmic) information theory, game theory, theoretical cognition and theoretical evolution for the definition of metrics of cognitive abilities.
- Adaptation of evaluation tools from comparative psychology and psychometrics to AI: item response theory, adaptive testing, hierarchical factor analysis.
- Characterisation and evaluation of artificial personalities.
- Evaluation methods for multi-resolutional perception in AI systems and agents.