Intelligent Industrial Processes & Enabling ICT

Intelligent Industrial Processes & Enabling ICT ... microcontroller can accomplish. ... image processing...

0 downloads 207 Views 4MB Size
Intelligent Industrial Processes & Enabling ICT A Machine Learning and Intelligence Perspective

Fredrik Sandin ([email protected]) with inputs from Lennart Gustafsson, EISLAB, Luleå University of Technology, 971 87 Luleå, Sweden. April 22, 2014

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Abstract Intelligent Industrial Processes (IIP) and Enabling Information and Communication Technology (Enabling ICT) are two out of the nine areas of excellence in research and innovation at the Luleå University of Technology (LTU), which are formed to foster interdisciplinary research and innovation in strategically important areas. This report presents a perspective on the role of machine learning and intelligence in these two areas, focusing in particular on future ICT for industrial process automation (ProcessIT) up to the year 2030. The study that is presented here complements similar studies made in other fields, with the common goal to create the first inputs for a broader discussion and formulation of strategic objectives in the form of a roadmap. This report presents my interpretation of the concept of Intelligent Industrial Processes and the role of ICT in that context, including novel information processing methods and devices that are inspired by biological circuits and systems. This report also includes brief introductions and definitions of important concepts; a summary of seven documents presenting international strategic agendas and objectives; a summary of identified strengths, weaknesses, opportunities and threats; a description of selected research trends with references to interesting results; a tentative outline of interesting research problems and first steps towards 2030; and a list of research groups with complementary competences that may merit future partnership. It is concluded that the Open Research and Innovation Platform that is outlined in a parallel study would be a valuable resource for machinelearning research, development and education because transparent access to data is a key enabling factor. In terms of machine-learning research it is concluded that we need to take the step from studies of isolated learning algorithms and applications to closed-loop learning architectures for large-scale sensor-actuator systems, possibly including humanmachine interaction, decision support systems and models of complex systems such as maintenance systems and markets. The aim to develop intelligent industrial processes using a new generation of ICT is an ambitious interdisciplinary initiative, which is likely to force us thinking beyond conventional methods and to educate a new generation of engineers that understand the necessary concepts.

Keywords – Affective computation, Automation, Bayesian statistics, Cognitive architecture, Cognitive computation, Complex systems, Cyber-physical systems, Feature learning, Grounded cognition, Information theory, Internet of things, Machine learning, Neuroinformatics, Neuromorphic engineering, Sensory-motor (loop), Statistical physics, Unsupervised learning.

2

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Preface This study is supported jointly by IIP and Enabling ICT. This report outlines a perspective on the role that machine learning and machine-intelligence–related research can have in these areas, complementing similar studies made in other fields in order to take the first steps towards a broader discussion and outline of a roadmap. The following people have contributed information or feedback during the work that is summarized in this report: Prof. Jerker Delsing (coordinator, IIP), Prof. Christer Åhlund (coordinator, Enabling ICT), Prof. emer. Lennart Gustafsson, Dr. Wolfgang Birk, Prof. Mikael Sjödahl, Prof. Johan Carlsson, Ph.D. student Blerim Emruli, and Ph.D. student Sergio Martin Del Campo Barraza. In particular I would like to thank Lennart Gustafsson, who participated in the literature study and related discussions, where he contributed valuable information. Lennart has described some of his views in short reports, which I summarize here in the form of text boxes with titles “Lennarts perspective on ...”. I would also like to thank Wolfgang Birk for useful discussions about the concept and implications of intelligent industrial processes. Finally, I thank Jerker Delsing and Christer Åhlund for the opportunity to study this field from a new perspective, which has resulted in some new insights, some of which are described in two complementary articles [1, 2] written during the period of this study.

3

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Contents 1 Introduction 1.1 Aims and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Role of Information and Communication Technology (ICT) . . . . . . . . . 1.3 Towards Intelligent Industrial Processes . . . . . . . . . . . . . . . . . . . . 2 Definitions and Basic Concepts 2.1 Intelligent Industrial Processes . . . . . . . . . . . . . 2.2 Enabling Information and Communication Technology 2.3 Machine Learning . . . . . . . . . . . . . . . . . . . . . 2.3.1 Relevance . . . . . . . . . . . . . . . . . . . . . 2.3.2 Modern Approach . . . . . . . . . . . . . . . . 2.3.3 Central Concepts . . . . . . . . . . . . . . . . . 2.3.4 Uses of Machine Learning . . . . . . . . . . . . 2.3.5 Feature Learning . . . . . . . . . . . . . . . . . 2.3.6 Model Selection and Checking . . . . . . . . . 2.3.7 Need for Education . . . . . . . . . . . . . . . . 2.4 Intelligence . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Motivation . . . . . . . . . . . . . . . . . . . . 2.4.2 From Machine Learning to Intelligence . . . . . 2.4.3 Formal Definition . . . . . . . . . . . . . . . . .

5 5 6 6

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

7 7 8 8 8 9 10 11 11 12 12 13 14 14 16

3 Strategic Agendas and Objectives 3.1 ICT in Horizon 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 European Technology Platform on Smart Systems Integration . . . . . . 3.3 BRAIN Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Human Brain Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Roadmap for U.S. Robotics: From Internet to Robotics . . . . . . . . . 3.6 Neuro-IT roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 European Roadmap for Industrial Process Automation (ProcessIT.EU) . 3.8 Comments and Interpretation . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

16 16 17 18 19 21 22 22 23

4 SWOT Analysis

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

25

5 Open Problems and Research Directions 26 5.1 First Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 6 Vision and Tentative Steps Towards 2030 30 6.1 First steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.2 Tentative Steps Towards 2030 . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.3 Competence and Partnerships . . . . . . . . . . . . . . . . . . . . . . . . . . 33 7 References

35

A List of High-Impact Journals

39

B Keyword Charts

40

Index

44

4

IIP & Enabling ICT: Machine Learning and Intelligence

1

April 22

Introduction

The most important long-term trend in the world economy is the automation of work processes that are otherwise performed manually. Sectors like process industry, manufacturing and farming have been significantly transformed by such innovations and technological developments. This report deals with the role of machine learning and intelligence for the further development of automation technology.

1.1

Aims and Methodology

This study focuses on machine learning and intelligence in the context of Intelligent Industrial Processes (IIP) and Enabling Information and Communication Technology (Enabling ICT), which are two areas of excellence in research and innovation at the Luleå University of Technology (LTU). The goal of this work is to provide a perspective on the role that machine-learning related research and development can have in the context of processindustry automation and other ICT related subjects in a 2030 timeframe, and how LTU can contribute. An open minded and visionary approach is encouraged. This report complements similar studies made in other fields, with a common goal to create inputs for a broader discussion and collection of feedback needed to make a joint roadmap. Therefore, I avoid duplicating information that is available in the parallel reports [3, 4]. In particular, general aspects related to the introduction of industrial processes and analyses of strategic objectives that are covered in those reports are omitted here. The perspective presented here includes brief definitions of central concepts; a summary of selected strategic agendas and objectives in roadmaps and Horizon 2020 calls; a description of identified strengths, weaknesses, opportunities and threats (SWOT); a description of research trends with references to interesting papers and results; and some comments on industrial relevance, requirements and groups/people with key competence. In addition, it is important to consider activities that are under development within other excellence areas at LTU, including Smart Machines and Materials and Future Mining. Ideally, common interests and possibilities for cooperation should be identified before the roadmap is outlined. Given that machine learning is an active field of research with thousands of potentially relevant papers published annually, it appears practically impossible to make a comprehensive survey given the limited time available for this study. Therefore, the topics that are selected and described here are influenced by strategic agendas and objectives, and previously recognized challenges and open problems identified by colleagues and industrial partners. I have used Scopus and the snowball method to investigate research trends and Google Trends to explore search interests of the general public. Scopus is selected because I find it easier to work with compared to the Web of Science. I have not found evidence in the peer-reviewed literature indicating that one of these tools should be superior over the other. Google Scholar is not considered for quantitative purposes because it tends to include papers in low-impact journals, popular scientific literature, unpublished reports and teaching material [5]. In addition to reading research papers and strategic agendas I have also considered conference presentations by a few outstanding researchers in the field. Regardless of methodology it appears difficult to predict the technological advances up to 2030 in this area, considering the rapid development of ICT and the significant changes that it has caused over the last fifteen years. The current substantial international investments in brain research and brain-inspired ICT can become a game changer (see the summary of the Human Brain Project in this report). We have already seen computers challenge humans in complex tasks such as Jeopardy and large-scale image analysis using deep learning methods. Therefore, the perspective outlined here is as a starting point that needs to be extended and revised over time.

5

IIP & Enabling ICT: Machine Learning and Intelligence

1.2

April 22

Role of Information and Communication Technology (ICT)

ICT plays a central role in automation and underpins innovation and competitiveness across private and public sectors. For example, ICT-related topics can be found in all prioritized areas of the Horizon2020 program for research and innovation, from ’Excellence Science’ to ’Industrial Leadership’ and ’Societal Challenges’. The motivation being that ICT accounts for 4.8% of the EU economy and 50% of European productivity growth. ICT also brings novel means to address societal challenges like sustainable healthcare and welfare of the aging population, security in urban environments, and reducing the environmental effects of energy and transportation systems. The role and use of ICT evolves quickly. For example, a few decades ago it was hard to imagine that mobile phones and computers that are connected across the globe would be central for the daily lives of many people. Half a century ago, a room-sized state-of-theart supercomputer was capable of one million floating-point operations per second, which is two to three orders of magnitude lower than a modern mobile phone or embedded microcontroller can accomplish. This development in combination with the increasing volume and value of data generated in ICT applications generate a growing need for automated data-analysis methods. This trend is particularly evident in the development of Internet-connected services and systems, including the emerging Internet of Things (IoT), where millions, and sometimes hundreds of millions of active users or entities require the use of automated methods for data analysis. The problem to analyze such big–data volumes is vital for the further development of society and viable industries [6], and new tools are also needed to maintain security and law in the digital age. That is why machine learning, which deals with methods for automated data analysis, is an active and important field of research.

1.3

Towards Intelligent Industrial Processes

ICT plays a key role in the context of industrial processes and industrial automation. That is why LTU has developed the concept of ProessIT at the local and European levels (in the form of ProcessIT.EU). Taking the step towards intelligent industrial processes is a challenging and exciting enterprise, which will force us to think beyond conventional concepts and explore new domains. Refer to the parallel report on automatic control [3, Sections 2.2–2.4] for a complementary introduction to Industrial Processes and Intelligent Industrial Processes, which deals with intelligent qualities at the level of components like control systems. That report also outlines the idea and motivation for an Open Research and Innovation Platform, including open software and data needed to efficiently collaborate across academic and industrial borders. If such a platform can be supported and developed, for example at the European ProcessIT.EU level, it would provide a valuable framework and source of data for application and implementation of machine learning methods for industrial process automation purposes. Here I want to emphasize something that I think is important for the development of Intelligent Industrial Processes: The interaction between the process and it’s environment is a key aspect, with high potential for further development using modern ICT and machine learning methods. This concept is illustrated in Figure 1 (the illustration on the cover of this document), which includes an industrial process surrounded by a complex environment. The level of intelligence of the process can be quantified in terms of how well the process reaches goals in a large set of simulated environments, weighted by the complexity of each particular environment [7] (see Section 2.4 for further information). In practice, the complexity function is not computable so the intelligence measure needs to be approximated, but this mathematical definition provides a common basis for understanding and describing a property of systems that resembles natural intelligence. In this perspective the task to make processes more intelligent is an interdisciplinary

6

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Figure 1: Industrial processes exist in complex, changing and competitive environments, which for example include the society, employees, customers, market, energy systems, supply systems, transportation systems, maintenance systems and so on. Qualitatively, an intelligent process should enable us to reach goals in such complex and changing environments with a reasonable effort. This will require an integrated ICT platform for decision support and control, which integrates information originating from many different systems. enterprise that includes, but also goes far beyond adaptation and optimization at the level of isolated devices and constituent systems. The industrial and academic partnerships that have been developed in the context of ProcessIT is an excellent stepping stone for further work in this direction, which will open up for new possibilities to collaborate across disciplines.

2

Definitions and Basic Concepts

This section introduces some basic concepts and definitions, which may be helpful for unfamiliar readers to understand what this field is about. Readers already familiar with the topics discussed here can skip this section. For a proper introduction to machine learning see the book by Bishop [8] or the more recent book by Murphy [9].

2.1

Intelligent Industrial Processes

The area of excellence in research and innovation focusing on Intelligent Industrial Processes is defined in the following way. “A versatile and competitive industry is important for Sweden’s and Europe’s future status as new players are emerging. To secure our position, constant improvement and development of industrial processes are required in order to increase productivity while reducing the pressures on the climate and the environment. A key area is ProcessIT (or Process industrial automation) in which several Swedish companies are world leaders in its development, delivery and application. The area is very important in order to maintain and further develop a competitive national process industry. The area constitutes a global market with growth potential for SMEs through which they can grow by developing and commercializing innovations via corporate, university and divisional collaborations. LTU is leading in Sweden within the area of rendering more efficient basic industries

7

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

and other industries using information and communication technology, so-called ProcessIT. To establish ourselves as a player on the European stage, we make use of our multidisciplinary strengths and the networks within which we have a leading position.” See the parallel report on Automatic Control [3] for definitions of Industrial Process and an introduction to the concept of Intelligent Industrial Processes that complements the discussion in this report.

2.2

Enabling Information and Communication Technology

The area of excellence in research and innovation focusing on Enabling Information and Communication Technology is defined in the following way. “Enabling ICT has a broad base in LTU’s research resources within the information and communication field. Together, we drive the development of new ICT-based knowledge and applications in the areas of e-health, smart regions, data centers and cloud services. Information and communication technology (ICT) is expanding rapidly and the need for research is increasing. ICT has infinitely many applications in society and business, and the requirements for accessibility, usability, reliability and safety are huge. Within Enabling ICT we gather LTU’s research resources related to ICT in order to address the research questions both in depth and from a multidisciplinary perspective. A multidisciplinary research approach facilitates new ways of exploring research questions as well as innovation. Enabling ICT focuses on increasing and improving the application areas and usability of ICT. This is made possible through ambient ICT for everyone in all situations and contexts and addresses the context-integrated technology with multimodal interaction.” As outlined above, ICT plays an important role in the automation of process industries. Machine learning and the discussion in subsequent sections is relevant for Enabling ICT in general, but it is the context spanning IIP and Enabling ICT that is the primary focus here.

2.3

Machine Learning

Machine learning is about automated methods for data analysis, including signal and image processing. In particular, machine learning methods are used to automatically detect patterns in data that can be used for various forms of decision making, including prediction. For example, a machine learning system trained on documents can be used to recognize spam or handwritten letters, and a system trained on condition monitoring signals from a machine can learn to detect faults. 2.3.1

Relevance

There is a growing need for automated data-analysis methods due to the rapidly increasing volume and value of data generated in ICT applications. This trend is particularly evident in the development of Internet-connected systems and services, where world-wide information and communication systems with millions, and sometimes hundreds of millions of active users require the use of automated methods for data analysis. The problem to analyze such big–data is vital for the further development of society and viable industries [6], and machine-learning tools are also needed to maintain security and law in the digital age. Similarly, the development of the IoT will create new needs and challenges in machine learning research. The importance of this field is evident in terms of the annual number of publications. Figure 2 presents the number of articles per year during the last two decades, as returned by Scopus for the queries “machine learning”, “machine intelligence” and “artificial intelligence”. According to Scopus, about five thousand papers related to the general keyword “machine learning” are written annually. The corresponding number of publications with

8

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

2013 2012 2011

artificial intelligence machine learning machine intelligence

2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992 1991 1990 100

101

102 103 Papers per year

104

Figure 2: Number of papers published annually according to Scopus. These results are obtained with a query on the form TITLE-ABS-KEY(). the keyword “artificial intelligence” is just above ten thousands per year. The actual number of publications in the area is higher because these queries are limited to one keyword, and Scopus does not index all publications. For example, the query “machine learning” (“artificial intelligence”) in Google Scholar returns 212000 (169000) results between 2010 and April 14, 2014. The search interest of the general public appears different compared to the publication trend, see Figure 3. 2.3.2

Modern Approach

The modern viewpoint is that the best way to develop automated methods for data analysis is to use and further develop the tools of probability theory [9]. This is natural because uncertainty plays an important role in most real-world applications. There are two major definitions of probability, known as the frequentist and Bayesian interpretations. In the frequentist view probabilities represent estimated frequencies of events in repeated trials. The Bayesian interpretation of probability is different and concerns the uncertainty about some particular aspect of the data, which means that it is fundamentally related to the concept of information. The basic rules of probability are the same in these two interpreta9

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

2004 2005 2006 2007 2008 2009 Consciousness Artificial intelligence Machine learning Neural networks Internet of things Machine intelligence

2010 2011 2012 2013

Relative search interest Figure 3: Averaged annual search interest extracted from Google Trends on January 21, 2014. Bars indicate the relative number of searches that have been done for each particular term. The terms “internet of things” and “consciousness” are included as reference. Another relevant term with relatively high search interest is “cognitive”, which presently is about five times more common than “artificial intelligence”. The trend of the term “computational intelligence” is qualitatively the same as that of the term “machine intelligence”. tions, but there are some important differences. One advantage of the Bayesian approach is that it can be used to model the uncertainty of events that did not occur, which is not possible in a frequentist approach. Regardless of the approach used, proper model checking is mandatory [10]. Information Theory and Statistical Physics [11] are two closely related fields that also play central roles in the development of Machine Learning. The systematic application of probability theory to inference problems in terms of uncertainty is called the Bayesian approach, but in practice also non-Bayesian methods are used in that context. 2.3.3

Central Concepts

Information representation and generalization are two aspects that lies at the core of machine learning research and applications. Data need to be represented in a compact and informative way in order to enable association with other data, inferences, estimation of probabilities, evaluation of functions etc. The way in which information is represented can have a signifiant effect on the size of data, in particular in the presence of noise in the underlying raw data. Efficient representations of information are sometimes referred to ass succinct, meaning that they are both informative and requires little space (few bytes of storage). The problem to learn efficient representations of data is roughly divided in two parts: • Learning of low-level features, so-called shallow learning. This learning process is typically unsupervised, meaning that self-organizing methods that do not require 10

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

the supervision of humans are used. • Learning of high-level feature hierarchies and structures, so-called deep learning. This learning process can be unsupervised or supervised, meaning that information about the context or interpretation of data is provided to the learning system. Generalization refers to the capacity of a machine-learning system to solve tasks that are not explicitly trained, thereby generalizing known information to novel instances of data. Learning of efficient information representations and the conditions under which generalization can be guaranteed are central problems in computational learning theory. 2.3.4

Uses of Machine Learning

There are at least three important uses of machine learning: • Improve decisions using historical data (data mining). • Solve complex problems that are difficult or costly to program by hand (e.g. speech or image recognition). • Enable adaptation to varying and changing conditions (e.g. in the environment, user preferences, system characteristics and sensor–motor system). The typical result of a machine learning method is a model, which partially is a result of the design-time assumptions and choices, and partially is adapted in a probabilistic fashion to training data, see Figure 4. The resulting model can be used for different purposes, input

learning algorithm

low-level features

model

Figure 4: Typical information processing stages in a machine learning application. for example classification of data, prediction of future or missing data, and categorization of similar data into different clusters. The output of a machine learning method can also be a human-readable symbolic model such as an algebraic or differential equation. An important aspect, which separate this interpretation of machine learning from the concepts of natural and artificial intelligence, is that the machine learning approach does not specify what the resulting model or information should be applied to. The result needs to be interpreted by a human, or implemented in another technical system to become useful. As such, machine learning does not automatically make technical systems autonomous. 2.3.5

Feature Learning

Traditionally the low-level features needed to apply a machine learning or pattern recognition method to a problem have been handcrafted using first principles or ad-hoc “feature extraction” methods. This is still common practice in engineering applications of machine learning. The modern approach, which originates from computational neuroscience research in the mid 1990s (e.g. [12]), is to learn the low-level features from the input data. There are both unsupervised feature learning and deep learning algorithms that can automatically learn feature representations from unlabeled and unstructured data, thereby generating state-of-the-art representations and results on challenging benchmark problems. This is an active field of research, which has already provided many useful results that can be adopted and further developed by the engineering community. This topic is highlighted as interesting for further research and development in subsequent sections. A general feature learning process is illustrated in Figure 5. A feature learning system typically involves two components:

11

IIP & Enabling ICT: Machine Learning and Intelligence

Unlabeled examples

Unsupervised learning

April 22

Learned features

Encoder

Feature vector

Figure 5: Typical processing stages included in models for unsupervised feature learning. • A (low-level) feature learning algorithm, which learns a set of features (e.g. basis functions) referred to as a dictionary or codebook from the input data. • An encoding algorithm that translates the input data into a (hopefully more efficient) representation based on the elements in the dictionary. Recent results indicate that the details of the feature learning algorithm are important when the size of the dictionary is limited or needs to be kept small, while random samples from the data space can provide efficient codes when a large dictionary is used [13]. Feature learning is an active research topic in most challenging applications of machine learning, for example computer vision, speech processing and text analysis. 2.3.6

Model Selection and Checking

The classical problems of underfitting and overfitting applies also to modern machine learning methods, and the methodology addressing these problems is called model selection [9, Section 5.3]. The basic principle (Bayesian Occam’s razor) is that one should pick the simplest model that explains the data, which is a general principle known as Occam’s razor in Physics and other fields of science. If the model is too complex it will typically lead to overfitting of the data, while simplistic models lead to underfitting. Formally, model selection can be done using the Bayesian information criterion (BIC), which includes a penalty related to the model complexity. Alternative approaches exist, for example based on a principle known as the minimum description length. The Bayesian approach to inference is most useful, but complex models always need to be tested and falsified [10,14,15]. That is how we can discover and correct mistakes. When a model is false (which is always the case to some extent), Bayesian updates concentrate the posterior on the best approximate distribution of the data in terms of the likelihood, which can fail catastrophically depending on how the model is misspecified and how it represents the parameters of interest. Therefore, Bayesian model checking (with nonBayesian methods) is necessary. Good statistical practice is to make posterior predictive checks by creating simulations of the data and comparing the result to the actual data. For further information see the discussion in [10]. Frequentist models also have limitations, which can result in inconsistent results and misleading estimates of confidence intervals and p-values in hypothesis testing. See the section on pathologies of frequentist statistics in Murphy’s book for further information [9, Section 6.6]. 2.3.7

Need for Education

Today there is a substantial repertoire of machine learning methods that can be applied to a variety of problems. When applied to real-world data, most learning algorithms are unable to produce optimal results automatically. Trained professionals are needed to 12

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

develop and check models, often requiring prior knowledge about the problem at hand. In combination with the increasing strategical importance of machine learning, this creates a need for educating engineers in probabilistic machine learning. International online machine learning courses attract thousands of students worldwide and are taken by some of our Ph.D. students, but we need to develop the education at LTU in this field. Lennarts perspective on the year 2030 (translated from Swedish) By 2030 computers will be able to evaluate different actions and autonomously choose activities. For those activities that can be given an algorithmic description, for example mathematical calculations, this has been very successful. For activities that are limited by rules, but in every situation allows many choices – the classic example is the game of chess – we have not been able to emulate the human player’s ability to ignore the vast amount of unproductive moves. Experts can play many games simultaneously against amateurs – they do not need to think about which move to make, they recognize similarities in the games and play what ought to be a strong move. Maybe we have underestimated how sophisticated the role of human perception is in this game. For another game with built-in randomness, backgammon, an artificial neural network called TD-Gammon (TD stands for Temporal Difference Learning) is playing at the international top level since the 1990s. Remarkably, the network does not learn by playing against human elite players, but against itself. It should also be mentioned that TD-Gammon make moves which first appeared inferior, but which upon analysis proved to be efficient. TDGammon has thus made its own game-related discoveries. Unlike chess playing, brute-force solutions have not been successful in backgammon and it is argued that the random mechanism explains that. In the case of backgammon we thus see an example of machine learning, in the case of chess we do not. It deserves to be mentioned that there is one company – IBM – that have done the development and research behind successes like Deep Blue and TD-Gammon. (Comment: IBM also developed Watson, which outperformed Jeopardy players in 2011. Last year IBM announced that Watson software is successfully used for management decisions in lung cancer treatment. What can Watson technology do for process industries, for example in terms of maintenance decision support, production planning and customer relations?) By 2030 it has been understood how humans build up their skills for chess. Thereby it will also be understood how humans solve (hopefully many types of) problems of the type “many possibilities, few appropriate choices”. This will imply that less computing power is needed to solve tasks of this type. Today, significant progress has been made towards automatic face recognition with advanced image transforms. Extensive research is conducted, motivated by the importance of the application. Face recognition has the character of “many possible choices, one correct choice”. Computer Strategies for face recognition is quite different than for chess. Anyone who is somewhat familiar with neuroscience recognize the division of analysis steps for face recognition and recognize much of the terminology. The biology and technology have much in common in face recognition, quite in contrast to the chess game. Why is that so? Maybe because face recognition is a matter of “low-level”, albeit complex, perception. It is not apparent that any breakthroughs are needed for computers to surpass humans in face recognition.

2.4

Intelligence

Intelligence [16,17] is the capability to automatically reach goals in complex and changing environments [7], forcing the intelligent entity (sometimes referred to as an “agent”) to generalize prior information to new situations. Intelligent behavior apparently is a complex phenomenon, which has evolved to enable organisms to deal with variable environmental circumstances. The research aiming at describing, modeling and artificially replicating such behavior is known by several names. For example, “artificial intelligence”, “computational intelligence”, “machine intelligence” and similar terms all refer to the same class of phenomena, with some differences concerning detailed approach, terminology and history. In a machine-intelligence perspective the opportunities to perform actions in an environment is a potentially important concept known as affordances, which goes beyond the elementary concept of estimating probabilities, see for example [18].

13

IIP & Enabling ICT: Machine Learning and Intelligence

2.4.1

April 22

Motivation

A general motivation for developing automated methods for data analysis and industrial automation is introduced above. Nature offers proof of concept, and numerous prototype models from which we can learn and find inspiration needed to develop ICT and applications with intelligent qualities. For example, even a tiny animal like a bee with one million neurons in the brain have remarkably sophisticated learning and cognitive problem-solving capabilities [1]. A bee is also much better prepared to navigate and survive in the natural environment than current robots. The task to mimic the properties of high-level cognitive circuits in the brain of a bee is certainly challenging, but appears relatively simple compared to the current strategic objectives of world-wide initiatives focusing at understanding the human brain (see the description of the BRAIN Initiative and the Human Brain Project in the next section), which has several orders of magnitude more neurons and neuron interconnections than the brain of a bee. There are also other, more simple prototypical intelligent organisms, for example worms (www.openworm.org) and plants [19]. 2.4.2

From Machine Learning to Intelligence

There are several aspects of intelligent behavior that are not understood, and there is no clear definition that distinguishes the concepts of machine learning and machine intelligence from each other. However, there tend to be differences in the interpretation and approach. In particular, machine learning tend to focus on data-driven models for analysis or decision making purposes (typically involving a human that interprets the result and defines the consequence), while intelligence involves autonomous sensory-motor function in an environment, see Figure 6. Intelligence tend to be a more general concept, which reward observation entity

environment action

Figure 6: Interaction between an intelligent entity (for example an industrial process or mechanical component) and the environment. In this basic scenario the reward signal is supervised. The use of unsupervised reward signals is an open research problem, which includes modeling of affective states. depends on machine-learning principles for learning of sensor information representations and inference of motor signals. A fundamental aspect of sensor-motor loops is that they enable prediction of causal consequences of actions, providing a principled way in which to explore and sample the world [20]. In a sense, the environment serves as a dynamic cost/loss function in the case of intelligent systems, while loss functions in machine learning typically are predefined parts of the model. The complexity of actions that an intelligent entity can perform is directly related to the complexity of the sensory information that it can interpret. Therefore, the development of advanced sensory-motor functions depend on efficient information representations, including mechanisms for extraction of invariant features and feature categories (symbols) from multi-sensory information. The problem to learn high-level symbolic knowledge from unstructured information is a central problem in Artificial Intelligence, Machine Learning, Cognitive Science and Cognitive Computation.

14

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Lennarts perspective on the role of simulation in biology I appreciate the message from Hesslow in “Conscious thought as simulation of behavior and perception” which I interpret as “there’s nothing very mystical about consciousness” (See [21] and references therein). But I note that “his” consciousness is centered on cognition, whereas other treatises are heavy on emotions. Hesslow’s simulation hypothesis is convincing to me. As far as “cognitive consciousness” is concerned I don’t see any residual mystery. Simulation on a computer of the simulation hypothesis doesn’t seem insurmountable. It should also offer possibilities from a robotics/artificial intelligence viewpoint. How could an “inner world” manifest itself in a simulation? Let me discuss an example. An example of an inner world in a human. We drive a car on a stretch of a windy road. We see the road and its undulations and the traffic on the road, we feel the acceleration transmitted from the car to the driver’s seat. We hear a bit of road noise, particularly if we have winter tires. We turn the steering wheel so as to stay on the right side of the road and to pass slower traffic. Both sensory and motor cortices are employed by our external world. We can do this same drive attentively, enjoying the fall color scenery, but we can also drive almost as an automat, if we are experienced drivers, in our thoughts being elsewhere. Now we tell a friend about the same car ride. If we had been attentive we would have noticed and remembered enough details to make the story vivid. As we tell we recall the drive, probably not in equal details from start to end but in its characteristics – we saw a fantastic display of yellow and even red on a stand of aspen on the side of the road, we particularly enjoyed the undulations of the road, then after a curve we saw a moose in the middle of the road and had to put on the brakes, almost in panic, and drove the last part of the trip in a rather shaky state. We don’t have a recorded movie of the car ride somewhere in our brain, instead we construct the ride from fragments that we know (or, more realistically, believe) fit together. As we go through this process our visual cortex is quite busy seeing trees and road and other cars, even though we sit in a comfortable sofa, enjoying a good cognac. But when we conjure up the moose we quickly put down the glass, not to shake it as our premotor cortex goes wild, remembering the panic brake. Our primary motor cortex is not much affected, but just for safety’s sake. All this happens in our inner world. An example of an inner world in a robot? What about a robot’s external and inner world? The development of driving robots is well advanced and the trip in the external world can already be accomplished, without the emotional content of course. The inner world is not as advanced. The robot is equipped with a movie camera, accelerometers, microphones and plenty of memory to record all data. An all-sensory presentation can be offered to anyone, but this is obviously not an inner world of the robot. A better resemblance to an inner world would be a “documentary” where short and “most relevant” intervals were chosen by the robot for presentation. It still would not be an inner world because the final presentation would consist of simply recordings. Can we come closer to robotic inner world? Imagine that our robot is an experienced motorist. It has made statistical descriptions of its previous trips and includes the present trip in the statistics but also makes notes of its peculiarities, particularly the moose incident. The robot, asked to tell about the trip, then accesses the statistics and the peculiarities and produces an animation. Less details, not all necessarily true of the present trip, but good enough to be shown or told by the robot. The selection of relevant material and making of the animation comes close to a robotic inner world as I understand it. You could say, and I would agree, that a purely cognitive inner world would be a meager world indeed. Could such a cognitive consciousness, or cognitive inner world, offer possibilities from a robotics/artificial intelligence viewpoint, as suggested above? Possible usefulness of inner worlds; theory of mind. The robot would need to extract the interesting information about the trip. It would seem like an easy task – the peculiar facts were stored as such, but let us expand the trip a little. Let us assume the robot saw a dangerous passing maneuver that led to a head-on collision. The robot naturally stored this as a peculiarity. Later, when interrogated by the police, it should tell about the collision, not about the moose. Easy enough to comprehend, besides the police would prod it in the right direction, if necessary. But a more general ability to assess an audience in order to anticipate its interests and choose relevant facts (or fabrications?) for presentation would of course be useful for communication with humans (and other robots?). This comes close to saying that the robot could use some “theory of mind”. In other words, the robot should have a sense of the inner world of others, be they humans or other robots. This in turn connects to the previously mentioned “emotional computing”, computing intended to recognize the emotional state of a human, with the purpose of establishing good communication.

15

IIP & Enabling ICT: Machine Learning and Intelligence

2.4.3

April 22

Formal Definition

In principle it is possible to formally define intelligence mathematically (so-called Universal Intelligence) in terms of complexity, information and computation [7]. Universal intelligence is defined by a mathematical equation, which is appealing because it leaves little room for ambiguities and different interpretations. Essentially, the intelligence of an entity is defined by a weighted sum of its performance over the space of all possible environments, where the weights are defined by the (approximate) complexity of each environment, see [7, p. 39].

3

Strategic Agendas and Objectives

This section presents some selected challenges and objectives described in international agendas, which illustrate a broader view on machine learning and intelligence, and technological developments that are targeted and expected in the foreseeable future.

3.1

ICT in Horizon 2020

EU investments in ICTs are due to increase by 46% under Horizon 2020 compared to FP7. This investment will support riskier ICT research and innovation that can deliver new business breakthroughs, often on the basis of emerging technologies. In particular, Horizon 2020 will support the development of [22]: • A new generation of components and systems including micro / nano-electronics and photonics technologies, components and embedded systems engineering, • Next generation computing, Advanced computing systems and technologies, • Infrastructures, technologies and services for the future Internet, • Content technologies and information management, including ICT for digital content and creativity, • Advanced interfaces and robots and Robotics and smart spaces. The work program “Leadership in enabling and industrial technologies Information and Communication Technologies” (LEIT) is of particular interest for the applied and industrially-motivated research at LTU. This program addresses a broad range of research needs motivated by strategic research agendas and objectives. It covers technology-driven research and development that is mostly application-independent, complemented by more application-driven research and innovation where components and systems are demonstrated, instantiated, integrated and validated. Calls within LEIT that are particularly relevant in the context considered here are summarized below. ICT 1: Smart Cyber-Physical Systems (CPS) This call focuses on the next generation of embedded ICT and IoT systems that are interconnected and collaborating, providing citizens and businesses with a wide range of innovative applications and services. These systems are increasingly embedded in all types of artifacts resulting in “smarter”, more intelligent, more energy-efficient and more comfortable systems, including transport systems, cars, factories, hospitals, offices, homes, cities and personal devices. ICT 2: Smart System Integration The aims of this call are to develop the next generations of smart systems (predictive, 16

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

reactive and cognitive) technologies and solutions based on miniaturisation and integration of heterogeneous technologies and functions and materials; and to establish European competitive ecosystems for the design, R & D, prototyping and testing, manufacturing and industrialisation of such systems. ICT 16: Big data – research This topic contribute to the Big Data challenge by addressing the fundamental research problems related to the scalability and responsiveness of analytics capabilities, such as privacy-aware machine learning, language understanding, data mining and visualization. Special focus is on industry-validated, user-defined challenges like predictions, and rigorous processes for monitoring and measurement. ICT 22: Multimodal and Natural computer interaction This call focuses on the interfaces between humans and devices and systems, which often is lagging behind and constitutes a bottleneck for seamless and efficient use of ICT. A multidisciplinary approach combining knowledge from both the technological and human sciences is supported. The goal is that new technologies should offer interactions that are closer to the communication patterns of humans, allowing for simple, intuitive and hence more “natural” communication with the system. ICT 30: Internet of Things and Platforms for Connected Smart Objects This topic focuses on the challenge to deliver an Internet of Things (IoT) extended into a web of platforms for connected devices and objects with dynamic and adaptive configuration capabilities supporting smart environments, businesses, services and people. The major challenge will be to overcome the fragmentation of vertically-oriented closed systems, architectures and application areas and move towards open systems and platforms that support multiple applications. This topic cuts across several LEIT-ICT challenges (smart systems integration, cyber-physical systems, smart networks, big data) and brings together different generic ICT technologies such as nano-electronics, wireless networks, low-power computing, adaptive and cognitive systems.

3.2

European Technology Platform on Smart Systems Integration

The European Technology Platform on Smart Systems Integration (EPoSS) is an industrydriven policy initiative contributing to EU’s strategy for the coming decade (EUROPE 2020) to become a smart, sustainable and inclusive economy. EPoSS involves major industrial companies and research organizations from more than twenty member states and defines research and development needs as well as policy requirements related to smart systems integration, including micro- and nanosystems [23]. EPoSS focuses on Smart Systems defined as intelligent, often miniaturised, technical subsystems with independent functionality that evolves from microsystems technology. According to EPoSS Smart Systems are able to sense and diagnose complex situations. Such systems are “predictive” and have the capability to make decisions and interact with the environment, as well as providing decision support. Smart Systems may also be energy autonomous and networked. The agenda includes discussions of Smart Systems in the context of the following application areas • Automotive, • Medical, • Internet of Things (IoT), • Information and Telecommunication,

17

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

• Safety and Security, • Aerospace. Three generations of Smart Systems technology are outlined in the agenda, starting with the technology that is commonly available already today (the first generation). • First generation systems include examples like object recognition devices, driver status monitoring systems and multifunctional devices for minimal invasive surgery. • Second generation systems include examples like artificial organs, advanced energy management systems, and environmental sensor networks. These systems are predicted to affect nearly all aspects of our daily life. • Third generation systems will combine technical “intelligence” and cognitive functions. The IoT is given as a primary example, where third generation technology will provide the indispensable interface between the virtual and the physical world. The chapter focusing on Smart Systems for the IoT (Chapter 7) is particularly relevant. IoT is a central research theme that bridges the activities within IIP, Enabling ICT and ProcessIT at LTU. From a technological point of view the following research themes are highlighted in the EPoSS agenda as necessary for the further development of IoT. • Energy harvesting, storage and efficiency. The goal is to realize nearly isentropic devices. • Intelligence of devices, in particular as regards context awareness and inter-machine communication. Context awareness is strongly related to information received via sensors, corresponding sensor networks and localisation capabilities, as well as possibilities to actuate on the environment. Context identification can also be social and user related. • Communication, in terms of physical wave transmission and protocols, will be the cornerstone of IoT. Machine-to-Machine technologies with context-awareness and situation-specific behaviour will be central. • Integration of chips and antennas on non-standard substrates like textiles and paper, even metal laminates and new substrates with conducting paths and bonding materials adapted to harsh environments and for environmentally friendly disposal. • Interoperability between devices regardless of communication standards and protocols. • Trust and Security is vital and requires a technically sound solution to guarantee privacy and security of customers. In addition to this it is pointed out that novel programming paradigms and further development of energy-efficient protocols and smart antennas are required to enable efficient processing and communications.

3.3

BRAIN Initiative

The US Brain Research through Advancing Innovative Neurotechnologies (BRAIN) initiative is a substantial investment in research and development focusing on providing the tools needed to study the brain and brain deceases [24]. With nearly 100 billion neurons and 100 trillion connections, the human brain remains one of the greatest mysteries in science and one of the greatest challenges in medicine. The goal is to accelerate the development and application of innovative technologies needed to produce a dynamic picture of the brain that, for the first time, shows how complex neural circuits interact in both 18

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

time and space enabling researchers to seek new ways to treat, cure and prevent brain disorders. It is expected that this development will fill major gaps in our current knowledge and provide opportunities for exploring how the brain enables us to record, process, utilize, store, and retrieve vast quantities of information at the speed of thought. The aim of this initiative is partially coherent with that of the Human Brain Project (outlined below). Both projects highlights the urgent need for developing a better understanding of the human brain, in particular because brain deceases are costly and increases linearly with the average lifetime of the population. There appears to be more focus on brain-inspired technology development in the Human Brain Project compared to the BRAIN Initiative.

3.4

Human Brain Project

The goal of the Human Brain Project [25], which is a European Flagship, is to build a new information and computing technology infrastructure for neuroscience and brainrelated research in medicine and computing. It supports a global collaborative effort to understand the human brain and its diseases, and ultimately to emulate its computational capabilities. It is expected that the most immediate impact on the European society will come through the impact on healthcare. The work plan of the project include (but is not limited to) the following parts. • Work in theoretical neuroscience will investigate the mathematical principles underlying the relationships between different levels of brain organisation and the plasticity mechanisms that subserve the acquisition, representation and long-term memorisation of information about the outside world. The results will help to identify the critical data needed for modelling, and to simplify detailed brain models for implementation in IT and specifically in neuromorphic computing systems. • The HBP will build and operate an integrated system of six ICT platforms providing high-quality services to researchers and technology developers inside and outside the HBP. Two of these platforms are particularly interesting: The Neuromorphic Computing Platform will allow non-expert engineers to perform experiments with Neuromorphic Computing Systems (NCS), hardware devices incorporating simplified versions of the brain models developed by the Brain Simulation Platform. The platform will provide access to three classes of NCS: systems based on physical (analogue or mixed-signal) emulations of brain models running much faster than real time; numerical models running in real time on digital multicore architectures; and hybrid systems. The Neurorobotics Platform will allow industry researchers to experiment with virtual robots controlled by brain models developed in the project. This platform will offer scientists and technology developers a software and hardware infrastructure allowing them to connect brain models to detailed simulations of robots, and will support the development of neurorobotic systems for applications in specific domains (manufacturing, services, automatic vehicles etc). It will also enable closed-loop experiments and studies of neuronal mechanisms responsible for specific cognitive capabilities and behaviors. The project description also includes descriptions of the expected impact in various fields, some of which are central for the development of intelligent machines and the further development of machine learning methodologies. Below is a summary of some parts that appear particularly relevant in the context considered here. Cognitive architectures (Section 1.2.1.3) Neuroimaging has greatly refined our understanding of cortical and subcortical function. 19

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Thanks to these techniques, we have relatively precise information about the areas of the human brain responsible for processing particular categories of visual information, so-called core knowledge systems, language processing, and representing other people’s minds (theory of mind). Recent studies characterise areas and regions in functional terms, focusing for example on internal coding principles and how activation varies with stimuli and tasks. High-resolution fMRI and multivariate analysis of activation patterns do in some cases allow precise inferences about neuronal codes. A hierarchical Bayesian perspective emerges as a cross-domain unifying principle: neuronal populations act as statistical predictive-coding devices that represent priors, sensory evidence, and posterior probabilities used to infer and anticipate upon external events. The project will focus on the following functions • Perception-action: Invariant visual recognition; mapping of perceptions to actions; representation of action meaning; multisensory perception of the body and the sense of self. • Multimodal sensory-motor integration. Integration of data from vision, audition, body representations and motor output. • Motivation, decision and reward. Decision-making; estimating confidence in decision and error correction; motivation, emotions and reward; goal-oriented behaviour. • Learning and memory. Memory for skills and habits (procedural memory); memory for facts and events (episodic memory); working memory. • Core knowledge of space, time and numbers. Fundamental circuits for spatial navigation and spatial memory; estimation and storage of duration, size and numbers of objects. • Capabilities characteristic of the human brain. Processing nested structures in language and in other domains (music, mathematics, action); generating and manipulating symbols; creating and processing representations of the self in relation to others. • Architectures supporting conscious processing. Brain networks enabling the extraction and maintenance of relevant information; representations of self-related information, including body states, decision confidence, and autobiographical knowledge. Future computing (Section 3.1.4) Computing technology faces obstacles that could significantly change the trend that has existed over the last fifty years. Specifically, with ever-increasing numbers of processing units the power consumption and the probability of component failures rise, potentially to unmanageable levels. These problems create a demand for new computing paradigms, in particular inspired by the architecture of the brain. The brain is different compared to computing systems in several ways, for example: it is made of heterogeneous components (a property recently shown to confer robustness to the system), the components behave stochastically (it is not possible to predict the output given the input), the components can switch dynamically between communicating synchronously and asynchronously, and each recipient neuron appears to give its own unique interpretation to the information it receives from other neurons. The brain is also hierarchically organised with massive recurrent connectivity and a small-world topology, which is completely different from the architecture of modern computers. Neuromorphic computing systems (Section 3.1.4.3) There has been great progress in addressing intractable problems such as driving automatically through city streets, understanding complex queries in natural language, and 20

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

automated machine translation. However, such applications require an enormous programming effort and powerful computing resources that require large amounts of energy. Therefore, there are large areas of the economy and of daily life in which ICT has a limited impact. The International Technology Roadmap for Semiconductors identifies neuromorphic technology – hardware devices inspired by the massively parallel architecture of the brain – as one of the most promising strategies for overcoming these limitations. Building neuromorphic-computing systems that can learn tasks without explicit programming is a key goal of the project. One such task is the extraction and categorisation of highlevel information from noisy sensor data. Potential applications include computer vision for robots, vehicles and industrial machinery; data mining for research, marketing and policing; real-time analysis of financial data for fraud detection or rapid detection of market trends; and monitoring of large-scale telecommunications, power distribution and transport networks. Neuromorphic computing will be especially valuable in applications requiring low power use and high resilience to failure, for example in large-scale environmental monitoring and monitoring in harsh industrial environments. Like the brain, such systems will have the ability to create implicit models of their environment, including abilities to predict the likely consequences of their decisions, and to choose the action most likely to lead to a given goal. Although less flexible and powerful than the human brain, such systems will be able to perform tasks beyond the capabilities of current ICT. Examples include technical assistance to humans, real-time diagnostics of complex machinery, autonomous navigation, self-repair, and health monitoring. Neuromorphic controllers will make it possible to automate sectors requiring nonrepeated actions that are difficult to standardize, for instance in the construction industry, services and the home. Though the first systems using neuromorphic controllers will probably be limited to relatively simple tasks, improvements in perceptual, motor and cognitive capabilities will allow more complex tasks.

3.5

Roadmap for U.S. Robotics: From Internet to Robotics

This roadmap [26] is based on input from 160 people and nine top universities in the US in the form of five workshops focusing on business/application drivers; current set of gaps to provide solutions to end-users; research and development priorities to enable delivery on the business drivers. In other words, the roadmap focus on robotics as an economic enabler. The workshops were topical across manufacturing, healthcare/medical robotics, service robotics, defense, and space. A roadmap for each one of these areas is included in the document. Perception in complex and unstructured environments is a challenge that requires further research. For example, a major barrier to the use of robots in factories is the high cost of engineering workcells, which typically are several times the cost of the robotic hardware. Robots need to perform tasks in environments with greater uncertainty than current systems can tolerate, and they need much improved perception systems in order to monitor the progress of their tasks and the tasks of those around them. Robots should also be able to estimate the emotional and physical state of humans, since this information is needed to maintain maximal productivity. One of the most complex fundamental problems that needs to be addressed is the integration of low-level continuous perception-action loops with high-level symbolic reasoning through the use of appropriate information representations. Future robots will exploit flexible and rich skill representations, and will observe humans and other robots to learn new skills autonomously. Architectures facilitating principled programming for agile, adaptive systems for uncertain environments involving direct physical and/or non-physical interactions with human users are needed.

21

IIP & Enabling ICT: Machine Learning and Intelligence

3.6

April 22

Neuro-IT roadmap

NeuroIT.net is a Network of Excellence sponsored by the FP7 FET initiative, which involve over 200 researchers from more than 100 institutes and SMEs in 16 nations. The aim is to build a critical mass of interdisciplinary research at the interface between Neurosciences and Information Technologies within the European Union and its associated states. The objective is to complement and move beyond the well established Neuroinformatics and Artificial Intelligence domains by fostering research addressing fundamental problems linked to modelling of cognitive and awareness processes. Focus is on unexplored research domains that can lead to breakthrough in the long term. A central guiding question is: “What can neuroscience do for IT”? Eight research directions are highlighted in the roadmap citeNeuroIT: • Brainship: Bi-directional brain computer interfaces and applications. • Bio-inspired hardware: The next generation of neuromorphic hardware. • Factor-10: Aims at a fully functional physical artifact that autonomously grows the volume of its body and its cognitive abilities (its “IQ”) by at least a factor of ten. • Acting in the physical world: Build complete systems which make optimum use of distributed intelligence embedded in the periphery (sensors, actuators, body morphology and materials) and at a system integration level. • Conscious Machines: Explore the the role of consciousness in forming the flexible adaptive behaviour of human beings, including deciding which information that is most likely to be behaviourally important. • Artificial Evolutionary Design: Automated design of artificial cognitive systems inspired by biological learning and evolution. • Constructed brain: A framework that allows for simulation of an entire brain, which can be used to enable systematic development of cognitive engineering principles within NeuroIT. • Tools for Neuroscience: Tools needed to study high-level neuronal levels of integration, critical to understanding brain function.

3.7

European Roadmap for Industrial Process Automation (ProcessIT.EU)

ProcessIT.EU was formed in 2010 and became an ARTEMIS Center of innovation Excellence in early 2011. This initiative is primarily focused on process automation and ICT for process industries and was formed by various partners, including end users, technology suppliers, academia, and public authorities. ProcessIT.EU is innovation driven and oriented towards identifying and implementing project activities that focus on new competitive automation technologies. LTU plays a central role in the creation and coordination of ProcessIT.EU. A first roadmap was published in 2013 [27], which is based on analysis of global trends and industrial needs, and aims to provide inspiration for future research and project proposals towards a set of “ideal concepts” that are described in the roadmap. The following parts of the roadmap appears particularly relevant and suited for application of machine learning and intelligence methods. Human-Machine Interface and Machine to Machine Communication Machine to Machine (M2M) communication needs to be further developed to enable evolvable industrial automation systems and to practically make use of the benefits of M2M while keeping manual configuration at a minimum, for example by the use of machine 22

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

learning methods. The development of human-machine interfaces must continue in order to improve the possibilities for efficient plant operations. It is expected that visualization, virtualisation, and simulations of a plant and its automation system will be introduced into daily operations, and that collaborative automation using networked services will play an increasingly important role. Instant Access to Virtual Dynamic Factory A virtual representation of factories that remain valid over the entire life cycle thereby allowing for accurate simulations is highlighted as a key enabler. This requires studies of robust parameter and state update mechanisms for complex dynamical models since real-world parameters are rarely constant. Mechanisms that allow for fast and robust virtualisation of the control system and its pairing with the virtual factory are needed. The goal is to enable a seamless transition between the virtual and physical representations, for example by providing models as a service of hardware components. Real-time Sensing & Networking in Challenging Environments Accurate real-time sensing is needed to increase the availability and uptime of processes, in particular by detecting problems at an earlier stage so that maintenance stops can be planned, thereby avoiding expensive unplanned stops. Robust sensing is also needed to verify virtual factory models. Sensors need to be designed for easy installation and should be compatible with surrounding systems while requiring a minimum amount of configuration. There is a need for context-driven, user-centric information, often involving big data that requires new methods for data processing. Management of Critical Knowledge for Maintenance Decision Support There is a need for technologies and methods to avoid “too” big data, for example in the form of ubiquitous self-diagnostic components, context awareness enabling only the information needed to be communicated, and efficient data to information transition and validation.

3.8

Comments and Interpretation

Several challenges and objectives that are highlighted in the ICT calls of Horizon 2020 matches current research initiatives and activities at LTU, for example in the Arrowhead project [28] and in the context of ProcessIT. The challenges and objectives outlined in ICT 1, ICT 16 and ICT 30 are particularly interesting from the perspective of machine learning and intelligence research. The challenges and objectives that are highlighted in the EPoSS agenda in the context of IoT matches current research initiatives at LTU well, since we are addressing several of these challenges. In particular, we are already developing ideas and concepts related to the third generation of technology and some of the highlighted research challenges, including aspects like cognitive functions, context-dependent processing, machine-machine interoperability, energy harvesting, and integration of electronics on non-standard substrates. The Human Brain Project aims to develop a new generation of neuromorphic hardware, artificial neural systems (typically in silicon) that mimics circuits and systems in biological nervous systems. Over the past two decades neuromorphic engineering research has focused on understanding low-level sensory processing and systems infrastructure. Efforts are now expanding to apply this knowledge and infrastructure to addressing more complex problems in perception, cognition, and learning. LTU has initiated collaboration with neuromorphic engineering groups at the Institute of Neuroinformatics (INI) in Zurich and the Cognitive Interaction Technology Center of Excellence (CITEC) in Bielefeld in a joint STINT-funded project focusing on bio-inspired computation (grant number IG20112025). In contrast to the objectives outlined in the Human Brain Project, which deals with

23

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

hardware for accelerated-time neuroscience simulations that is unsuitable for most applications involving interactions with a natural environment, our focus is on biological-time neuromorphic sensors and processing architectures that are suitable for applications. The integration of low-level continuous perception-action loops with high-level symbolic reasoning through the use of appropriate information representations is one of the fundamental enabling problems that are highlighted in US Robotics roadmap. LTU and it’s associated partners have some experience of modeling high-level symbolic reasoning, see for example [29], and the integration of low-level continuous perception-action loops in combination with challenging applications is a natural topic for further research. The NeuroIT roadmap is interesting as it aims beyond the well established Neuroinformatics and Artificial Intelligence domains, focusing on fundamental problems linking modeling of cognitive and conscious processes. Several roadmaps highlight the need for capabilities to improve the interaction with humans in various applications, including detection of emotional states and affective computing [30]. Lennarts perspective on the year 2030 (translated from Swedish) Quite recently the possibility to equip computers with affective, or emotional, skills has started to be explored. The intention is to improve human–machine interactions by making machines detect human affective reactions and can take them into account. By 2030 you may forget that you are talking to a machine in a conversation with a robot. The robot can even fake human emotions (people can too, though rarely fully convincingly). Today we know much more about human cognitive abilities than her affective qualities, her emotions. There is a tradition to consider rational behaviour and emotions as each other’s opposites and there are certainly examples of irrational, emotionally driven actions. On the other hand, “unemotional” is not a positive review of human behaviour. Why, then, do people have affective qualities alongside their cognitive? Or, maybe the question is better phrased as “Why do people have affective qualities alongside their, evolutionary newer, cognitive qualities?”. It is not obvious how to provide computers with their own affects, and that concept may even be frightening. On the other hand, a human without affects can also be frightening. There are reasons to believe that a human affectively values and chooses what activities she cognitively treats. A note on the role of emotions (not translated): As I see it emotions is something that gauges the state of an organism. Different emotions gauge different aspect of this state. Hunger gauges level of sustenance, fear gauges danger, etc. Lust is at the center of it all, it gauges possibility of reproduction. It is all about the survival of, perhaps not the individual organism, but of the genes it carries. Emotions are not binary but I think of them as allowing only low resolution (how happy are you, on a scale of one to ten?). They sum up an individual’s state in a vague but essential way. A note on the role of cognition (not translated): In the competition for survival cognition comes in handy. It may subserve emotions. Maybe it can take on some tasks of emotions completely – regular meals may substitute for hunger, clever foresight may forestall danger – but some emotions, such as happiness and lust are essential to the value of life and while they could perhaps be helped along by cognition they should not be substituted for. Emotions are accessed by the highest cortical structure, the prefrontal cortex, and can therefore exert influence on cognition, and even dominate “thinking”. One such influence of emotions is on choice of action where it is claimed to steer the individual away from the choice with the most unfavorable consequences – most unfavorable in the short term i.e. (it is not easy to see the survival value of such a strategy!). A dichotomy is often expressed for thinking and feeling, i.e. for cognition and emotions.

It should be pointed out that while these strategic agendas focus entirely on the human brain and human-level intelligence (motivated by healthcare), there are good reasons to find inspiration in other types of animals in the work towards intelligent machines and processes. See for example [1] and references therein. The ProcessIT.EU roadmap includes several challenges and ideal concepts that are 24

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

suited for development and application of machine learning and intelligence methods. It is the only roadmap focusing on industrial processes that I consider here because it is up-todate and based on former roadmaps in the area, at international and national levels, and it is written in close collaboration with people at LTU who are familiar with relevant needs and opportunities. Refer to the review of strategic agendas in the parallel reports [3, 4] focusing on automatic control and communications, respectively, for complementary views including additional roadmaps for industrial process automation.

4

SWOT Analysis

This section outlines a first attempt to analyze strengths, weaknesses, opportunities and threats (SWOT) concerning the further development of machine-learning and intelligence related activities at LTU. This is my personal view and it may not reflect the general opinion. Further discussion is required before the analysis can be used and presented in a roadmap. The SWOT analysis is summarized in Figure 7 and it is briefly commented below.

STRENGTHS   -­‐  Related  subjects  represented   -­‐  Applica9on  areas  represented   -­‐  Relevance  for  LTU  focus  areas   -­‐  Basic  lab  resources  exist   -­‐  Mo9vated  individuals  

OPPORTUNITIES   -­‐  Increasing  relevance  for  society   -­‐  Industrial  interest  and  relevance   -­‐  Established  contacts  with  industry   -­‐  Par9cipa9on  in  large  EU  projects   -­‐  ProcessIT  recognized  at  EU  level  

WEAKNESSES   -­‐  Subject  does  not  exist   -­‐  Subcri9cal  and  scaDered  mass   -­‐  Lack  of  high-­‐impact  publica9ons   -­‐  Insufficient  basic  educa9on   -­‐  Diversifica9on  of  faculty  

THREATS   -­‐  Insufficient  funding  of  faculty   -­‐  Unable  to  aDract  students  /  seniors   -­‐  Unable  to  combine  applied  and   groundbreaking  research   -­‐  Loosing  key  faculty  members   -­‐  Prejudice,  obsolete  viewpoints,  hype  

Internal

External

Figure 7: Identified strengths, weaknesses, opportunities and threats (DRAFT). The strategic agendas and objectives outlined above show that machine learning and the development of ICT with cognitive and intelligent qualities is a strategically important topic. Therefore, a weakness is that there is no such research subject and education at LTU. Students are not educated in modern statistics, advanced programming and other concepts needed to be well-prepared for research and development in this area. Fortunately, there are several related subjects and research areas with faculty members showing interest and competences in the field, forming a basis and stepping stones for the further development and organization of activities in this area. Some examples are: • Computer Science, • Control Engineering, • EISLAB, • Mathematical Statistics,

25

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

• Operation, Maintenance and Acoustics, • Scientific Computing, • Signal Processing. There are also numerous application areas and industries showing interests in the field, for example process industry, mining, energy, and related technology suppliers such as SKF. Application-driven projects tend to require a broad range of technical competences, which in combination with few faculty members creates diversification and a risk that groundbreaking research needed to make progress and develop educated state-of-the-art skills in this area cannot be combined. In general, the funding situation in academia is a major threat, as diversification forces many researchers to work overtime with duties of administrative character, thereby making insufficient room for creative developments leading to high-impact publications.

5

Open Problems and Research Directions

Animals have remarkable abilities to learn and autonomously adapt in ever changing and uncertain environments. Biological learners are able to automatically construct effective internal representations of percepts as part of the learning process, and to infer appropriate actions needed to reach goals (apparently defined by conscious emotions) necessary for survival. This is not so in technology, creating a remarkable gap and potential for further development of automation technology. In practice all current approaches to machine learning and intelligence require human supervision, for example to • Data analysis (initial, exploratory etc.) • Design the architecture, • Choose learning algorithm, • Select training data, • Design feature representations or feature learning algorithms, • Select exogenous learning parameters, • Perform model selection and checking, • Decide when to stop learning, • Update models when conditions change. Although we are beginning to understand learning systems used by brains, many aspects of autonomous learning remains a challenge. The strong dependence on human supervision is greatly retarding the development and deployment of autonomous learning systems, and present research is addressing these problems in many different ways. Research and development of brain-inspired information technology aims to provide new mathematical theories and models, computer simulations of brain circuits and sensory-motor systems, and neuromorphic sensory-motor systems needed to address this challenge (see Section 3).

26

IIP & Enabling ICT: Machine Learning and Intelligence

5.1

April 22

First Principles

Science and engineering are based on a methodological, model-based approach. The central idea is that the behavior of any physical system is governed by underlying dynamical equations, and much work is concerned with discovering these dynamical equations and making use of their consequences. A formal explanation of why it is possible to discover and define comprehensible mathematical models of complex physical systems has challenged physicists and mathematicians for centuries. It is only in the last few years that some insights have been made concerning why and to what extent that is possible. Using complexity theory and a general Liouvillian description of physical systems (classical or quantum) it was recently proven [31] that extracting the dynamical equations from experimental data, however precise, is a computationally hard problem (it is NP hard). As far as I know, that work presents the first algorithm for extracting the underlying dynamical equations from experimental data, providing a novel and important result in the context of system identification theory [32] and a challenge for the strategical objective to develop accurate virtual models of plants and factories. This result implies that finding the dynamical equations that best approximates data, or testing a dynamical model against experimental data, are in general computationally intractable. Why, then, is the model-based approach so successful and commonly used? The authors conclude that [31] “Experience would seem to suggest that, while general classical and quantum dynamical equations may be impossible to deduce from experimental data, the dynamics that we actually encounter are typically much easier to analyze. Our results pose the interesting question of why this should be, and whether there is some general physical principle that rules out intractable dynamics”. A related note is that a process with m input variables and n output variables can be controlled in 2mn different ways regardless of the particular control algorithm used, leading to a combinatorial problem that is challenging by itself. A remarkable aspect of most results in science and engineering is that they involve mathematical models that are valid at some scale, independently of the underlying, shorterscale details. For example, physical models have a hierarchical structure, which is apparent in the sense that physical models can be systematically renormalized into macroscopic effective models. Why is that so? An interesting result addressing this question was published last year [33], which demonstrates that models taken from diverse areas of science show weak dependence of macroscopic observables on microscopic details. This tendency is named “parameter space compression”, and it outlines why relatively simple mathematical approximations of microscopically complex and uncertain physical systems can be formulated. Perhaps this tendency also applies to industrial plants, offering possibilities to accurately describe complex physical systems with relative simple mathematical models? Clearly, some industrial processes are successfully modeled by professionals using first principles and system identification techniques.

5.2

Machine Learning

The aspects that are outlined above concerning first principles are relevant also in machine learning because the key problem is to automatically discover the structure of a system or information, so that useful inferences and predictions can be made. Such models can be realized both in the form of “hybrid models” and combinations of “soft” and “hard” models, where first principles are combined with empirical or probabilistic models, or in the form of fully data-driven probabilistic models. One interesting, recent example of the latter type is multi-modal symbolic regression (MMSR), which is a learning algorithm for construction of non-linear symbolic representations of discrete dynamical systems from unlabeled, time-series data [34]. Hierarchical representations of images and other types of data are successfully learned and used in the numerous applications of deep learning [35], which give state-of-the-art results on many different benchmark problems, including possibilities to prove learning and complexity properties [36]. It is clear from the circuitry of brains

27

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

2013 2012

compressed sensing sparse coding

2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 2000 100

101

102 Papers per year

103

Figure 8: Number of papers published annually including the keywords “sparse coding” or “compressed sensing” according to Scopus. This result is obtained with a query on the form TITLE-ABS-KEY(). The sparse-coding trend is nearly exponential, indicating that the concept is general and have broad implications [38,39,42]. The citation trend shows a similar exponential tendency [39], and has surpassed ten thousand citations globally per year. and the dynamics of neuronal learning that biology uses deep hierarchical structures for learning. In order to make significant progress toward human-like relatively autonomous learning capabilities, it is of fundamental importance to explore deep learning architectures and algorithms [37]. A related field of research has emerged during the last decade based on the concept of sparse representations [38, 39], which initially were motivated in terms of self-organization in the visual system [12], and concerns the basis of why deep learning is at all possible. Sparse representation is closely related to another quickly emerging topic known as “compressed sensing” or “sparse sampling”, which concerns signal-sampling. The Nyquist– Shannon sampling theorem states that a signal can be perfectly reconstructed from discrete samples if the highest frequency in the signal is half (or less) of the sampling rate. However, given information about the structure of a signal, for example through learning / optimization, it is possible to reconstruct the signal with fewer samples than the Nyquist–Shannon theorem suggests [38–40]. The number of publications in this field increases rapidly, see Figure 8, and the concept is successfully applied to various problems, such as feature learning and compressed imaging with one-pixel cameras, see for example [41]. LTU has initiated some work in this area within the SKF University Technology Center [43] in the context of condition monitoring. In general, the concept of sparse representation and sampling offer new perspectives on several challenging problems [39]. For example, I see the following possibilities for further research and development: • Sparse representations and models of complex and dynamic physical systems, for example process models and models of machines for condition-monitoring and maintenance purposes. This includes the problem to adapt information representations and models when the system changes, for example as a consequence of maintenance or replacement of mechanical or electronic components. 28

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

• Sparse representations of unconventional and cross-modal [44] sources of information, such as documents, biometric data, affective states of workers, and states of complex systems such as a customer or the market. Integration of such information representations in decision management systems and human-machine interaction systems, for example in the form of learning by apprenticeship / imitation. • Sparse and grounded [45, 46] representations of information and services in collaborative automations systems, possibly enabling a semantic-invariant and automated approach to system interoperability [2]. • Sparse representations and specifications of “algorithms” (sensory-motor programs), possibly with inherent adaptation abilities, leading to a radically new approach to programming. An algorithm exists in a high-dimensional mathematical space where all useful algorithms span an extremely sparse subset of the space. The present programming paradigm is based on an exact labour-intensive representation of the trajectory(/ies) of the algorithm in that space, possibly leaving some room for code generation tools or optimizers to choose the preferred path. • Sparse sampling in wireless sensor networks and large-scale distributed systems, enabling qualitative hypothesis tests and measurements in scenarios that are too complex, or resource-wise not feasible to implement with conventional sampling and computing methods. Lennarts perspective on sensory grounding and embodied cognition To me it is self-evident that some concepts and the words that express them must be irreducibly grounded in sensory experience. Then other words may be defined from such a set of sensory grounded words. A lynx may be roughly described using “cat” and “big” but it would then be futile to describe “cat” using other felines. “Big” may be described using “small” provided “small” has not already been described using “big”. I even think it would be advantageous to have a sensory grounding for “lynx” and “big”, sensory grounding whenever possible. Such words as “epistemology” probably cannot be sensory grounded so there is room for more traditional descriptions using other words. Embodied cognition is a concept related to sensory grounding but wider since it includes the motor output. Lawrence Barsalou has written extensively on and strongly in favor of sensory grounding. I have read some of his papers and papers he refers to. Below I give my comments to some of these papers, hoping some of them will prove useful. L. Barsalou (2010): “Grounded Cognition: Past, Present and Future” [45]. A brief overview proclaiming that grounded cognition becomes ever more prominent in many fields. The strength of the paper is the extensive literature list, backing this view. M. Wilson (2002): “Six views of embodied cognition”. A general claim: “... human cognition ... may instead have deep roots in sensorimotor processing.” One specific view is “Off-line cognition is body based”. Wilson argues at length (and convincingly) in favor of this view. M. Wilson et al. (2005): “The Case for Motor Involvement in Perceiving Conspecifics”. A paper full of arguments leading up to the proposition that seeing others act, which generates imitative covert motor activity, “... generating top-down expectations and prediction of the unfolding action.” The idea is related to the simulation theory by Hesslow, who is also referred to. K. Simmons & L. Barsalou (2010): “The similarity-in-topography principle: Reconciling theories of conceptual deficits”. A terribly wordy paper on a very important topic: How do you best explain conceptual deficits that result from brain damage? The authors claim Conceptual topography theory (CTT), including the similarity-in-topography (SIT) principle does the job. Basically the paper deals with the organization of knowledge and how it is impaired from brain lesions. ...

29

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

The following list summarizes some outstanding challenges and research objectives in machine learning, see Section 3 and [37] for further information. • What are effective shallow / feature learning algorithms? • What are effective deep (multi-scale, scale-free) learning algorithms? • How to make machines efficiently use unsupervised reward signals? • How to make learning machines and systems that autonomously adapt in real-time to a complex and changing environment? • How can efficient communication across human-machine interfaces be learned, for example in the form of apprenticeship learning? In order to address these challenges and take a significant step forward we need to change focus from individual learning algorithms to the development of real-world architectures for autonomous learning machines and systems. This will require a clear strategical agenda and organization with necessary resources, which potentially could be created in the form of an Open Research and Innovation Platform for process industries [3, 27].

6

Vision and Tentative Steps Towards 2030

In machine learning applications, access to high-quality data is often more critical for good results than the details of model design and selection, which are of secondary importance. Therefore, the Open Research and Innovation Platform for process-industry related research, development and education that is proposed earlier [3, 27] is motivated also from the perspective considered in this work. An open platform with databases and transparent access to information (ideally also actuators / decision support systems) in some selected real-world processes would be a valuable and unique resource for machine learning development, model checking and education. Ideally, the platform should provide transparent cloud computing support and data access for large-scale simulations. The development of processes, machines and systems with more intelligent qualities will require a change of focus from isolated learning algorithms and machine learning applications to closed-loop sensor-actuation systems and architectures, which in principle can include human decision making and complex environmental systems such as markets etc. A central concept presented above is the interpretation of an intelligent industrial process as an entity in a complex and changing environment, see Figure 1, interacting with numerous ICT systems in order to obtain the information needed to reach the goals defined by humans. In that perspective there is a clear analogy between the intelligence of an industrial process and an animal, which concerns the ability of the process to reach goals in the environment [7]. It is also reasonable to consider novel approaches to “intelligent components” in this context, for example control systems, condition monitoring systems, mechanical components offering model descriptions that are automatically adapted to the machines and processes in which they are deployed, and reliable cloud services and components for eHealth and smart environment applications. The development of the Internet of Things and integration of Cyber-Physical Systems in industry and society, for example driven by strategic agendas and current ICT calls, see Section 3, creates an increasing need for machine learning methods that are suited for applications of such technology. LTU has well-established research in this area, forming an important stepping stone for the development of machine learning and intelligence related activities. Although history advocates for caution in speculating about the future of this field, I think that the time period up to 2030 will be an exciting one. During the next decade substantial resources are invested world wide in brain research and neuroinformatics, with clear strategic objectives to create a new kind of information technology inspired by the 30

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

autonomous, fault-tolerant and scaling properties of brains, for example in the form of the Human Brain Project and the BRAIN initiative, see Section 3. Also, large companies like IBM and Google are developing neuromorphic computing technology and large-scale machine learning architectures for increasingly general tasks, for example demonstrated in the form of Watson who challenge humans in Jeopardy and is successfully used for decision support in healthcare. The history of biomimetic AI is short, it is only during the last decade that some tools needed to study the corresponding natural systems in a reasonably accurate way are invented. For these reasons, I think that we will see a substantial development of the field up to 2030, which is difficult to predict. LTU certainly needs to invest and support competence development in this area in order to be able to supply the region with state-of-the-art competence in ICT one decade from now. Relevant interests and competence exist in several groups at LTU, see Section 4, but a common strategic agenda and properly funded organization needs to be developed. In the following I outline a few proposals as input to the discussion.

6.1

First steps

In a short-term perspective we need to focus on problems, see Section 3 and Section 5, which can be addressed with existing competence, tools and collaborations. In particular, we need to focus on relatively small-scale systems, eventually targeting more complex architectures and systems when the necessary framework is developed. The following list summarizes some possible developments that can be targeted in a 2020 timeframe. • Component model as a service [27], first step towards virtual plants / factories. This will require competence in automatic control, combination of soft and hard models (hybrid models), probabilistic machine learning, component-based software design, and service-oriented architectures. A first demonstrator can likely be ready by 2020. Note that this concept involves fundamental challenges calling for groundbreaking research, see Section 5.1. Possible extension to learning of symbolic equations and hybrid models. • Automated condition monitoring and maintenance of machines, for example in the context of the SKF University Technology Center at LTU. This could be a first step towards autonomous machines and processes that control operational parameters and support systems (e.g. lubrication) in order to maximize lifetime and production efficiency in a more general fashion than what is possible today. This will require competence in signal processing, probabilistic machine learning (in particular feature learning and Bayesian methods), hybrid models, digital design, electronics, and appropriate test rigs and use cases with related competence. Proof of concept and a first demonstrator can likely be ready by 2020. • Robust sensors and software components for Cyber-Physical Systems, with integrated probabilistic / hybrid models for detection of operational anomalies, including investigations of adaptive and self-correcting computing architectures suitable for changing system and signal configurations. This could be a first step towards robust components and systems needed to realize the concept of intelligent industrial processes and enabling ICT for reliable eHealth, smart regions and cloud services. This will require competence in component-based software design, computer science, probabilistic machine learning, cognitive computation, and appropriate test rigs or use cases. Proof of concept and a first demonstrator can likely be ready by 2020. See also the discussion in Section 5.2. • Robust communication architecture for Cyber-Physical Systems, possibly aiming at automated, semantic-invariant and self-correcting interoperability of location-aware components in harsh industrial environments. This will require competence in communication hardware and software, service-oriented architectures, computer science, 31

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

probabilistic machine learning and cognitive computation. Given that initial steps have been taken [2,47] and that the background knowledge is available, a first demonstrator can likely be ready by 2020. • Novel modalities for human-machine interaction using wireless biometric sensors and actuators, including estimation and prediction of affective states, and aspects related to machine learning by apprenticeship [48]. Work in this direction can open up for new approaches to process control, maintenance, safety and learning in industrial environments. This will require competence in cognitive and affective computation, electronics and embedded systems design, and probabilistic machine learning. A first demonstrator limited to a particular application can likely be ready by 2020. • . . . (to be completed and concluded after a broader discussion)

6.2

Tentative Steps Towards 2030

In addition to continuing the research and development work outlined above towards general technologies, commercialization and broader use, I propose that the following trends and challenges are some of the guiding stars for the further development towards 2030. • The environment of process industries becomes increasingly competitive, calling for more efficient control, simulation and decision support systems that integrate necessary information from a variety of complex ICT systems, such as maintenance, business, raw material, and human resource systems, see Figure 1. • The complexity of software in industrial embedded systems and Cyber-Physical Systems will continue to grow rapidly (exponentially according to some), leading to grand challenges in software design, maintenance and security, calling for a new approach to software and system design. See also the discussion in Section 5.2. A related note: the present software design philosophy and culture, which dictates that software should be perfected before deployment, is fundamentally at odds with Nature’s approach to the problem. In Biology, systems are continuously “programmed” by the environment at multiple levels, for example in terms of replicator dynamics and survival of the fittest, and by cognitive learning. • Technology for efficient human-machine interaction will become increasingly important as technical systems tend to become more complex, resource constraints more challenging and safety considered more important. In addition, cultural and societal development tend to increase the likelihood that professionals change working environment, leading to challenges in knowledge management and building attractive working environments. • Global investments and research efforts in brain research and brain-inspired ICT, in combination with machine-learning development by large companies like IBM and Google, will have profound consequences for information technology. This development will take some time, but I think that by 2030 the consequences will be significant. It is about time to start educating engineers that can take part in and contribute to this development. In general, we need to change focus from isolated ICT systems and machine-learning methods to sensor-actuator functions and large-scale system behavior. That appears to be the only way towards realization of intelligent industrial processes, which will enable us to reach goals more efficiently in an increasingly competitive world. That development will require an ICT framework enabling transparent access to data, sensors, decision support systems and computing resources.

32

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

The only conclusion that I want to present at this point, before a broader discussion and collection of feedback, is that the Open Research and Innovation Platform that is outlined in a parallel report [3] is motivated also from this machine-learning perspective. I conjecture that in order to enable the development of intelligent industrial processes and machines we need to develop the education in order to provide engineers with necessary competence. Many of the machine-learning concepts and methods that are discussed in this report can naturally be used in other domains, for example for improving the life-quality and safety of elderly, for decision support to habitants in urban environments, and for monitoring of infrastructure such as energy systems, water supply networks, buildings, bridges etc. These aspects could be further developed and discussed in another context.

6.3

Competence and Partnerships

It is essential to strengthen the collaboration in this field with other universities and institutes, in particular because there are few people working in the field at LTU. There are hundreds of groups working in fields related to machine learning, artificial intelligence and cognitive computation. Some of these groups and individuals have an outstanding impact in their field and related applications, which implies that they are easy to identify using search tools. However, these groups and individuals may be difficult to approach for collaboration purposes, unless there is a specific and well-defined problem that merits collaboration. The following groups and individuals are selected because of established or implicit contacts, or publications that are closely related to publications at LTU or a specific aspect of the perspective presented here, or targeted industrial applications that closely matches or complements applications relevant at LTU, or because of other circumstances that may justify collaboration, including geographical location. Note that the electronic version of this document includes hyperlinks to homepages, which do not appear in print. • Umeå University, Sweden Cognitive Computing Intelligent Computing Complex Networks • University of Skövde, Sweden Artificial Intelligence Lab Interaction Lab • Örebro University, Sweden Centre for Applied Autonomous Sensor Systems (AASS) • Norwegian University of Science and Technology (NTNU) CRAB Lab (Complex, Robust, Adaptive, Bio-inspired solutions) • University of Stavanger, Norway Sparse Feature Learning • Helsinki Institute of Information Technology, Finland Probabilistic Adaptive Systems • Aalto University, Finland Semantic Computing Research Group (SeCo)

33

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

• Institute of Neuroinformatics, Switzerland Neuromorphic Cognitive Systems • Dalle Molle Institute for Artificial Intelligence (deep learning), Switzerland • Center of Excellence Cognitive Interaction Technology (CITEC), Germany Neuromorphic Behaving Systems (Emergentist Semantics) • University of Osnabrück, Germany Institute of Cognitive Science • Max Planck Institute for Intelligent Systems (Tübingen group), Germany • University of California at Berkeley, USA Redwood Center for Theoretical Neuroscience (sparse coding) • University of California, Irvine, USA UC Irvine Machine Learning Repository In addition, there are numerous partners associated with current projects at LTU that potentially can contribute competence, tools and use cases. In the search, analysis and summary of this list I have certainly omitted and overlooked some potentially interesting new partners.

34

IIP & Enabling ICT: Machine Learning and Intelligence

7

April 22

References

[1] F. Sandin, A. I. Khan, A. G. Dyer, A. H. M. Amin, G. Indiveri, E. Chicca, and E. Osipov, “Concept learning in neuromorphic vision systems: What can we learn from insects?,” Journal of Software Engineering and Applications 7 no. 5, (2014) 387–395. www.scirp.org/journal/PaperInformation.aspx?PaperID=45803. Open Access, Special Issue on Computer Vision. [2] B. Emruli, F. Sandin, and J. Delsing, “Vector space architecture for emergent interoperability of systems by learning from demonstration,” Biologically Inspired Cognitive Architectures 9 (2014) . Special Issue on Neural-Symbolic Networks for Cognitive Capacities. [3] W. Birk, “Intelligent Industrial Processes – Automatic Control Perspective,” tech. rep., Luleå University of Technology, 2014. E-mail: [email protected] [4] E. Osipov and V. Vyatkin, “Intelligent industrial processes – enabling research challenges by dependable communication and computation,” tech. rep., Luleå University of Technology, 2014. E-mail: [email protected] [5] I. Aguillo, “Is google scholar useful for bibliometrics? a webometric analysis,” Scientometrics 91 no. 2, (2012) 343–351. [6] J. Manyika, M. Chui, B. Brown, J. Bughin, R. Dobbs, C. Roxburgh, and A. H. Byers, “Big data: The next frontier for innovation, competition, and productivity,” May, 2011. http://www.mckinsey.com/Insights/MGI/Research/Technology_ and_Innovation/Big_data_The_next_frontier_for_innovation. [7] S. Legg and M. Hutter, “Universal intelligence: A definition of machine intelligence,” Minds and Machines 17 no. 4, (2007) 391–444. http://dx.doi.org/10.1007/s11023-007-9079-x. [8] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., 2006. [9] K. P. Murphy, Machine Learning: A Probabilistic Perspective (Adaptive Computation and Machine Learning series). The MIT Press, 2012. [10] A. Gelman and C. R. Shalizi, “Philosophy and the practice of bayesian statistics,” British Journal of Mathematical and Statistical Psychology 66 no. 1, (2013) 8–38. http://dx.doi.org/10.1111/j.2044-8317.2011.02037.x. [11] K. Friston, “A free energy principle for biological systems,” Entropy 14 no. 11, (2012) 2100–2121. http://www.mdpi.com/1099-4300/14/11/2100. [12] B. Olshausen and D. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature 381 (1996) 607–609. [13] A. Coates and A. Ng, “The importance of encoding versus training with sparse coding and vector quantization,” in Proceedings of the 28th International Conference on Machine Learning (ICML-11), L. Getoor and T. Scheffer, eds., ICML ’11, pp. 921–928. ACM, New York, NY, USA, June, 2011. [14] G. E. P. Box, “Sampling and Bayes’ Inference in Scientific Modelling and Robustness,” Journal of the Royal Statistical Society A 143 no. 4, (1980) 383–430. [15] E. T. Jaynes, Probability Theory: The Logic of Science. Cambridge University Press, 2003.

35

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

[16] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Prentice Hall Press, 3rd ed., 2009. [17] J. Hawkins and S. Blakeslee, On Intelligence. Times Books, 2004. [18] H. Koppula and A. Saxena, “Physically grounded spatio-temporal object affordances,” in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds., vol. 8691 of Lecture Notes in Computer Science, pp. 831–847. Springer International Publishing, 2014. [19] A. Trewavas, “Plant intelligence,” Naturwissenschaften 92 no. 9, (2005) 401–413. http://dx.doi.org/10.1007/s00114-005-0014-9. [20] K. Friston, R. Adams, L. Perrinet, and M. Breakspear, “Perceptions as hypotheses: saccades as experiments,” Frontiers in Psychology 3 no. 151, (2012) . [21] G. Hesslow, “The current status of the simulation theory of cognition,” Brain Research 1428 no. 0, (2012) 71–79. The Cognitive Neuroscience of Thought. [22] “Information & communication technologies in horizon 2020.” Http://ec.europa.eu/digital-agenda/en/information-communication-technologieshorizon-2020. Accessed: 2014-04-17. [23] “Strategic research agenda of the european technology platform on smart systems integration (eposs),” tech. rep., 2009. www.smart-systems-integration.org. [24] “Brain research through advancing innovative neurotechnologies (brain).” Www.nih.gov/science/brain/. Accessed: 2014-04-17. [25] “Human brain project.” Www.humanbrainproject.eu. Accessed: 2014-04-17. [26] “A Roadmap for U.S. Robotics: From Internet to Robotics,” tech. rep., 2013. www.robotics-vo.us. [27] P. Lingman, J. Gustafsson, A. O. E. Johansson, O. Ventä, M. Vilkko, S. Saari, J. Tornberg, and A. Siimes, “European roadmap for industrial process automation,” tech. rep., ProcessIT.EU, ARTEMIS Center of Innovation Excellence, 2013. [28] “Arrowhead – artemis innovation pilot project.” Www.arrowhead.eu. Accessed: 2014-04-17. [29] B. Emruli and F. Sandin, “Analogical mapping with sparse distributed memory: A simple model that learns to generalize from examples,” Cognitive Computation 6 no. 1, (2014) 74–88. [30] R. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: analysis of affective physiological state,” Pattern Analysis and Machine Intelligence, IEEE Transactions on 23 no. 10, (2001) 1175–1191. [31] T. S. Cubitt, J. Eisert, and M. M. Wolf, “Extracting dynamical equations from experimental data is np hard,” Phys. Rev. Lett. 108 (Mar, 2012) 120503. http://link.aps.org/doi/10.1103/PhysRevLett.108.120503. [32] L. Ljung, “Perspectives on system identification,” Annual Reviews in Control 34 no. 1, (2010) 1 – 12.

36

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

[33] B. B. Machta, R. Chachra, M. K. Transtrum, and J. P. Sethna, “Parameter space compression underlies emergent theories and predictive models,” Science 342 no. 6158, (2013) 604–607, http://www.sciencemag.org/content/342/6158/604.full.pdf. http://www.sciencemag.org/content/342/6158/604.abstract. [34] D. L. Ly and H. Lipson, “Learning symbolic representations of hybrid dynamical systems,” Journal of Machine Learning Research 13 (2012) 3585–3618. [35] Y. Bengio, “Learning deep architectures for ai,” Found. Trends Mach. Learn. 2 no. 1, (2009) 1–127. [36] S. Arora, “Provable bounds for learning some deep representations,” in Proceedings of The 31st International Conference on Machine Learning, E. P. Xing and T. Jebara, eds., vol. 32 of JMLR Workshop and Conference Proceedings, pp. 584–592. arXiv:1310.6343v1. [37] “Future challenges for the science and engineering of learning, final workshop report,” tech. rep., National Science Foundation, 2007. Rodney Douglas and Terry Sejnowski (eds.). [38] A. Bruckstein, D. Donoho, and M. Elad, “From sparse solutions of systems of equations to sparse modeling of signals and images,” SIAM Review 51 no. 1, (2009) 34–81. [39] M. Elad, “Sparse and redundant representation modeling – what next?,” Signal Processing Letters, IEEE 19 no. 12, (Dec, 2012) 922–928. [40] E. J. Candès, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Communications on Pure and Applied Mathematics 59 no. 8, (2006) 1207–1223. [41] “Compressive imaging: A new single-pixel camera.” Http://dsp.rice.edu/cscamera. Accessed: 2014-04-20. [42] M. Stéphane, A Wavelet Tour of Signal Processing (Third Edition: The Sparse Way). Academic Press, Boston, third edition ed., 2009. [43] S. del Campo, K. Albertsson, J. Nilsson, J. Eliasson, and F. Sandin, “Fpga prototype of machine learning analog-to-feature converter for event-based succinct representation of signals,” in Machine Learning for Signal Processing (MLSP), 2013 IEEE International Workshop on, pp. 1–6. Sept, 2013. http://pure.ltu.se/portal/files/43648751/mlsp2013.pdf. [44] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, “Zero-shot learning through cross-modal transfer,” in Advances in Neural Information Processing Systems 26, C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, eds., pp. 935–943. 2013. http://papers.nips.cc/paper/ 5027-zero-shot-learning-through-cross-modal-transfer.pdf. [45] L. W. Barsalou, “Grounded cognition: Past, present, and future,” Topics in Cognitive Science 2 no. 4, (2010) 716–724. [46] L. W. Barsalou, “Grounded cognition,” Annual Review of Psychology 59 (2008) 617–645. [47] D. Kleyko, N. Lyamin, E. Osipov, and L. Riliskis, “Dependable mac layer architecture based on holographic data representation using hyper-dimensional binary spatter codes,” in Multiple Access Communications, B. Bellalta, A. Vinel, 37

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

M. Jonsson, J. Barcelo, R. Maslennikov, P. Chatzimisios, and D. Malone, eds., vol. 7642 of Lecture Notes in Computer Science, pp. 134–145. Springer Berlin Heidelberg, 2012. [48] P. Abbeel and A. Y. Ng, “Exploration and apprenticeship learning in reinforcement learning,” in Proceedings of the 22Nd International Conference on Machine Learning, ICML ’05, pp. 1–8. ACM, New York, NY, USA, 2005. [49] D. Wang, C. Song, and A.-L. Barabasi, “Quantifying long-term scientific impact,” Science 342 no. 6154, (2013) 127–132.

38

IIP & Enabling ICT: Machine Learning and Intelligence

A

April 22

List of High-Impact Journals

Table 1: Selection of journals among the top-500 with the highest Scimago Journal Rank (SJR) on January 7, 2014; www.scimagojr.com. The SJR is based on weighted references, see the homepage for technical details, which is not a formal representation of “scientific impact”. Some progress in that direction has recently been made [49] but is not available in a public tool as far as I know. A subset of these journals may be relevant for publication, others are included here as information sources, for example neuroscience-related journals. Nr

Title

SJR

Cites/Doc. (2 years)

9 18 37 43 51 55 57 62 65 73 89 108 130 131 139 140 153 158 159 181 183 196 199 214 218 219 228 238 252 253 263 302 308 325 332 343 345 348 355 361 363 368 375 383 384 394 405 406 408 413 421 425 442 446 447 448 455 467 474 475 478 488

Annual Review of Neuroscience Nature Science Physics Reports Neuron Nature Neuroscience Trends in Cognitive Sciences Nature Reviews Neuroscience Trends in Neurosciences IEEE Transactions on Pattern Analysis and Machine Intelligence Annals of Statistics ACM Computing Surveys Physical Review X International Journal of Computer Vision SIAM Review Journal of the ACM ACM Transactions on Intelligent Systems and Technology Proceedings of the National Academy of Sciences of the United States IEEE Transactions on Automatic Control Psychological Review IEEE Journal of Solid-State Circuits Proceedings of the Annual ACM Symposium on Theory of Computing Automatica IEEE Transactions on Evolutionary Computation Physical Review Letters Journal of Neuroscience Semantic Web and Information Systems International Journal of Robotics Research IEEE Transactions on Information Theory IEEE Transactions on Fuzzy Systems Cerebral Cortex Brain Research Reviews IEEE Transactions on Signal Processing Cognitive Psychology Organizational Behavior and Human Decision Processes Information Systems Research Psychological Science IEEE Transactions on Industrial Electronics Foundations of Computational Mathematics Proceedings of the IEEE IEEE Transactions on Software Engineering Swarm and Evolutionary Computation IEEE Symposium on VLSI Circuits, Digest of Technical Papers IEEE Transactions on Robotics IEEE Transactions on Mobile Computing Web Semantics Geometry and Topology Mathematical Programming Argument and Computation Journal of Field Robotics Computers and Operations Research SIAM Journal on Computing Journal of Statistical Software IEEE Transactions on Smart Grid ACM Transactions on Knowledge Discovery from Data Artificial Intelligence Human Brain Mapping IEEE Transactions on Services Computing IEEE Journal on Selected Topics in Signal Processing Information Sciences Journal of Cognitive Neuroscience SIAM Journal on Optimization

17.24 14.75 10.62 10.06 9.39 8.88 8.78 8.65 8.44 8.09 7.34 6.75 6.18 6.17 5.97 5.95 5.58 5.47 5.46 5.03 5.02 4.84 4.83 4.6 4.54 4.53 4.43 4.3 4.13 4.13 4.03 3.74 3.72 3.64 3.57 3.53 3.52 3.48 3.43 3.41 3.39 3.36 3.33 3.3 3.3 3.26 3.18 3.18 3.17 3.16 3.14 3.12 3.05 3.04 3.03 3.03 3 2.99 2.97 2.96 2.92 2.88

22.7 24.49 18.29 22.46 13.37 12.91 17.94 18.34 15.38 9.43 3.54 8.72 6.42 6.12 7.27 4.91 15.38 10.05 4.77 8.97 5 2.84 5.06 7.99 6.93 7.32 0.83 4.39 3.82 6.99 6.69 8.42 4.49 4.74 2.66 3.12 5.01 8.2 2.2 9.21 5.83 9.44 1.69 4.51 4.36 3.09 0.86 2.5 4 3.39 3.08 1.88 5.22 13.49 3.88 3.59 6.11 5.88 5.85 4.8 5.03 2.74

39

IIP & Enabling ICT: Machine Learning and Intelligence

B

April 22

Keyword Charts

Learning systems Machine learning Learning algorithms Algorithms Machine-learning Artificial intelligence Data mining Neural networks Support vector machines Classification of information Human Humans Feature extraction Data sets Algorithm Classification Machine learning techniques Mathematical models Pattern recognition Optimization Robot learning Decision trees Forecasting Problem solving Controlled study Computer simulation Information retrieval Prediction Methodology Regression analysis Semantics Database systems Machine learning methods Artificial neural network Bioinformatics Education 103

104 Number of papers in all fields

Figure 9: Keywords and paper counts related to machine learning. These papers are retrieved with the query TITLE-ABS-KEY(“machine learning”) at Scopus, selecting papers published after year 2000. The paper counts for each keyword are extracted from the “Keywords” menu in Scopus. The query term is highlighted in boldface. Note that the paper counts for the keyword “machine learning” are divided on two rows, making “learning systems” to appear at the top of the list.

40

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Artificial intelligence Algorithms Neural networks Computer simulation Humans Decision support systems Learning systems Mathematical models Methodology Optimization Pattern Recognition Automated Decision making Problem solving Knowledge based systems Automated pattern recognition Computer science Reproducibility of Results Sensitivity and Specificity Computer assisted diagnosis Semantics Decision theory Genetic algorithms Image processing Computer software Fuzzy sets Image Interpretation Computer-Assisted Computer Simulation Information retrieval Learning algorithms Feature extraction Expert systems Sensitivity and specificity Pattern recognition Computer vision 104 105 Number of papers in all fields Figure 10: Keywords and paper counts related to artificial intelligence. These papers are retrieved with the query TITLE-ABS-KEY(“artificial intelligence”) at Scopus, selecting papers published after year 2000. The paper counts for each keyword are extracted from the “Keywords” menu in Scopus. The query term is highlighted in boldface.

41

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Machine intelligence Artificial intelligence Learning systems Neural networks Algorithms Informatics Robotics Fuzzy sets Computer simulation Computer vision Intelligent systems Mathematical models Decision making Pattern recognition Fuzzy logic Sensors Automation Human computer interaction Intelligent robots Knowledge based systems Semantics Computer software Human Robots Optimization Problem solving Learning algorithms Image analysis Algorithm Intelligent control Feature extraction Genetic algorithms Expert systems 0

50

100 150 200 250 300 350 Number of papers in all fields

400

Figure 11: Keywords and paper counts related to machine intelligence. These papers are retrieved with the query TITLE-ABS-KEY(“machine intelligence”) at Scopus, selecting papers published after year 2000. The paper counts for each keyword are extracted from the “Keywords” menu in Scopus. The query term is highlighted in boldface.

42

IIP & Enabling ICT: Machine Learning and Intelligence

April 22

Learning systems Neural networks Algorithms Learning algorithms Mathematical models Artificial intelligence Computer simulation Students Problem solving Machine learning Education Optimization Teaching Classification of information E-learning Pattern recognition Feature extraction Classifiers Engineering education Data mining Fuzzy sets Database systems Vectors Curricula Knowledge based systems Knowledge acquisition Genetic algorithms Backpropagation World Wide Web Internet Learning Support vector machines Computational complexity Probability Data reduction Information technology Data sets Computer software 104 Number of papers in all fields

105

Figure 12: Keywords and paper counts related to learning systems. These papers are retrieved with the query TITLE-ABS-KEY(“learning systems”) at Scopus, selecting papers published after year 2000. The paper counts for each keyword are extracted from the “Keywords” menu in Scopus. The query term is highlighted in boldface.

43

Index Affective computation, 24 Affordance, 13 Automation, 5 Bayesian, 10 Bayesian information criterion (BIC), 12 Big data, 6, 8, 17 BRAIN, 18 CITEC, 23 Cognitive architectures, 19 Compressed sensing, 28 Cyber-physical systems (CPS), 16 Enabling ICT, 8 EPoSS, 17 Research themes, 18 Feature learning, 10, 11 Frequentist statistics Limitations of, 12 Future computing, 20 Future mining, 5 Games, 13 Generalization, 11 Grounding, 29 Horizon 2020 ICT, 6, 16 Human brain project (HBP), 19 ICT, 5, 6 IIP, 5, 7 Information representation, 10 Information theory, 10 INI, 23 Intelligence, 13 Definition, 16 Insects and plants, 14 Motivation, 14 Relation to machine learning, 14 Intelligent industrial process, 6 Internet of things (IoT), 6

Uses of, 11 MMSR, 27 Model checking, 12 Model selection, 12 NeuroIT, 22 Neuromorphic computing systems (NCS), 19, 20 Neurorobotics, 19 Open platform, 6 Parameter space compression, 27 Partnerships, 33 ProcessIT, 6 Roadmap, 22 Reward signals, 14 Robotics, 21 Scopus, 5 Sensory-motor function, 14 Simulation hypothesis, 15 Smart machines, 5 Smart objects, 17 Smart system, 16, 17 Technology generations, 18 Sparse coding, 28 Research directions, 28 Sparse representation, 28 Statistical physics, 10 Succinct representation, 10 SWOT, 25 Symbols, 14 System identification, 27 Universal intelligence, 16

LEIT, 16 Limitations, 5 LTU, 5 Machine learning, 6 Books, 7 Challenges and open problems, 30 Competence at LTU, 25 Education, 13 Relevance, 8 44