A review of some techniques for inclusion of domain-knowledge into deep neural networks

Abstract

We present a survey of ways in which existing scientific knowledge are included when constructing models with neural networks. The inclusion of domain-knowledge is of special interest not just to constructing scientific assistants, but also, many other areas that involve understanding data using human-machine collaboration. In many such instances, machine-based model construction may benefit significantly from being provided with human-knowledge of the domain encoded in a sufficiently precise form. This paper examines the inclusion of domain-knowledge by means of changes to: the input, the loss-function, and the architecture of deep networks. The categorisation is for ease of exposition: in practice we expect a combination of such changes will be employed. In each category, we describe techniques that have been shown to yield significant changes in the performance of deep neural networks.

Figure 1

An example of AI for Science. The human-in-the-loop is a biologist. The biologist conducts experiments in a biological system, obtains experimental observations. The biologist then extracts data that can be used to construct machine learning model(s). Additionally, the machine learning system has access to domain knowledge that can be obtained from the biologist. The machine learning system then conveys its explanations to the biologist.

Figure 2

The plots from showing gains in predictive accuracy of (a) multilayer perceptron (MLP), and (b) graph neural network (GNN) with the inclusion of domain-knowledge. The domain knowledge inclusion method in (a) is a simple technique known as ‘propositionalisation’ ; and, the method in (b) is a general technique of incorporating domain-knowledge using bottom-graph construction. The results shown are over 70 datasets. No importance to be given to the line joining two points; this is done for visualisation purpose only.

Figure 3

Informal descriptions of (a) logical; and (b) numerical constraints.

Figure 4

Construction of a deep model M from data (D) using a learner ($\mathcal{L}$). We use $\pi$ to denote the structure (organisation of various layers, their interconnections etc.) and $\mathit{\theta }$ to denote the parameters (synaptic weights) of the deep network. L denotes the loss function (for example, cross-entropy loss in case of classification).

Figure 5

Some implications of using domain-knowledge for commonly-used deep network architectures. Although attention-mechanism has also been used recently in many deep network architectures, we mention it only for RNNs and transformers as it is more prominently being used for sequence learning.

Figure 6

Introducing background knowledge into deep network by transforming data. $\mathcal{T}$ is a transformation block that takes input data D, background knowledge (BK) and outputs transformed data $D^{\prime }$ that is then used to construct a deep model using a learner $\mathcal{L}$.

Figure 7

Introducing background knowledge into deep network by transforming the loss function L. $\mathcal{T}$ block takes an input loss L and outputs a new loss function $L^{\prime }$ by transforming (augmenting or modifying) L based on background knowledge (BK). The learner $\mathcal{L}$ then constructs a deep model using the original data D and the new loss function $L^{\prime }$.

Figure 8

Introducing background knowledge into deep network by transforming the model (structure and parameter). In (a), the transformation block $\mathcal{T}$ takes a input structure of a model $\pi$ and outputs a transformed structure ${\pi }^{\prime }$ based on background knowledge (BK). In (b), the transformation block $\mathcal{T}$ takes a set of parameters $\mathit{\theta }$ of a model and outputs a transformed set of parameters ${\pi }^{\prime }$ based on background knowledge (BK).