Integrated information in discrete dynamical systems: motivation and theoretical framework

Abstract

This paper introduces a time- and state-dependent measure of integrated information, phi, which captures the repertoire of causal states available to a system as a whole. Specifically, phi quantifies how much information is generated (uncertainty is reduced) when a system enters a particular state through causal interactions among its elements, above and beyond the information generated independently by its parts. Such mathematical characterization is motivated by the observation that integrated information captures two key phenomenological properties of consciousness: (i) there is a large repertoire of conscious experiences so that, when one particular experience occurs, it generates a large amount of information by ruling out all the others; and (ii) this information is integrated, in that each experience appears as a whole that cannot be decomposed into independent parts. This paper extends previous work on stationary systems and applies integrated information to discrete networks as a function of their dynamics and causal architecture. An analysis of basic examples indicates the following: (i) phi varies depending on the state entered by a network, being higher if active and inactive elements are balanced and lower if the network is inactive or hyperactive. (ii) phi varies for systems with identical or similar surface dynamics depending on the underlying causal architecture, being low for systems that merely copy or replay activity states. (iii) phi varies as a function of network architecture. High phi values can be obtained by architectures that conjoin functional specialization with functional integration. Strictly modular and homogeneous systems cannot generate high phi because the former lack integration, whereas the latter lack information. Feedforward and lattice architectures are capable of generating high phi but are inefficient. (iv) In Hopfield networks, phi is low for attractor states and neutral states, but increases if the networks are optimized to achieve tension between local and global interactions. These basic examples appear to match well against neurobiological evidence concerning the neural substrates of consciousness. More generally, phi appears to be a useful metric to characterize the capacity of any physical system to integrate information.

Figure 1. Effective information generated by entering a particular state.

A system of three connected AND-gates transitions from state x ₀ = 110 at time zero to x ₁ = 001 at time one. The a priori repertoire is the uniform distribution on the 8 possible outputs of the elements of the system. The causal architecture of the system specifies that state 110 is the unique cause of x ₁, so the a posteriori repertoire (shown in cyan) assigns probability 1 to state 110 and 0 to all other states. Effective information generated by the system transitioning to x ₁ is 3 bits.

Figure 2. Effective information: a few examples.

Each panel depicts a different system, which has entered a particular state. The a priori and a posteriori repertoires are shown and effective information is measured. (A) is a simple system of two elements that copy each other's previous outputs (a couple). Effective information is 2 bits, less than for the system in Figure 1 since the repertoire of outputs is smaller. (B) shows the AND-gate system of Figure 1 entering the state 000. This state is less informative than 001 since the a posteriori repertoire specified by the system includes four perturbations; effective information is reduced to 1 bit. The systems in (C) and (D) generate no effective information. In (C) the elements always fire regardless of their inputs, corresponding to an inescapable fixed point. In (D) the elements fire or are silent at random, so that the prior state is irrelevant. In both cases the a posteriori repertoire is the maximum entropy distribution since no alternatives have been ruled out, so effective information is zero.

Figure 3. Integrated information for a system of two disjoint couples.

The panels analyze the same system of two disjoint couples from three different perspectives. The interactions in the system are displayed in cyan. Those interactions that occur within a part are shown in red, and those between parts are in dark blue. (A) computes effective information for the entire system X, finding it to be 4 bits. (B) computes effective information generated by each of the couples independently and then computes integrated information φ(x ₁), finding it to be 0 bits since the two couples do not interact. Notice that the combined a posteriori repertoire of the parts coincides with the a posteriori repertoire of the system; the parts account for all the interactions within X. (C) considers a partition of the system other than the minimum information partition. Since is not isolated it cannot account for the effect of interactions with internally; they are treated as extrinsic noise and result in specifying a maximum entropy a posteriori repertoire. Effective information generated across the partition is 4 bits.

Figure 4. Effective information generated across the minimum information partition.

(A) depicts the interactions within the system that are quantified by effective information of the entire system. (B) disentangles the interactions, showing interactions within parts in red, and interactions between parts in dark blue. (C) is a schematic of the relationship between the repertoires specified by the system and the parts. Effective information, represented by the arrows, is the entropy of a lower repertoire relative to an upper one. φ(x ₁) is the entropy of the a posteriori repertoire of the system relative to the combined a posteriori repertoire of the minimal parts.

Figure 5. Decomposing systems into overlapping complexes.

In this example elements are parity gates: they fire if they receive an odd number of spikes. Links without arrows are bidirectional. The system is decomposed into three of its complexes, shown in shades of gray. Observe that: i) complexes can overlap; ii) a complex can interact causally with elements not part of it; iii) groups of elements with identical architectures generate different amounts of integrated information, depending on their ports-in and ports-out (compare subset A, the dark gray filled-in circle, with subset B, the right-hand circle).

Figure 6. Elements driven by a complex do not contribute to integrated information.

The system is constructed using the AND-gate system of Figure 1, with the addition of three elements copying the inner triple. The AND-triple forms a main complex, as do the couples. However, the entire system generates no integrated information and does not form a complex, since X generates no information over and above that generated by subset A.

Figure 7. Analyzing systems in terms of elementary components.

(A) and (C) show systems that on the surface appear to generate a large amount of integrated information. The units in (A) have a repertoire of 2ⁿ outputs, with the bottom unit copying the top. Integrated information is n bits. Analyzing the internal structure of the system in (B) we find n disjoint couples, each integrating 1 bit of information; the entire system however is not integrated. (C) shows a system of binary units. The top unit receives inputs from 8 other units and performs an AND-gate like operation, firing if and only if all 8 inputs are spikes. Increasing the number of inputs appears to easily increase φ without limit. (D) examines a possible implementation of the internal architecture of the top unit using binary AND-gates. The architecture has a bottleneck, shown in red, so that φ = 1 bit no matter the number of input units.

Figure 8. Integrated information and extrinsic inputs.

The gray box represents a main complex. Red arrows are input from the environment. Black arrows depict strong feedforward connections; gray arrows are weaker modulatory connections. The black zig-zag represents the bulk of the main complex. The current state of row R^a is determined by extrinsic inputs, which are treated as extrinsic noise. However the current state of row R^b together with the feedforward architecture of the system together specify the prior state of R^a, so that the system is able to distinguish extrinsic inputs once they have caused an interaction between elements within the main complex. Similarly row R^c specifies higher-order invariants in the prior state of row R^b.

Figure 9. Integrated information peaks in balanced states.

(A–D) show four discrete systems; lines represent bi-directional connections. Elements fire if they receive two or more spikes. The graph shows integrated information as a function of the number of elements firing. Integrated information is computed by averaging over all states with a particular number of elements firing. Integrated information is low for hyperactive and inactive states when too many or too few elements are firing, and high for balanced states lying between the two extremes. Note that in (A) the value of φ for 7 elements firing is undefined, since no state with seven elements firing is possible given the causal architecture.

Figure 10. Bistable dynamics.

The system has connectivity as in Figure 9C, with altered element behavior. If an element receives less than two spikes it fires with probability .15. If it receives 2 or more spikes it fires with certainty, unless more than half the elements fired in the two times step prior, in which case all elements are silent. The graph plots φ and the percentage of elements firing, as the system is run for 120 time steps. The system implements a bistable dynamics, and is unable to sustain high values of φ.

Figure 11. System cycling (via binary counting) through 16 firing patterns.

The system cycles through the firing patterns 0000, 0001, 0010, …, 1101, 1110, 1111; counting in binary from 0 to 15 and repeating. (A) and (B) show the system, implemented with interacting memoryless elements, in two different states. (C) shows a system with identical dynamics, implemented using four elements independently replaying a list of instructions. Since there are no causal interactions, the replay generates no integrated information, in contrast to the memoryless system.

Figure 12. Optimized network of AND-gates.

The network is optimized to generate high integrated information in a single state, that shown. Each element implements an AND-gate.

Figure 13. Integrated information in a strongly modular network.

The system is composed of three four-element modules. The elements fire if they receive two or more spikes. The entire system forms a complex (light gray) with φ(x ₁) = .7 bits; however, the architecture is strongly modular so that the main complexes (dark gray) are the modules, each generating φ = 1.2 bits of integrated information across the total partition, more than the system as a whole.

Figure 14. Integrated information in homogeneous systems.

The systems have all-to-all connectivity, including self-connections. (A) shows a parity system: each element fires if it receives an odd number of spikes and is silent otherwise. The MIP is the total partition and integrated information is 1 bit. (B) shows a majority-rule system where elements fire if they receive three or more spikes. The MIP is the bipartition shown. The a posteriori repertoire specified by each part contains three perturbations, with weights .09, .09, and .8 respectively. The combined a posteriori repertoire contains 9 perturbations of varying weights, as shown. The a posteriori repertoire of the system contains 5 equally weighted perturbations. Integrated information is .79 bits.

Figure 15. Integrated information for lattice architectures.

(A) is an n×n XOR-lattice. The minimum information partition is given by a vertical or horizontal midpartition. Integrated information is n bits; and so can be increased without limit by scaling up the (highly inefficient) architecture. (B) and (C) show integrated information for a Game of Life grid in two different configurations. Cells in the grid are either ON or OFF. Each cell has 8 neighbors, the grid is assumed to wrap around to form a torus. A cell that is OFF switches to ON in the next time step if exactly 3 of its neighbors are ON. An ON cell remains ON if two or three neighbors are ON; otherwise it switches to OFF. (D) shows long-range connections short-circuiting the perimeter bottleneck intrinsic to lattice architectures.

Figure 16. Integrated information in feedforward networks.

(A) shows a tree-like hierarchical feedforward network. Effective information from the sensory sheet below the grid (not shown) is high, 1.6 bits, but the ability of the network to integrate information is limited by the bottleneck at the grandmother cell n ₁. (B) and (C) show the network with an additional grandmother cell (the network is flattened out, so the second grandmother appears at the bottom). A redundant grandmother results in zero integrated information, whereas if the grandmother is not redundant φ increase to .5 bits. (D) and (E) depict a grid-like feedforward architecture that does not suffer from the bottleneck of (A). Integrated information increases with network size.

Figure 17. Integrated information in a Hopfield network.

(A-D) show a sample run of a Hopfield network with 8 elements and all-to-all connectivity. The network has embedded attractors 11110000, 11001100, 10101010 and their mirror images. A sample run is depicted at T = .45 and initial state 11111111. (E-G) show integrated information as a function of temperature (computed using bipartitions) for the corresponding states; the colored enclosures are matched with the graphs. The system forms a complex for low temperatures (blue), but breaks down for higher temperatures (red), so that subcomplexes arise.

Figure 18. A functionally integrated and functionally specialized probabilistic network can sustain high φ.

(A) shows a functionally integrated and functionally specialized network; black arrows represent connections of weight ≥.5 and red arrows connections with weight ≤−.5. Weaker connections are not shown to reduce clutter; see Text S1, section 13. The elements operate according to the rules of a Hopfield network with T = .35. The network is initialized with a random firing patter and allowed to run for 800 time steps. (B) shows φ for each firing pattern that occurs during the run.