Qualia: the geometry of integrated information

Abstract

According to the integrated information theory, the quantity of consciousness is the amount of integrated information generated by a complex of elements, and the quality of experience is specified by the informational relationships it generates. This paper outlines a framework for characterizing the informational relationships generated by such systems. Qualia space (Q) is a space having an axis for each possible state (activity pattern) of a complex. Within Q, each submechanism specifies a point corresponding to a repertoire of system states. Arrows between repertoires in Q define informational relationships. Together, these arrows specify a quale -- a shape that completely and univocally characterizes the quality of a conscious experience. Phi -- the height of this shape -- is the quantity of consciousness associated with the experience. Entanglement measures how irreducible informational relationships are to their component relationships, specifying concepts and modes. Several corollaries follow from these premises. The quale is determined by both the mechanism and state of the system. Thus, two different systems having identical activity patterns may generate different qualia. Conversely, the same quale may be generated by two systems that differ in both activity and connectivity. Both active and inactive elements specify a quale, but elements that are inactivated do not. Also, the activation of an element affects experience by changing the shape of the quale. The subdivision of experience into modalities and submodalities corresponds to subshapes in Q. In principle, different aspects of experience may be classified as different shapes in Q, and the similarity between experiences reduces to similarities between shapes. Finally, specific qualities, such as the "redness" of red, while generated by a local mechanism, cannot be reduced to it, but require considering the entire quale. Ultimately, the present framework may offer a principled way for translating qualitative properties of experience into mathematics.

Figure 1. Effective information.

(A): A “photodiode” consisting of a sensor and detector unit; the detector unit fires. For the entire system of two units there are four possible states: 00, 10, 01 and 11. The potential repertoire X₀(maxH) is the maximum entropy distribution on the four states. If the detector fires, its mechanism specifies that the sensor fired at time zero, thus ruling out 2 of the 4 possible states of the system, the actual repertoire is X₀(mech,x₁) = (0,0,½,½) on the four states. The prior state of the detector makes no difference to the current state of the system, so the states 01 and 11 are indistinguishable to the mechanism. Relative entropy (also known as Kullback-Leibler divergence) between two probability distributions p and q is H[p∥q] = Σ p _i log₂(p _i/q_i), so that effective information (entropy of the actual repertoire relative to the potential) is 1 bit. Integrated information. Left-hand side: two double-couples. (B): the system as a whole generates 4 bits of effective information by specifying that elements n² and n³ were on at time t = 0. (CD): The information generated by the system as a whole is completely accounted for by the parts, taken independently. The minimum information partition (MIP) is the decomposition of the system into those (minimal) parts that leave the least information unaccounted for. (E): the actual repertoire of the whole is identical to the combined actual repertoires of the parts (the product of their respective probability distributions), so that relative entropy is zero. The system generates no information above and beyond the parts, so it cannot be considered a single entity. Right-hand side: an integrated system. Elements in the system are ON if they receive 2 or more spikes. The system enters state x₁ = 1000. (B′): the mechanism specifies a unique prior state that causes (leads to) state x₁, so the system generates 4 bits of effective information. All other perturbations are ruled out since they cause different outputs. (C′D′): effective information generated by the two minimal parts, considered as systems in their own right. External inputs (dotted black arrows) are treated as extrinsic noise. (E′): the information generated by the whole (cyan arrows) over and above the parts (purple arrows). This is computed as the entropy of the actual repertoire of the whole relative to the combined actual repertoires of the parts: Φ(x₁) = 2 bits. The system generates information above and beyond its parts, so it can be considered a single entity (a complex).

Figure 2. The lattice L of combinations of submechanisms within a system of 4 elements.

(A): A system X of 4 elements and 9 connections. (B): Connections in the system are grouped into 4 submechanisms {m¹, m², m³, m⁴} contained within Conn, the set of all connections in X. (C): The lattice of combinations of the 4 submechanisms. (D): The union of submechanisms m¹ and m² is submechanism m¹². (E): The intersection of submechanisms m¹²³ and m¹²⁴ is submechanism m¹². (F): The complement of submechanism m¹² is ¬m¹² = m³⁴. (G): A q-edge is a path from the bottom of the lattice to the top, constructed by engaging each submechanism in sequence. (H): The q-fold generated by all q-arrows of the form r→r∪m¹, for different contexts r. (I): The down-set ↓{m¹, m²} and the dual up-set ↑¬{m¹, m²}.

Figure 3. A quale.

(A): Qualia space for a system of 4 elements is 16-dimensional (with an axis for each of the 2⁴ possible states of the complex); the axes are flattened onto the page. Upon entering state x₁ = 1000, the complex generates a quale or shape in Q-space. The quale is generated as follows. The maximum entropy potential repertoire (the “bottom” of the quale) is a point assigning equal probability to all states. Engaging a submechanism (in this case the pair of connections r = {c¹²,c²¹}) “sharpens” the maximum entropy distribution into an actual repertoire, which is another point in Q-space. The q-arrow linking the two distributions (without and with r engaged) geometrically realizes the informational relationship specified by the connections in r. The “length” (divergence) of the q-arrow expresses how much the connection sharpens the distribution (the effective information it generates or relative entropy between the two distributions); the direction in Q-space expresses the particular way in which the connection sharpens the distribution. (B): Adding additional connections further sharpens the actual repertoire, specifying new points in Q-space and the corresponding q-arrows. The figure shows 16 out of the 399 points in the quale; those generated by combinations of the 4 submechanisms progressively engaged in the insets. The insets around the quale show the repertoires generated along two q-edges (starting at the bottom left, going clockwise and anti-clockwise respectively) formed by q-arrows that engage the 4 sets of connections in two different orders (pink arrows are connections that are engaged; black arrows are connections that have already been engaged in the q-edge). Cyan bars represent probabilities assigned to the 16 possible prior states. Together, the q-edges enclose a shape, the quale, which completely specifies the quality of the experience. Effective information (in bits) of q-arrows in the q-edge is shown alongside.

Figure 4. Entanglement.

(A): A silent AND-gate. (B): The quale generated by the system (maroon arrows). Notice that connections c¹³ and c²³ generate more information in the full context (.33 bits at the top of the quale) than in the null context (.08 at the bottom). The actual repertoires generated by submechanisms of the system are shown alongside in cyan. Repertoire X₀({c¹³},x₁) assigns probability 2/3 to states where n¹ was silent and 1/3 to states where it was not: the concept “n¹ probably did not fire”. The actual repertoire of the whole, X₀({c¹³,c²³},x₁), specifies “n¹ and n² did not both fire”, which cannot be recovered from the concepts generated by the two connections taken singly. Entanglement is computed by measuring the entropy of the actual repertoire of the whole relative to the product of the repertoires generated by the two connections singly, shown in gray. (C): A system of three elements, two of which implement the operation NOISY COPY: element n¹ spikes with p = 0 if it receives silent input, and p = ½ if it receives a spike; this is the same operation performed by an AND-gate when one of its wires is treated as noise. (D): By construction, the informational relationships generated by connections c⁴⁵ and c⁵⁴ in the null context is the same as connections c¹³ and c²³ in panel B. However, the qualia generated by the AND-gate and NOISY-COPY system differ because of how the informational relationships tangle at the top of the qualia; an AND-gate is not simply the combination of two NOISY COPY gates, as can be seen by comparing panels B and D. In the disentangled system, panel D, the actual repertoire of the whole coincides with a product of marginalizations of the actual repertoires of the individual connections.

Figure 5. Modes.

A mode is a maximally densely tangled q-arrow at the top of the quale. (A): A system containing an AND and COPY gate. (B): The quale generated by X. Connections c¹³ and c²³ are tangled at the top of the quale with γ = .25 bits. (C): The system as a whole is not tangled: entanglement between connection c³² and connections {c¹³,c²³} is zero. Thus, the up-set ↑¬{c¹³,c²³} is a mode: it is not contained in a larger up-set with higher γ. (D) Cartoon of a hierarchy of modes in a complex quale.

Figure 6. The relationship between qualia and Φ.

(A): The quale generated by the system in Fig. 3. The down-set ↓MIP of the minimum information partition forms a natural “base” for the complex. The informational relationships among the parts are built on top of the informational relationships generated within the minimal parts. From this perspective the Φ q-arrow (in black) represents the “height” of the quale above its base; the “length” (divergence) of the Φ-q-arrow expresses the breathing room in the system. (B): The quale generated by the pair of couples in Fig. 1B. Although the system generates the same amount of effective information and the same actual repertoire (as a whole) as the system in panel A, it does not do so as a single entity. The system breaks into two independent components (the down-set ↓MIP contains the entire quale). The system reduces to its MIP (base); integrated information Φ = 0 so there is no breathing room and no experience is generated. The system breaks into two disjoint components, each of which forms a complex with Φ>0.

Figure 7. The quale depends on the mechanism and the state.

(AB): The same system (an AND-gate and two XOR-gates) in two different states generates two different qualia, two different experiences. (BC): Two systems in the same state, but with different mechanisms, generate different qualia.

Figure 8. Isomorphisms between qualia.

(A): The simplest possible system: a sensor and a detector, where the detector copies the prior state of the sensor. The quale generated by the system when the detector is ON is a single q-arrow with effective information of 1 bit. The q-arrow specifies the sensor was ON in the previous time step. (B) When the detector is OFF, the system generates a different quale, where the q-arrow points in a different direction – towards a different actual repertoire – specifying that the detector was OFF. Effective information is again 1 bit. (C): A reflection of Q-space generated by relabeling the outputs of n¹ (flipping 0 and 1) induces an isomorphism between the two qualia. (DE): The qualia generated by a silent AND-gate and a firing OR-gate respectively. The two qualia are isomorphic, which can be seen by flipping the roles of 0 and 1.

Figure 9. Context-dependency of informational relationships.

(A): The same set of connections engaged in two different contexts (red arrows) for the system in Fig. 3. At the bottom of the quale (in the null context) the connections generate 1.1 bits of information, whereas the up-set of the connections, in the full context, generates 1.8 bits of information. (B): A system of AND-gates. The four cyan elements generate 1.5 bits of information in the null context and 4 bits of information in the full context. (CDEF): The relationship between Φ and context-dependence. Each panel shows a system of 8 AND-gates with two sets of connections chosen, shown in red and cyan (in panel E a connection is chosen twice). Each point in the graphs shows the average value of the difference: “r in full context – r in null context” = ei(X₀(¬r,x₁)→X₀(T,x₁))−ei(X₀(maxH)→X₀(r,x₁)), averaged across network states where Φ is in the range [k,k+0.5), as k varies from 0 to 3.5 bits. The graphs show that, context as Φ increases, the information generated by a set of connections in the full context increases relative to the same connections in the null.

Figure 10. Collapse of a q-fold.

(A): The quale generated by the system in Fig 3. (B): The connections in cyan are removed and replaced with noise. The quale collapses onto a subquale.

Figure 11. When an element becomes active, it changes the shape of the quale.

(A): the quale generated by the system in Fig. 3, when x₁ = 1000. (B): If element n³ becomes active, changing the firing pattern to x₁ = 1010, the quale changes shape. The firing of an additional element changes almost all of the actual repertoires (see insets).

Figure 12. Inactive versus inactivated elements.

(A): The quale generated by the system in Fig. 3 when no elements are firing. The shape is not drawn to scale, and is considerably smaller than that generated for x₁ = 1000 or 1010: effective information of the whole ei(X₀(maxH)→X₀(T,x₁)) = 1.2 bits, as opposed to 4 bits when element n¹ is spiking. The actual repertoire of the whole is not specified precisely. (B): By contrast, if element n² is inactivated – rather than merely inactive – and connections with source n² are replaced with noise, the quale collapses.

Figure 13. Tangled concepts can generate more information about their inputs than their atomic subconcepts.

(A): An element extracts information from a set of four sensors. If the input received by the sensor layer is pure noise (the maximum entropy distribution on 2⁴ = 16 possible firing patterns) then the best a single element can do, on average, is to extract 1 bit of information. An efficient strategy is to COPY the output of one of the sensors, so that the element generates a concept of the form ON/OFF. (B): An element that spikes if it receives a BAR: 1100, 0110 or 0011. If a bar is presented, the 4 connections together generate 2.4 bits of information, whereas the individual connections generate 0.08 bits independently. For the 4 connections to generate more information as a whole than separately they must be tangled: γ = 0.25 bits. If the input pattern is not a bar, the element generates 0.3 bits, so that it performs worse than the COPY, on average, on maxent noise. However, if bars are sufficiently common in the input, then the element generates more information than a COPY element. (C): Two elements COPY their inputs. This produces the maximum possible average effective information (2 bits for 2 binary elements) assuming the inputs are maxent distributed. The elements are not tangled, γ = 0, and so the whole generates information equal to the sum of the parts. (D): A cartoon cortical area: a subsystem that receives more inputs than there are elements. If there is some statistical structure to the inputs (certain patterns are more common than others), the system can form concepts specific to the input structure. The 2 binary elements shown generate 4 bits of information about the input pattern, more than the elements taken individually (2.7 bits). On average, using maxent, the 2 elements generate 1.8 bits, less than the COPY elements. However, if the inputs are structured and so not maxent, the elements can generate more information about other cortical areas than they should “by right” by tangling informational relationships into concepts and modes.

Figure 14. Learning to distinguish new experiences enriches the shape of the quale generated by a system.

(A): A system of elements, containing two detectors (AND-gates that respond to >1 spike) and four sensors, on which we focus attention. The sensors have all-to-all connections with the detectors. Both detectors are firing, which occurs for any of the sensor patterns 1011, 1010 and 0011 (amongst others): “wine”. (B): The quale generated by the system. The maroon and gray submechanisms (containing 4 connections targeting each detector) generate a single q-arrow due to the redundancy of the all-to-all connectivity. The system generates the same quale in response to three different sensor patterns: “rosé wine” (1011), “red wine” (1010) and “white wine” (0011). (C): The system learns to distinguish between types of wine by pruning three connections; as before detectors are AND-gates, however, since their inputs differ they are no longer redundant. (DEF): The three sensor patterns generate three different qualia. Moreover, each quale is richer than in panel B: the single q-arrow has split into 4 q-arrows, reflecting the increased richness in how the taste of different wines is specified.

Figure 15. Modes depend on network structure and network activity.

Elements in all panels are AND-gates firing if they receive 2 or more spikes. Lines represent bidirectional arrows. (ABC): Modes and network structure. (A): A honeycomb grid (with bidirectional connections and torus edges) generates a single mode. γ(“orange”) computes entanglement for the elements colored orange in panel B. (B): Removing most of the diagonal connections, results in a system containing two weakly tangled modes, shown in cyan and orange, arranged in a chessboard-pattern. The single diagonal connection loosely tangles the two modes. (C): A diagonal slice of a feedforward grid. Each layer of the grid is a separate mode, disentangled from the others. (DEF): Modes and network activity. (D): “Nothingness”. A silent system forms a single, homogeneous mode. (E): “Pure red”. The system as a whole forms a weak mode (orange). The strongest mode (cyan) is created by the firing of a single element. (F): “A composite experience”. A more complex firing pattern results in multiple overlapping modes, two of which are shown. (G): A 2D cartoon of modes in a quale. At the top is the color mode. Currently, the system is exposed to a red stimulus, so the informational relationships within the mode specify the redness of red: the direction of the q-arrows within the mode – and how they are tangled – is what makes red different from green or blue. However, the context afforded to red – the fact that it is a visual rather than auditory experience – is not a property of the color mode. The color mode is contained in a series of larger modes: form, vision, perception, which fill in the context in which the redness of red is specified. The vision mode as a whole is a tangled concept, which cannot be decomposed into independent subconcepts, even though the submodes, such as color and motion, have a certain amount of independence. Color is always associated with a shape of some kind (a totally red visual field is a particular shape), and also motion (awareness of lack of motion is awareness of a kind of motion), and so forth. The quale of the entire system itself forms a mode since γ>0.

Figure 16. The qualia generated by topographical grids and categorizing pyramids.

(A) A honeycomb grid and a schematic representation of part of the quale generated by the grid. In the grid, each element is bidirectionally connected to its 6 neighbors, and fires if it receives 3 or more spikes. The cell at the center of the gray area is silent, and so generates the concept “local activity below threshold”. Three of the connections targeting the cell are shown in pink; the corresponding q-arrows at the bottom of the quale are tangled into the overarching concept given by the larger gray q-arrow. Similarly for the cell at the center of the brown area that – as shown in the quale – tangles the connections shown in blue. The quale shows how the grid generates two concepts for “local activity below threshold” in two different regions (the two deformed cubes generated by pink and cyan q-arrows). The concept generated by th pink q-arrows taken as a whole is represented by a gray q-arrow at the bottom of the quale; similarly a brown q-arrow is drawn for the concept generated by the cyan q-arrows as a whole. The combined concept “activity below threshold in the gray and brown regions” does not exist for the grid because the brown and gray q-arrows are not tangled at the bottom of the quale. The overarching informational relationship generated by the gray, beige and brown areas together does form a single concept in the quale “regional activity below threshold”. (B): Part of a categorizing pyramid extracting invariants from a grid and a schematic of the quale. The categorizing pyramid has near all-to-all connectivity, so there is no topographic structure, which is reflected in the quale by tangling “all the way down”. In contrast to the grid, where the topographic structure serves to prevent concepts from tangling at the bottom of the quale, giving the experience a spatial aspect, the all-to-all connectivity results in all concepts tangling into a single indivisible experience similar to color or smell.

Figure 17. Hierarchical experiences.

(A): Higher-order feature detectors extract a hierarchy of patterns (edges, features, and faces) from a retina-like grid. (B): A schematic depiction of the quale generated by the hierarchy; since each pattern-detector contains many elements and connections, the actual quale will be vastly more complicated than the simple cartoon shown here. The actual repertoires generated along two q-edges are shown. First, consider the clockwise q-edge. The cyan connections – targeting the edge detectors – specify that the image presented to the retina contains certain edges. The edge and feature detectors taken together specify that the edges coalesce into features such as a mouth, nose and eyes. Finally, all the connections in the hierarchy specify the particular face that is shown to the retina. Going around anti-clockwise, the “face” connections on their own specify that the retina-grid was presented with a face-like object, however, the details of the face are unspecified, since the concepts for mouth etc. are not generated by the face-neurons. Engaging the connections targeting the feature-neurons fills out some of the details of the face, the broad outlines of how the nose, mouth and eyes appear. Finally, adding connections targeting the edge-neurons specifies the face precisely. The informational relationships generated by neurons in a tangled quale cannot be described in isolation.