Quantifying 'causality' in complex systems: understanding transfer entropy

Abstract

'Causal' direction is of great importance when dealing with complex systems. Often big volumes of data in the form of time series are available and it is important to develop methods that can inform about possible causal connections between the different observables. Here we investigate the ability of the Transfer Entropy measure to identify causal relations embedded in emergent coherent correlations. We do this by firstly applying Transfer Entropy to an amended Ising model. In addition we use a simple Random Transition model to test the reliability of Transfer Entropy as a measure of 'causal' direction in the presence of stochastic fluctuations. In particular we systematically study the effect of the finite size of data sets.

Figure 1. Susceptibility on the Ising model with lengths L=10,25,50,100 obtained using equation (9).

Peaks can be seen at respective .

Figure 2. Covariance on the Ising model with lengths L=10,25,50,100 obtained using equation (10).

Figure 3. Mutual Information on the Ising model with lengths L=10,25,50,100 obtained using equation (4).

Figure 4. Transfer Entropy and on the Ising model of lengths L=50 obtained using equation (5).

Peaks for both direction are at .

Figure 5. Transfer Entropy on the Ising model of lengths L=10,25,50,100 obtained using equation (5).

Peaks can be seen at respective .

Figure 6. Transfer Entropy on the Ising model of lengths L=10,25,50,100 obtained using equation (5).

Peaks can be seen at respective .

Figure 7. Susceptibility on the amended Ising model of lengths L=10,25,50,100 obtained using equation (9).

Peaks can be seen at respective .

Figure 8. Covariance on the amended Ising model of lengths L=10,25,50,100 obtained using equation (10).

Peaks can be seen at respective , similar to Figure (2) of the Ising model.

Figure 9. Mutual Information on the amended Ising model with lengths L=10,25,50,100 obtained using equation (4).

Not much different from results on the Ising model in Figure 3.

Figure 10. Transfer Entropy and on the amended Ising model of lengths and , obtained using equation (5).

Direction at time lag is indicated. Very different from result on Ising model in Figure 4.

Figure 11. Transfer Entropy on the Ising model of lengths L=10,25,50,100 obtained using equation (5).

Values continue to increase after which is very different from Figure (5).

Figure 12. Transfer Entropy on the Ising model of lengths L=10,25,50,100 obtained using equation (5).

Peaks can be seen at respective , similar to Ising model results in Figure (6).

Figure 13. versus for different time lags in amended Ising model with and using equation (5).

The figure shows the effect of separation in time.

Figure 14. A different view of Figure (13) where versus for different temperatures is plotted instead.

. Figure highlights time lag detection.

Figure 15. in Figure 17 up to .

Transfer Entropy stabilizes due to Boltzmann distribution that approaches uniform distribution at higher temperatures.

Figure 16. , and in the Ising model with .

due to distance (separation) in space where is closer to than . The nearest neighbour effect is observed.

Figure 17. , and in the amended Ising model with and .

due to implanted ‘causal’ lag. The effect of separation in space is no longer visible.

Figure 18. (Expected rate of change) of sites , and on amended Ising model with and .

Figure 19. on amended Ising model with and displaying phase-transition like behaviour.

Figure 20. on amended Ising model with and .

All with phase-transition like jump.

Figure 21. Analytical Transfer Entropy versus time lags of the Random Transition model with (hence ) and in equation (16) where is varied but fixed.

is monotonically increasing with respect to . is affected by . Figure illustrates how the internal dynamics of influences when is the target variable. Transfer Entropy changes even though external influence is constant.

Figure 22. Analytical Transfer Entropy versus time lags of the Random Transition model with (hence ) and in equation (16) where fixed and is varied.

Only at , does not effect and values remain constant. For at , Transfer Entropy is affected by . and coincides. Figure shows how the internal dynamics of influences when is the source variable.

Figure 23. Transfer Entropy versus number of state (number of chosen bins) for Cases and .

are uniformly distributed. Analytical values obtained from substituting respective values in equation (17). Simulated values are acquired using equation (5) on simulated data of varying sample size (length of time series) where . Error bars are displaying two standard deviation values above and two standard deviation below (some bars are very small, it can barely be seen). The aim is primarily to display how choosing has to be made according to length, , of available time series. For large the error bar becomes smaller than the width of the curve.

Figure 24. Transfer Entropy using equation (17) on simulated null model with varying sample size or length of time series, where .

Analytical values are all . Error bars in the first figure are displaying two standard deviation values above and two standard deviation below. For large the error bar becomes smaller than the width of the curve. In order to use the null model as surrogates, still has to be chosen in accordance to .