Upper bound for the Mutual Information Rate (MIR) in an Active Network

In a recent publication [10], we have argued that the mutual information rate (MIR) between two elements in an active chaotic network, namely, the amount of information per unit time that can be realized in one element, k, by measuring another element, l, regarded as I_C, is given by the sum of the conditional Lyapunov exponents associated with the synchronization manifold (regarded as λ^∥) minus the positive conditional Lyapunov exponents associated with the transversal manifold (regarded as λ^⊥). So, I_C = λ^∥−λ^⊥.

As shown in [11], if one has N = 2 coupled chaotic systems, which produce at most two positive Lyapunov exponents λ₁, λ₂ with λ₁>λ₂, then λ^∥ = λ₁ and λ^⊥ = λ₂. Denote the trajectory of the element k in the network by x_k. For larger number of elements, N, the approaches proposed in [10] remain valid whenever the coordinate transformation X_kl∥ = x_k+x_l (which defines the synchronization manifold) and X_kl⊥ = x_k−x_l (which defines the transversal manifold) successfully separates the two systems k and l from the whole network. Such a situation arises in networks of chaotic maps of the interval connected by a diffusively (also known as electrically or linear) all-to-all topology, where every element is connected to all the other elements. These approaches were also shown to be approximately valid for chaotic networks of oscillators connected by a diffusively all-to-all topology. The purpose of the present work is to extend these approaches and ideas to active networks with arbitrary topologies.

Consider an active network formed by N equal elements, x_i (i = 1,…,N), where every D-dimensional element has a different set of initial conditions, i.e., x₁≠x₂≠…≠xN. The network is described by(1)where G_ij is the ij element of the coupling matrix. Since we choose in order for a synchronization manifold to exist by the subspace η = x₁ = x₂ = x₃ = … = x_N, we can call this matrix the Laplacian matrix.

The way small perturbations propagate in the network [21] is described by the i (i = 1,…,N) variational equations of Eqs. (1), namely writing x_i = η+δx_i and expanding Eq. (1) in δx_i,(2)obtained by linearly expanding Eq. (1).

Making x_i = ξ, which can be easily numerically done by setting the elements with equal initial conditions and taking H(x_j) = x_j, Eq. (2) can be made block diagonal resulting in(3)where γ_i are the eigenvalues (positive defined) of the Laplacian matrix ordered such that γ_i+1≥γ_i. Note that γ₁ = 0.

Notice that the network dynamics is described by Eq. (1), which assumes that every element has different initial conditions and therefore different trajectories (except when the elements are completely synchronized). On the other hand, Eq. (3) that provides the conditional exponents considers that all the initial conditions are equal. While Eq. (2) provides the set of Lyapunov exponents of an attractor, Eq. (3) provides the Lyapunov exponents of the synchronization manifold and its transversal directions. Notice also that when dealing with linear dynamics, the Lyapunov exponents [obtained from Eq. (2)] are equal to the conditional exponents [obtained from Eq. (3)] independently on the initial conditions.

Then, the upper bound of the MIR that can be measured from an element x_k by observing another element x_l, i.e. the upper bound of the MIR in the communication channel cⁱ⁻¹ is(4)with i∈(2,…,N), and λⁱ representing the sum of all the positive Lyapunov exponents of the equation for the mode ξ_i, in Eq. (3). So, λ¹ is the sum of the positive conditional exponents obtained from the separated variational equations, using the smallest eigenvalue associated with the exponential divergence between nearby trajectories around ξ, the synchronous state, and λⁱ (i>1) are the sum of the positive conditional exponents of one of the possible desynchronous oscillation modes. Each eigenvalue γ_i produces a set of conditional exponents , with m = 1,…,D.

Although Eq. (4) gives the upper bound for the amount of information between modes of oscillation, for some simple network geometries, as the ones studied here, we can relate the amount of information exchanged between two vibrational modes to the amount of information between two elements of the network, and therefore, Eq. (4) can be used to calculate an upper bound for the MIR exchanged between pairs of elements in the network. For larger and complex networks, this association is non-trivial, and we rely on the reasonable argument that a pair of elements in an active network cannot transmit more information than some of the i−1 values of .

The inequality in Eq. (4) can be interpreted in the following way. The right hand side of Eq. (4) calculates the amount of information that one could transmit if the whole network were completely synchronous with the state ξ, which is only true when complete synchronization takes place and when all the nodes have equal dynamics. Typically, we expect that the elements of the network will not be completely synchronous to ξ and in realistic networks, the nodes will not be equal. Thus, the amount of information provided by the right part of Eq. (4) overestimates the exact MIR which, due to desynchronization in the network, should be smaller than the calculated one.

Equation (5) allows one to calculate the MIR between oscillation modes of larger networks with arbitrary topology rescaling the MIR curve ( vs. σ) obtained from two coupled elements. Denoting σ^*(N = 2) as the strength value for which the curve for λ² reaches a relevant value, say, its maximum value, then the coupling strength for which this same maximum is reached for λⁱ in a network composed by N elements is given by(5)where γ_i(N) represents the ith largest eigenvalue of the N-elements network. If the network has an all-to-all topology, thus, σ^*(N = 2) represents the strength value for which the curve of reaches a relevant value, and σ^*(N) the strength value that this same value for is reached.

Notice that symmetries in the connecting network topology leads to the presence of degenerate eigenvalues ( = equal eigenvalues) in the Laplacian matrix, which means that there are less independent channels of communication along which information flows. Calling Q the number of degenerate eigenvalues of the Laplacian matrix, Eq. (4) will provide N−Q different values.

As the coupling strength σ is varied, the quantities that measure information change correspondingly. For practical reasons, it is important that we can link the way these quantities (see Sec. Methods) change with the way the different types of synchronization show up in the network. In short, there are three main types of synchronization observed in our examples (see [11]): burst phase synchronization (BPS), when at least one pair of neurons are synchronous in the slow time-scale but desynchronous in the fast time-scale, phase synchronization (PS), when all pairs of neurons are phase synchronous, and complete synchronization (CS), when all pairs of neurons are completely synchronous. The coupling strength for which these synchronous phenomena appear are denoted by σ_BPS, σ_PS, and σ_CS (with no superscript index).

Finally, there are a few more relevant coupling strengths, which characterize each communication channel. First, , for which λⁱ equals the value of λ¹, with i≥2. For , the communication channel cⁱ⁻¹ (whose upper rate of information transmission depends on the two oscillation modes ξ₁ and ξ_i) behaves in a self-excitable way, i.e., λ¹<λⁱ. For , λ¹≥λⁱ. Secondly, σ^i* indicates the coupling strength at which is maximal. Thirdly, indicates the coupling strength for which the communication channel cⁱ⁻¹ becomes “stable”, i.e., λⁱ<0. At σ = σ^i* the self-excitable channel capacity of the channel cⁱ⁻¹ is reached and at , the non-self-excitable channel capacity is reached. Finally, σ_C is the coupling for which the network capacity is reached, and then, when the KS-entropy of the network is maximal.

The MIR in networks of coupled Hindmarsh-Rose neurons

We investigate how information is transmitted in self-excitable networks composed of N bidirectionally coupled Hindmarsh-Rose neurons [18]:(6)The parameter r modulates the slow dynamics and is set equal to 0.005, such that each neuron is chaotic. The index i≠j assumes values within the set [1,…,N]. S_k represents the subsystem formed by the variables (x_k, y_k, z_k) and S_l represents the subsystem formed by the variables (x_l, y_l, z_l), where k = [1,…,N−1] and l = [k+1,…,N]. The Laplacian matrix is symmetric, so G_ji = G_ij, and σG_ji is the strength of the electrical coupling between the neurons, and we take for I_i the value I_i = 3.25.

In order to simulate the neuron network and to calculate the Lyapunov exponents through Eq. (2), we use the initial conditions x = −1.3078+η, y = −7.3218+η, and z = 3.3530+η, where η is an uniform random number within [0,0.02]. To calculate the conditional Lyapunov exponents, we use the equal initial conditions, x = −1.3078, y = −7.3218, and z = 3.3530.

All-to-all coupling.

Here, we analyze the case where N neurons are fully connected to every other neuron. The Laplacian matrix has N eigenvalues, γ₁ = 0, and N−1 degenerate ones γ_i = N, i = 2,…,N. Every pair of neurons exchange an equal amount of MIR. Although, there are N×(N−1)/2 pairs of neurons, there is actually only one independent channel of communication, i.e., a perturbation applied at some point of the network should be equally propagated to all other points in the network. In Fig. 1(A), we show the MIR, I_C, calculated using the approaches in Refs. [10], [11], I_P, calculated using the right hand-side of Eq. (4), and I_S, calculated encoding the trajectory between pair of neurons, and the Kolmogorov-Sinai entropy, H_KS, for a network composed by N = 2 neurons. In (B), we show these same quantities for a network formed by N = 4 neurons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. The quantities I_C (black circles), I_P (red squares), I_S (green diamonds), and H_KS (blue diamonds), for two (A) and four (B) coupled neurons, in an all-to-all topology.

Notice that since there are only two different eigenvalues, there is only one channel of communication whose upper bound for the MIR is given by I_P = |λ¹−λ²|. Also, I_S and I_C represent the mutual information exchanged between any two pairs of elements in the system. In (A), σ^2* = 0.092, σ_BPS≅0.2, , σ_PS = 0.47, and σ_CS = 0.5. In (B), σ^2* = 0.046, σ_BPS≅0.1, , σ_PS = 0.24, and σ_CS = 0.25. CS indicates the coupling interval σ≥σ_CS for which there exists complete synchronization.

https://doi.org/10.1371/journal.pone.0003479.g001

While for σ≅0 and σ≥σ_CS, we have that I_C≅I_P≅I_S, for σ≅σ^2* (when the self-excitable channel capacity is reached) it is clear that I_P should be an upper bound for the MIR, since not only I_P>I_C but also I_P>I_S. Notice the good agreement between I_C and I_S, except for , when I_S>H_KS, which violates Eq. (11).

The star symbol indicates the value of the coupling, σ_BPS, for which burst phase synchronization (BPS) appears while the spikes are highly desynchronous. The appearance of BPS coincides with the moment where all the quantifiers for the MIR are large, and close to a coupling strength, σ_C, for which the network capacity is reached (when H_KS is maximal).

At this point, the network is sufficiently desynchronous to generate a large amount of entropy, which implies a large λⁱ, for i≥2. This is an ideal configuration for the maximization of the MIR. There exists phase synchrony in the subspace of the slow time-scale z variables (which is responsible for the bursting-spiking behavior), but there is no synchrony in the (x,y) subspace. This supports the binding hypothesis, a fundamental concept of neurobiology [19] which sustains that neural networks coding the same feature or object are functionally bounded. It also simultaneously supports the works of [22], which show that desynchronization seems to play an important role in the perception of objects as well. Whenever λ² approaches zero, at σ = σ_CS, there is a drastic reduction in the value of H_KS as well as I_P, since the network is in complete synchronization (CS), when all the variables of one neuron equals the variables of the other neurons.

Therefore, for coupling strengths larger than the one indicated by the star symbol, and smaller than the one where CS takes place, there is still one time-scale, the fast time-scale, which is out of synchrony.

For , the only independent communication channel is of the non-self-excitable type. That means λⁱ≤λ¹ (i≥2), and as the coupling strength increases, H_KS decreases and I_P increases.

Note that the curve for I_P shown in Fig. 1(B) can be obtained by rescaling the curve shown in Fig. 1(A), applying Eq. (5).

Star coupling.

We consider N = 4. There is a central neuron, denoted by S₁, bidirectionally connected to the other three (S_k, k = 2,3,4), but none of the others are connected among themselves. The eigenvalues of the Laplacian matrix are γ₁ = 0, γ_2,3 = 1, γ₄ = N.

To treat general types of networks, it is useful to define two quantities related to the excitability of the communication channels. The here called non-self-excitable (NSE) robustness parameter of the channel cⁱ⁻¹ (i≥2) as and the self-excitable (SE) robustness parameter for the communication channel cⁱ⁻¹ as (i≥2). It is also useful to define a quantity that measures the distance between the eigenvalues, the normalized spectral distance (NED) between the two eigenvalues.

Having a large NED between the ith largest and the first largest eigenvalues (γ_i−γ₂)/N, results in a non-self-excitable channel, cⁱ⁻¹, with a large NSE robustness parameter that implies that the channel preserves its NSE character under large alterations of the coupling strength. On the other hand, having a large NED between the largest and the ith largest eigenvalues (γ_N−γ_i)/N, results in a self-excitable channel, cⁱ⁻¹, with a large self-excitable robustness parameter that implies that the channel preserves its SE character under large alterations of the coupling strength.

So, for the star topology network, not only the NED between γ_N and γ_N−1 is large but also between γ_N and γ_N−2, and therefore, are large. This provides a network whose channels c¹ and c² have a large MIR for a large coupling strength alteration. Note that if γ_N−1 is far away from γ_N that implies that γ_N−2 is also far away from γ_N. Thus, a reasonable spectral distance between γ_N−1 and γ_N is a “biological requirement” for the proper function of the network, since even for larger coupling strengths there will be at least one oscillation mode which is desynchronous, a configuration that enables perturbation (meaning external stimuli) to be propagated within the network [23].

The largest eigenvalue is related to an oscillation mode where all the outer neurons are in synchrony with each other but desynchronous with the central neuron. So, here it is clear the association between |λ¹−λ⁴| and the MIR between the central neuron with an outer neuron, since λ¹ represents the amount of information of the synchronous trajectories among all the neurons, while λ⁴ is the amount of information of the desynchronous trajectories between the central neuron and any outer neuron. The other eigenvalues (γ₂,γ₃) represent directions transverse to the synchronization manifold in which the outer neurons become desynchronous with the central neuron in waves wrapping commensurately around the central neuron [21]. Thus, λ² and λ³ are related to the error in the transmission between two outer neurons, k and l, with k,l≠1. Notice that this network topology has two independent channels of communication.

Note that the MIR between S₁ and an outer neuron (upper bound represented by and I_S represented by I_S (1, k), in Fig. (2) is larger (smaller) than the MIR between two outer neurons (upper bound represented by and I_S is represented by I_S (k, l), in Fig. (2), for small coupling (for when the channel c³ is self-excitable, and ). Similar to what happens to nearest-neighbour networks, the self-excitable and the non-self-excitable channel capacities of the channel associated with the transmission of information between closer elements (the channel c³) are achieved for a smaller value of the coupling strength than the one necessary to make the channels associated with the transmission of information between more distant elements (the channel c¹) to achieve its two channel capacities. That property permits this network, for , to transmits simultaneously reliable information using the channel c³ and with a higher rate using the channel c¹.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. MIR between the central neuron and an outer one (black circles), , (resp. I_S (1, k), in green line), and between two outer ones (red squares), , (resp. I_S (k, l), in blue line).

Blue diamonds represents the KS-entropy. Other quantities are σ^4* = 0.181, σ^2* = 0.044, , , , σ_BPS = 0.265, σ_PS = 0.92, and σ_CS = 1.0. The star indicates the parameter for which BPS first appears.

https://doi.org/10.1371/journal.pone.0003479.g002

Notice, in Fig. 2, that . So, when the channel capacity of the channel c¹ is reached, also H_KS of the network is maximal, and the network operates with its capacity.

Another point that we want to emphasize in this network is that while a large NED between γ_N and γ_N−1 provides a network whose channel c¹ is self-excitable and can transmit information at a large rate for a large coupling strength interval, a large NED between γ₃ and γ₂ leads to a non-self-excitable channel c³ even for small values of the coupling amplitudes, and it remains non-self-excitable for a large variation of the coupling strength. Thus, while a large NED between the second and the first largest eigenvalues leads to a network whose channels are predominantly of the self-excitable types, a large NED between the second largest and the third largest eigenvalues provide a network whose communication channels are predominantly of the non-self-excitable types.

Eigenvalues conditions

Finding network topologies and coupling strengths in order to have a network that operates in a desired fashion is not a trivial task (see Sec. Methods). An ideal way to proceed would be to evolve the network topology in order to achieve some desired behaviour. In this paper, we are interested in maximizing simultaneously I_P, the KS-entropy, and the average 〈I_P〉, for a large range of the coupling strength, characteristics of a quasi-optimal network. However, evolving a network in order to find a quasi-optimal one would require the calculation of the MIR in every communication channel and H_KS for every evolution step. For a typical evolution, which requires 10⁶ evolution steps, such an approach is impractical.

Based on our previous discussions, however, a quasi-optimal network topology can be realized by only selecting an appropriate set of eigenvalues which have some specific NED. Evolving a network by the method of Sec. Methods using a cost function which is a function of only the eigenvalues of the Laplacian matrix is a practical and physible task.

The present section is dedicated to describe the derivation of this cost function.

We can think of two most relevant sets of eigenvalues which create quasi-optimal networks, and they are represented in Fig. 3. Either it is desired eigenvalues that produce a network predominantly self-excitable [SE, in Fig. 3] or predominantly non-self-excitable [NSE, in Fig. 3].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Representation of the eigenvalues sets that produce quasi-optimal self-excitable (SE) and non-self-excitable active networks (NSE).

https://doi.org/10.1371/journal.pone.0003479.g003

In a network whose communication channels are predominantly self-excitable, it is required that the NED (γ_N−γ_N−1)/N is maximal and (γ_N−1)/N minimal. Therefore, we want a network for which the cost function(7)is maximal.

A network whose eigenvalues maximize B₁ has self-excitable channels for a large variation of the coupling strength. As a consequence, 〈I_P〉 as well as H_KS is large for .

In a network whose communication channels are predominantly non-self-excitable, it is required that the NED (γ₃−γ₂)/N is maximal and (γ₂)/N minimal. Therefore, we want a network for which the cost function(8)is maximal.

A network whose eigenvalues maximize the condition in Eq. (8) have non-self-excitable channels for a large variation of the coupling strength. As a consequence, 〈I_P〉 is large for , which is a small coupling range, but since there is still one oscillation mode that is unstable (the mode ξ₂), H_KS is still large for a large range of the coupling strength . Most of the channels will transmit information in a reliable way, since the error in the transmission, provided by λⁱ (i≥2), of most of the channels will be zero, once λⁱ<0.

Since degenerate eigenvalues produce networks with less vibrational modes and therefore less independent channels of communication, we assume in the following the absence of such degenerate eigenvalues. In addition, we assume that there is a finite distance between eigenvalues so that the network becomes robust under rewiring, and therefore, perturbing G_ij will not easily create degenerate eigenvalues.

A network that is completely synchronous and has no unstable modes does not provide an appropriate environment for the transmission of information about an external stimulus, because they prevent the propagation of perturbations. Networks that can be easily completely synchronized (for small coupling strengths) requires the minimization of γ_N−γ₂, or in terms of the eigenratio, the minimization of γ_N/γ₂. We are not interested in such a case. To construct network topologies that are good for complete synchronization, see Refs. [21], [24], [25], [26].

Quasi-optimal topologies for information transmission

Before explaining how we obtain quasi-optimal network topologies for information transmission, it is important to discuss the type of topology expected to be found by maximizing either B₁, in Eq. (7) or B₂, in Eq. (8). Notice that Laplacians whose eigenvalues maximize B₁ are a perturbed version of the star topology, and the ones that maximize B₂ are a perturbed version of the all-to-all topology. In addition, in order to have a network that presents many independent modes of oscillations it is required that the Laplacian matrix presents as much as possible, a large number of non-degenerate eigenvalues. That can be arranged by rewiring (perturbing) networks possessing either the star or the nearest-neighbour topology, breaking the symmetry.

In order to calculate a Laplacian from a quasi-optimal network, we propose an approach described in Sec. Methods, based on the reconstruction of the network by evolving techniques, simulating the process responsible for the growing or rewiring of real biological networks, a process which tries to maximize or minimize some cost function.

In order to better understand how a network evolves (grows) in accordance with the maximization of the cost functions in Eqs. (7) and (8), we first find the network configurations with a small number of elements. To be specific, we choose N = 8 elements. To show that indeed the calculated network topologies produce active networks that operate as desired, we calculate the average upper bound value of the MIR [Eq. (10)] for neural networks described by Eqs. (6) with the topology obtained by the evolution technique, and compare with other network topologies. Figure 4 shows 〈I_P〉, the average channel capacity, calculated for networks composed of 8 elements, using one of the many topologies obtained by evolving the network maximizing B₁ (circles, denoted in Fig. by “evolving 1”), all-to-all topology (squares), star topology (diamonds), nearest-neighbor (upper triangle), and maximizing B₂ (down triangle, denoted in Fig. by “evolving 2”). The star points to the value of , when c¹, the most unstable communication channel (a self-excitable channel), becomes non-self-excitable.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. The average value of the upper bound MIR, 〈I_P〉 [as defined in Eq. (10)] for active networks composed of 8 elements using one of the many topologies obtained by evolving the network maximizing B₁ (circles), all-to-all topology (squares), star topology (diamonds), nearest-neighbor (upper triangle), and maximizing B₂ (down triangle).

The values of indicated by the starts are (evolving 1), (all-to-all), (star), (nearest-neighbor), and (evolving 2). The evolving 1 network has a Laplacian with relevant eigenvalues γ₇ = 3.0000, γ₈ = 6.1004, which produces a cost function equal to B₁ = 1.033. The evolving 2 network has a Laplacian with relevant eigenvalues γ₂ = 0.2243 and γ₃ = 1.4107, which produces a cost function equal to B₂ = 5.2893.

https://doi.org/10.1371/journal.pone.0003479.g004

As desired the evolving network 1 has a large upper bound for the MIR (as measured by 〈I_P〉) for a large range of the coupling strength, since the network has predominantly self-excitable channels. The channel c¹ has a large robustness parameter , i.e., it is a self-excitable channel for , where . In contrast to the other topologies, in the star, nearest-neighbour, and all-to-all topologies, is smaller and is larger. Even though most of the channels in the evolving 2 topology are of the non-self-excitable type, 〈I_P〉 remains large even for higher values of the coupling strength. That is due to the channel c¹ which turns into a self-excitable channel only for σ>2.

The KS-entropies of the 5 active networks whose 〈I_P〉 are shown in Fig. 4 are shown in Fig. 5. Typically, the network capacities are reached for roughly the same coupling strength for which the maximum of 〈I_P〉, is reached. In between the coupling strength for which the network capacities and the maximal of 〈I_P〉 are reached, λ³ becomes negative. At this point, also BPS appears in the slow time-scale, suggesting that this phenomena is the behavioral signature of a network that is able to transmit not only large amounts of information between pairs of elements (high MIR) but also overall within the network (high H_KS).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. KS-entropy for the same active networks of Fig. 4 composed of 8 elements.

https://doi.org/10.1371/journal.pone.0003479.g005

Note however, that since the evolving networks have a small number of elements, the cost function cannot reach higher values and therefore, the networks are not as quasi-optimal as they can be. For that reason, we proceed now to evolve larger networks, with N = 32.

Maximization of the cost function B₁ leads to the network connectivity shown in Fig. 6(A) and maximization of the cost function B₂ leads to the network connectivity shown in Fig. 6(B). In (A), the network has the topology of a perturbed star, a neuron connected to all the other outer neurons, thus a hub, and each outer neuron is sparsely connected to other outer neurons. The arrow points to the hub. In (B),the network has the topology of a perturbed all-to-all network, where elements are almost all-to-all connected. Note that there is one element, the neuron S₃₂, which is only connected to one neuron, the S₁. This isolated neuron is responsible to produce the large spectral gap between the eigenvalues γ₃ and γ₂.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. A point in this figure in the coordinate k×l means that the elements S_k and S_l are connected with equal couplings in a bidirectional fashion.

In (A), a 32 elements network, constructed by maximizing the cost function B₁ in Eq. (7) and in (B), 32 elements network, constructed by maximizing the cost function B₂ in Eq. (8). In (A), the network has the topology of a perturbed star, a hub of neurons connected to all the other neurons, where each outer neuron is sparsely connected to other neurons. The arrow points to the hub. In (B),the network has the topology of a perturbed all-to-all network, where elements are almost all-to-all connected. Note that there is one element, the neuron S₃₂, which is only connected to one neuron, the S₁. This isolated neuron is responsible to produce the large spectral gap between the eigenvalues γ₃ and γ₂. In (A), the relevant eigenvalues are γ₃₁ = 4.97272, γ₃₂ = 32, which produce a cost function equal to B₁ = 5.43478. In (B), the relevant eigenvalues are γ₂ = 0.99761, γ₃ = 27.09788, which produce a cost function equal to B₂ = 26.1628.

https://doi.org/10.1371/journal.pone.0003479.g006

〈I_P〉 for the network topology represented in Fig. 6(A) is shown in Fig. 7 as circles, and 〈I_P〉 for the network topology represented in Fig. 6(B) is shown in Fig. 7 as squares. We see that the star topology, whose connectivity is represented in 6(A), has larger 〈I_P〉 for a larger coupling strength than the topology whose connectivity is represented in 6(B). Other relevant parameters of the network whose topology is represented in 6(A) are , , , σ_CS = 0.9762 and for the topology represented in 6(B) are , , , and σ_CS = 0.9761.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. 〈I_P〉 for the networks shown in Fig. 6(A–B) by circles and squares, respectively.

https://doi.org/10.1371/journal.pone.0003479.g007

It is worth to comment that the neocortex is being simulated in the Blue Brain project, by roughly creating a large network composed of many small networks possessing the star topology. By doing that, one tries to recreate the way minicolumnar structures [19] are connected to minicolumnar structures of the neocortex [27]. Each minicolumn can be idealized as formed by a pyramidal neuron (the hub) connected to its interneurons, the outer neurons in the star topology, which are responsible for the connections among this minicolumn (small network) to others minicolumn. So, the used topology to simulate minicolumns is an good topology in what concerns the transmission of information.

Active networks formed by non-chaotic elements

The purpose of the present work is to describe how information is transmitted via an active media, a network formed by dynamical systems. There are three possible asymptotic stable behaviours for an autonomous dynamical system: chaotic, periodic, or quasi-periodic. A quasi-periodic behaviour can be usually replaced by either a chaotic or a periodic one, by an arbitrary perturbation. For that reason, we neglect such a state and focus the attention on active channels that are either chaotic or periodic.

Equation (4) is defined for positive exponents. However, such an equation can also be used to calculate an upper bound for the rate of mutual information in systems that also possess negative Lyapunov exponents. Consider first a one-dimensional contracting system being perturbed by a random stimulus. Further consider that the stimulus changes the intrinsic dynamics of this system. This mimics the process under which an active element adapts to the presence of a stimulus.

Suppose the stimulus, θ_n, can be described by a discrete binary random source with equal probabilities of generating ‘0’ or ‘1’. Whenever θ_n = 0, the system presents the dynamics x_n+1 = x_n/2, otherwise x_n+1 = (1+x_n)/2. It is easy to see that the only Lyapunov exponent of this mapping, λ₁, which is equal to the conditional exponent, λ¹, is negative. Negative exponents do not contribute to the production of information. From Eq. (4) one would arrive at I_P = 0. However, all the information about the stimulus is contained in the trajectory. If one measures the trajectory x_n, one knows exactly what the stimulus was, either a ‘0’ or a ‘1’. The amount of information contained in the stimulus is log(2) per iteration which equals the absolute value of the Lyapunov exponent, |λ₁|. In fact, it is easy to show that I_C = I_P = |λ¹| = |λ₁| = log(2), or if we use the interpretation of [28], I_C = I_P = λ, where λ = |λ₁| is the positive Lyapunov exponent of the time-inverse chaotic trajectory, x_n+m, x_n+m−1, …, x₀, which equals the rate of information production of the random source. So, in this type of active communication channel, one would consider in Eq. (4) the positive Lyapunov exponents of the time-inverse trajectory, or the absolute value for the negative Lyapunov exponent.

Another example was given in [11]. In this reference we have shown that a chaotic stimulus perturbing an active system with a space contracting dynamics (a negative Lyapunov exponent) might produce a fractal set. We assume that one wants to obtain information about the stimulus by observing the fractal set. The rate of information retrieved about the stimulus on this fractal set equals the rate of information produced by the fractal set. This amount is given by D₁|λ|, where D₁ is the information dimension of the fractal set and |λ| the absolute value of the negative Lyapunov exponent. In fact, D₁|λ| is also the rate of information produced by the stimulus. So, if an active system has a space contracting dynamics, the channel capacity equals the rate of information produced by the stimulus. In other words, the amount of information that the system allows to be transmitted equals the amount of information produced by the chaotic stimulus.

The role of a time-dependent stimulus in an active network

The most general way of modelling the action of an arbitrary stimulus perturbing an active network is by stimulating it using uncorrelated white noise. Let us assume that we have a large network with all the channels operating in non-self-excitable fashion. We also assume that all the transversal eigenmodes of oscillations except one are stable, and therefore do not suffer the influence of the noise. Let us also assume that the noise is acting only on one structurally stable ( = far from bifurcation points) element, S_k. To calculate the upper bound of the MIR between the element S_k and another element S_l in the network, we assume that the action of the noise does not alter the value of λ¹. Then, the noise on the element S_k is propagated along the vibrational mode associated with the one unstable transversal direction, whose conditional exponent is λ². As a consequence, the action of the noise might only increase λ², while not affecting the negativeness of all the other exponents (λ^m, m>2), associated with stable transversal modes of oscillation. That means that the channels responsible for transmitting large amounts of information (associated with λ^m, with m large) will not be affected. So, for such types of noises, Eq. (4) of the autonomous network is an upper bound for the non-autonomous network.

Consider now a situation where the noise acts equally on all the elements of an active network. The mapping(9)was proposed as a way to understand such a case. In this mapping, we consider ρ≥0 and x_n, y_n∈[0,1], which can be accomplished by applying the mod(1) operation.

Note that the term that enters equally in all the maps has statistical properties of an uniformly distributed random noise. Calculating I_P for ρ = 0 (the noise-free map) we arrive at I_P≅2σ, for small σ, while the true MIR I_C≅2(σ−ρ). These results are confirmed by exact numerical calculation of the Lyapunov exponents of Eq. (9) as well as the calculation of the conditional exponents of the variational equations. So, this example suggests that Eq. (4) calculated for an autonomous non-perturbed network gives the upper bound for the mutual information rate in a non-autonomous network.

Self-excitability

In Ref. [11] self-excitability was defined in the following way. An active network formed by N elements, is said to be self-excitable if H_KS (N, σ)>H_KS (N, σ = 0), which means that the KS-entropy of the network increases as the coupling strength is increased. Thus, for non self-excitable systems, an increase in the coupling strength among the elements forming the network leads to a decrease in the KS-entropy of the network.

Here, we adopt also a more flexible definition, in terms of the properties of each communication channel. We define that a communication channel cⁱ behaves in a self-excitable fashion if λⁱ>λ¹. It behaves in a non-self-excitable fashion if λⁱ≤λ¹.

Mutual Information Rate (MIR), channel capacity, and network capacity

In this work, the rate with which information is exchanged between two elements of the network is calculated by different ways. Using the approaches of Refs. [10], [11], we can have an estimate of the real value of the MIR, and we refer to this estimate as I_C. Whenever we use Eq. (4) to calculate the upper bound for the MIR, we will refer to it as I_P. Finally, whenever we calculate the MIR through the symbolic encoding of the trajectory, we refer to it as I_S.

We define the channel capacity of a communication channel formed by two oscillation modes depending on whether the channel behaves in a self-excitable fashion or not. So, for the studied network, every communication channel possess two channel capacities, the self-excitable capacity and the non-self-excitable one. A channel cⁱ operates with its self-excitable capacity when is maximal, what happens at the parameter σ^(i+1)*. It operates with its non-self-excitable capacity when λⁱ⁺¹ = 0.

We also define the channel capacity in an average sense. In that case, the averaged channel capacity is given by the maximal value of the average value(10)

The network capacity of a network composed of N elements, C_N(N), is defined to be the maximum value of the Kolmogorov-Sinai (KS) entropy, H_KS, of the network. For chaotic networks, the KS-entropy, as shown by Pesin [29], is the sum of all the positive Lyapunov exponents. Notice that if I denotes the MIR then(11)

As shown in Ref. [11] and from the many examples treated here, C_N(N)∝N, and so, the network capacity grows linearly with the number of elements in an active network.

Understanding Eq. (4): Positiveness of the MIR for self-excitable channels in the (non-linear) HR network

To show that indeed should be positive in case of a self-excitable channel in the HR network, one can imagine that in Eq. (1) the coupling strength is arbitrarily small and that N = 2. At this situation, the Lyapunov exponent spectra obtained from Eq. (2) are a first-order perturbative version of the conditional exponents, and they appear organized by their strengths. One arrives at λ₁≅λ² and λ₂≅λ¹, which means that the largest Lyapunov exponent equals the transversal conditional exponent and the second largest Lyapunov exponent equals the conditional exponent associated with the synchronous manifold. Using similar arguments to the ones in Refs. [10], [11], [30], we have that the MIR is given by the largest Lyapunov exponent minus the second largest, and therefore, I_C = λ₁−λ₂, which can be put in terms of conditional exponents as I_P≤λ²−λ¹, or as represented in Eq. (4), I_P≤|λ¹−λ²|.

Understanding Eq. (4): The inequality in Eq. (4)

To explain the reason of the inequality in Eq. (4), consider the following two coupled maps:(12)with s = 1 and x_n, y_n∈[0,1]. For this mapping, the MIR can be written in terms of the Lyapunov Exponents [11], [31]. For two coupled systems, the MIR can be exactly calculated by I_C = λ₁−λ₂, since λ^∥ = λ₁ and λ^⊥ = λ₂, assuming that both λ₁ and λ₂ are positive. Calculating the conditional exponents numerically, we can show that I_P≥I_C, and thus I_P is an upper bound for the MIR. For more details on this inequality, see [12]

Evolutionary construction of a network

In our simulations, we have evolved networks of equal bidirectional couplings [32]. That means that the Laplacian in Eq. (1) is a symmetric matrix of dimension N with integer entries {0,1} for the off diagonal elements, and the diagonal elements equal to , with i≠j.

Finding the network topologies which maximize B in Eq. (7) is impractical even for moderately large N. Figuring out by “brute force” which Laplacian produces the desired eigenvalue spectra would require the inspection of a number of configurations. To overcome this difficulty, Ref. [20] proposed an evolutionary procedure in order to reconstruct the network in order to maximize some cost function. Their procedure has two main steps regarded as mutation and selection. The mutation steps correspond to a random modification of the pattern of connections. The selection steps consist in accepting or rejecting the mutated network, in accordance with the criterion of maximization of the cost function B, in Eq. (7).

We consider a random initial network configuration, with N elements, which produce an initial Laplacian G₀, whose eigenvalues produce a value B₀ for the cost function. We take at random one element of this network and delete all links connected to it. In the following, we choose randomly a new degree k to this element and connect this element (in a bidirectional way) to k other elements randomly chosen. This procedure generates a new network that possesses the Laplacian G′, whose eigenvalues produce a value B′. To decide if this mutation is accepted or not, we calculate Δε = B′−B₀. If Δε>0, the new network whose Laplacian is G′ is accepted. If, on the other hand, Δε<0, we still accept the new mutation, but with a probability p(Δε) = exp(−Δε/T). If a mutation is accepted then the network whose Laplacian is G₀ is replaced by the network whose Laplacian is G′.

The parameter T is a kind of “temperature” which controls the level of noise responsible for the mutations. It controls whether the evolution process converges or not. Usually, for high temperatures one expects the evolution never to converge, since new mutations that maximizes B are often not accepted. In our simulations, we have used T≅0.0005.

These steps are applied iteratively up to the point when |Δε| = 0 for about 10,000 steps, being that we consider an evolution time of the order of 1,000,000 steps. That means that the evolution process has converged after the elapse of some time to an equilibrium state. If for more than one network topology |Δε| = 0 for about 10,000 steps, we choose the network that has the larger B value.

This constraint avoids the task of finding the best network topology. However, we consider that a reasonably low number of mutations would recreate what usually happens in real networks.