Figures
Abstract
Metanorms is a mechanism proposed to promote cooperation in social dilemmas. Recent experimental results show that network structures that underlie social interactions influence the emergence of norms that promote cooperation. We generalize Axelrod's analysis of metanorms dynamics to interactions unfolding on networks through simulation and mathematical modeling. Network topology strongly influences the effectiveness of the metanorms mechanism in establishing cooperation. In particular, we find that average degree, clustering coefficient and the average number of triplets per node play key roles in sustaining or collapsing cooperation.
Citation: Galán JM, Łatek MM, Rizi SMM (2011) Axelrod's Metanorm Games on Networks. PLoS ONE 6(5): e20474. https://doi.org/10.1371/journal.pone.0020474
Editor: Matjaz Perc, University of Maribor, Slovenia
Received: March 31, 2011; Accepted: April 26, 2011; Published: May 31, 2011
Copyright: © 2011 Galán et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: JMG is supported by the Spanish MICINN projects CSD2010-00034 (CONSOLIDER-INGENIO 2010) and DPI2010-16920, and by the Junta de Castilla y León, references BU034A08 and GREX251-2009. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
A social dilemma is a situation where the interest of the individual conflicts with the preference of the collective [1]. Each person entangled in a social dilemma has rational arguments to follow a behavior that in the aggregate leads to unfavorable outcomes for the collective. Social dilemmas are found in diverse contexts. For example, economic social dilemmas include problems associated with the provision of public goods such as national security, public health and environmental protection, where individuals can make investments into a common pool to provide a costly, non-excludable asset that benefits all regardless of how much they contribute to creating it [2]–[5]. Such “collective action” problems [6]–[8] are not limited to human social behavior. Biology abounds with examples of social dilemmas. Foraging yeast cells secrete enzymes to lyse their environment, producing a valuable common good that can be used by other cells [9]; groups of meerkats take turns as sentinels and give eventual alarm calls to the group [10]. Given the relevance of a large number of situations that correspond to the definition of social dilemma, the scientific community has expended significant capital to model and propose solutions to social dilemmas.
In the most common formalization, social dilemmas are modeled as games in which players follow different strategies. Social dilemma games are characterized by the presence of at least one deficient equilibrium: a situation that is an equilibrium, so no player has incentives to change his behavior individually, but it is not Pareto optimal, because there exists at least another possible outcome that every player prefers to the current one. Often the strategy that is collectively preferred is considered cooperative; therefore cooperators provide a benefit to the group at some cost while defectors exploit the group by reaping the benefits without bearing the costs of cooperation.
Proposed methods of avoiding the generally undesirable outcomes of social dilemmas vary widely and frequently depend on context. Kollock [11] classifies these methods based on whether players are assumed egoist and whether they can change the rules of the game. His classification divides solutions to social dilemmas into motivational [12]–[16], strategic [17]–[20] or structural [2], [12], [21]–[32] (See Figure 1). In motivational solutions like moral persuasion, a player gives some weight to the results other players obtain. In strategic solutions such as reciprocity, conditional association and grim triggers, an egoistic player influences other players' behavior by expanding the range of strategies he considers. Neither solution requires coordinated or top-down modifications of the rules of the game. In structural solutions such as sanctions, central authority or privatization, the rules of the game are changed to solve the dilemma.
Solutions to social dilemmas can be classified as motivational, strategic or structural depending on whether players are assumed egoist and whether the rules of the game can be changed.
A mixed structural-strategic solution proposed to obtain collectively rational outcomes in social dilemmas is a sanction system in which each player can punish other players that deviate from cooperation. This type of self-imposed norm has the crucial advantage of giving the players the opportunity to sanction norm deviants selectively [33]. Behaviorally, “a norm exists in a given social setting to the extent that individuals usually act in a certain way and are often punished when seen not to be acting in this way” [34]. This notion of norms is based on social norm as opposed to legal norms, moral norms, private norms, habits or fads [35]. [36], [37] contain more extensive reviews of the meaning of social norms, and [38], [39] discuss the sociological and economic foundations of norms while [40] couches social norms in evolutionary game theory. Nevertheless, this mechanism to promote cooperation can be riddled with difficulties if punishment is costly. The punisher usually assumes the cost of promoting punishment or vigilance. This punishment cost instigates a second order “instrumental dilemma” in which players have incentives to not punish, hence causing the solution to the first “elemental dilemma” to collapse [41].
Second-order norm deviance has been studied experimentally [41]. For instance, in a controlled experiment subjects were given the option to learn of others' contributions to a public good before deciding to punish them. This mechanism mitigated the free-rider problem to some extent [3], but it created other problems like punishing high contributors [42]. A plethora of other ideas have been suggested as solutions. The threat of expulsion or ostracism seems to improve the cooperation in providing public goods [43]. Costly signaling, may result in advantageous alliances, since cooperation constitutes an honest signal of the member's quality as a coalition partner or competitor [44], [45]. Hypocritical cooperation, that is, defecting at the first level while urging others to cooperate through participation in the sanctioning system, creates more robust second-order cooperation [46]. Conformism as a psychological bias toward copying the majority can also help to stabilize cooperation [47]. group selection mechanisms, competition at two different levels, within groups and between groups [48], [49] or indirect reciprocity, the idea that good reputation will be rewarded by others, [50] have also proven to promote cooperative behavior.
The insufficiency of selective punishment as a condition to promote cooperation in social dilemmas prompted Axelrod to propose metanorms, that is, norms about how individuals follow other norms, as a mechanism to support collective cooperation in social dilemmas in evolutionary contexts [34]. Although controversial [51], [52], metanorms are touted as a mechanism for sustainable cooperative strategies in which players adhere to norms, punish defectors, and punish those who do not punish defectors [33], [38] . Mathematical analysis coupled with extensive simulation has shown that metanorms can induce both collectively and individually rational stable equilibria and that the efficiency of metanorms as a solution to social dilemmas depends on the structure of the payoff matrix. Incentives for not following norms can counterintuitively enhance the preservation of the cooperative solution; decay in punishment can cause the norm to collapse; and the details of the evolutionary algorithm, for example more explorative strategies denoted by higher levels of mutation noise, can help to preserve the norm [53]. These results suggest that metanorms as a solution to social dilemmas cannot be considered universal, because the context of the specific problem can influence its efficacy.
All theoretical research on metanorms conducted so far has assumed an evolutionary game played on a global interaction network where every player interacts with all other players. However, a more realistic view of social interactions entails embedding players in social networks that differ markedly from a completely connected interaction graph [54]. Network structures that underlie social interactions affect outcomes of such interactions; therefore shape solutions to social dilemmas. Some experimental research has sought to account for the effect of network structure on social dilemmas [55], [56]. Furthermore, experimental works on dynamics of norm enforcement and metanorms suggest that characteristics of social relations, especially interdependence, influence the emergence of norms significantly [57]–[61]. This finding implies that we may overlook relevant aspects of the problem by focusing on the direct consequences of sanctioning norm deviance without accounting for the properties of social relations over which norms and metanorms are defined. Consequently, understanding the role of network structures is essential for explaining norm enforcement.
Given that different topologies or structures of social networks can influence outcomes of social interactions [54], we adapt the metanorms game to arbitrary interaction networks and analyze the influence of network topology on the emergence of cooperation through mathematical analysis and computer simulation This hybrid methodological approach has proven to be useful in analyzing complex social models [62], [63] and extends the growing literature on games on networks [64]–[77] that is currently evolving from stylized network structures to more general interaction topologies.
The paper is organized as follows. First, we extend the metanorms game to play on networks. We then examine the dynamics and stability of a simplified version of the metanorms game mathematically. Next, we present simulation results to confirm some of the conclusions obtained analytically. Finally we present the conclusions of the work.
Methods
Metanorms Games on Networks
We set up the metanorms game on networks by embedding 50 agents on a network developed by a network generation algorithm. We use 50 agents instead of 20 in Axelrod's default setting to make higher-order network statistics more interpretable. We used the Barabási-Albert algorithm to generate networks with discrete Pareto degree distributions [78], the Watts algorithm [79] with different values of rewiring probability (β) that smoothly interpolates between extreme cases of a regular lattice and a random network, traversing “small world” networks [80] along the way, and the Erdös-Rényi random networks [55]. A link between two agents represents an opportunity for direct interaction between them. A set of all direct links to an agent is the neighborhood of the agent. To explore the effect of clustering in the networks more clearly, we have also considered agents with a distance or radius of two where radius is defined as the minimum number of edges that it takes to link one agent to another (See Figure 2).
Once agents are embedded on the underlying network structure, they play a repeated game that consists of three decisions or stages:
- Agents decide whether to cooperate or defect. A defecting agent obtains Temptation payoff (T = 3) and inflicts on each of the remaining agents in the population Hurt payoff (H = −1). If agents cooperate, no one's payoff is altered. Here we assume that the spillover cost of defection is global.
- Agents observe other agents in their neighborhood who defected in stage 1 with a certain probability. For each observed defection, agents decide whether to punish the defector or not. Punishment is costly: one must pay Enforcement cost (E = −2) to impose Punishment cost (P = −9) on the defector. The opportunity to observe defection, and hence the possibility to punish it, is conditional on the existence of a link connecting defectors and punishers.
- The third step includes the concept of metanorms: agents who fail to punish observed defection should be punished. Similar to the previous step, an agent who fails to punish an observed defection may not be caught. The probability of being seen not punishing a defecting agent given that defection is observed is the same as the probability of observing such defection. Network topology plays a critical role in this step: it determines who can see unpunished defection. Observing a defection requires links among the defector, un-punishing agent and metapunisher. A metapunisher pays Meta-enforcement cost (ME = −2) to meta-punish (MP = −9) an agent who decided not to punish a defector.
Six sample networks with N = 50 and = 2. For Watts' small world network, rewiring probability β was set to 0.2. Subfigures A—C on the upper panel represent networks with radius 1. Subfigures D—E on the lower panel have neighborhoods expanded to radius 2.
Parameters boldness and vengefulness characterize an agent's strategy. Boldness is an agent's propensity to defect, and determines the outcome of the first stage of the game. An agent that can defect will defect, if its boldness is greater than a random probability of being observed. Vengefulness is an agent's propensity to punish agents that it has observed defecting in the second stage of the game and to meta-punish agents that it has observed not punishing a defecting agent in the third. An agent punishes observed defectors or observed un-punishers with a probability equal to vengefulness. Following the original implementation by Axelrod, boldness and vengefulness are set as 3-bit strings denoting eight evenly distributed values from 0 to 1 (0/7, 1/7, …,7/7). Initial values of agents' boldness and vengefulness are determined randomly at the beginning of each simulation run and updated by an evolutionary mechanism.
The game is played four rounds called a generation. At the beginning of each generation, agents' payoffs are set to zero; at the end of a generation all payoffs for each round are accumulated and computed for each agent, and agents can change their strategies according to evolutionary forces of selection and mutation. We have adapted to local network structures a variant of selection mechanisms called roulette wheel in which the most successful agents in a given generation are the most likely to spread [81]: an agent picks a strategy played by other agents in its neighborhood with probability proportional to the other agents' fitness where an agent's fitness is equal to the difference between its payoff and the minimum payoff obtained in the neighborhood. Whenever an agent replicates a bitstring by invoking the selection mechanism, every bit of the bitstring has a certain probability of being flipped from 0 to 1 and vice versa called mutation rate. The game continues with a new generation playing with new agent strategies.
Results
Mathematical analysis
Given a specific network structure, the state of the game is a certain realization of agents' joint strategies, so the number of possible game states is 6450 corresponding to 64 strategies that any of the 50 players may choose. For any positive mutation rate, the model is an irreducible positive recurrent and ergodic discrete-time finite Markov chain [63], since the mutation operator guarantees the non-zero probability of transition from any state to any other state in one single step. This observation means that in the long run, the probability of finding the metanorms game in any of its states is non-zero and independent of the initial conditions of the game. This result guides our simulation experiments, because it guarantees that if we run simulations for long enough the limiting distribution approximates to the occupancy distribution.
The size of the state space of the game makes calculating the transition matrix of the Markov chain infeasible. We have to resort to other strategies to gain insights from a mathematical analysis of the model. In this section we propose a simpler mathematical abstraction of the metanorms game that is amenable to mathematical analysis and graphic visualization. This abstracted model suggests areas of stability and basins of attraction in the model and illustrates the expected dynamics of the metanorms graphically. We should stress that the conclusions of this analysis come from the simplified model, not the original one, so they must be verified by simulation.
Let's begin formalizing the model. Assume an undirected network defined by a set of agents
as nodes and a set of unweighted links among them
. The payoff of agent i playing the metanorms game is defined by:
(1)
where T, H, E, P, ME, MP are the payoffs of the model, n is the number of agents, and is the set of agents linked to any given
. This set defines the neighborhood of i.
denotes the number of neighbors or degrees for agent i. Two indicator functions are also used:
and vengefulness and boldness for each agent is denoted as vi and bi. Using vengefulness and boldness we calculate the expected payoff of agent i in one round as:(2)
Eq. 2 depends on the exact realization of the network topology and exact strategies of each agent in the network. Let us now rewrite the Eq. 2 in terms of common statistics of the network topology.
First, let be the first-order degree distribution of network
. The clustering of an agent i with at least two neighbors is defined as:
We define clustering coefficient for a degree in a given network as:
Assuming homogeneity in vengefulness and boldness in the population, we can simplify the expected payoff of agent i as follows:
(3)
Eq. 3 expresses expected payoffs of homogeneous agents on a given network as a function of the first degree distribution, clustering distribution and the average degree of the network.
represents the average number of triplets per agent in network
. We call this statistic interconnectedness of network
. In other words, the dynamics and expected outcomes of the metanorms game may be highly influenced by the agents' average number of interactions and a certain measure of clustering of these interactions.
To characterize the long run outcomes of the game, we use the concept of evolutionary stable state (ESS) to identify the stability points of the game. This notion is inspired by the ideas proposed by Maynard Smith and Price [82] and developed by Weibull [83] and Colman [84]. An ESS in the metanorms game [53] is a state where:
- Every agent in the population Θ receives the same expected payoff, so evolutionary selection pressures will not lead the system away from the state,
- Any agent m that changes its strategy with bm as its new boldness and vm its new vengefulness, receives a strictly lower expected payoff than any other agent in the incumbent population I ≡ Θ-{m}, so if a single mutation occurs, the mutant agent will not be able to invade the population:
- Once a single agent m has changed its strategy, all other agents in the incumbent population I receive the same expected payoff, so a single mutant cannot distort the composition of the population except maybe by random drift:
These three conditions above are enough to expect that any mutant will be removed from the game, providing strong restriction for stability in the dynamics of the model. If the system is not homogeneous, these conditions are not sufficient to guarantee in general that, if they are fulfilled in a certain state, the system will tend to revert to such a state after a single mutation. If the three conditions prevail in a certain state, we expect any mutant to be removed from the game, but the specific strategy among the incumbent population that will replace the mutant depends on the selection mechanism.
At this point we can establish two necessary conditions for a state to be evolutionary stable by assuming continuity in agent properties in Eq. 3. Let m be an arbitrary, but potentially mutant, agent with bm as its boldness and vm as its vengefulness in a given population of agents Θ. Let I be the set of incumbent agents in the population Θ excluding m. The following equation is a necessary condition for the population of agents to be in ESS. This condition can be easily grasped by realizing that if every agent has the same expected payoff as the necessary condition for ESS, and Eq. 4 does not hold for some agents m and i, the potentially mutant m can get a differential advantage over incumbent i by changing its boldness bm, meaning that the state under study cannot be evolutionary stable:
Similarly, we can obtain another necessary condition by substituting vm for bm in Eq. 4.(5)
We can use Eq. 3 to evaluate the necessary derivatives:
Generalizing the demonstration provided in [53], it can be proved that the system may have two different ESS, one where the norm collapses (bi = 1, vi = 0 for all i) and eventually another where the norm is established. This last ESS only appears depending on the relation between the average degree and the average number of triplets by agent, features that are determined by the network topology of the game.
Evaluating gradients from Eq. 6 for any network topology and population characteristics leads to gradient maps of predicted population movements. The legend for these maps is described on Figure 3. For any constant value of average degree, the theoretical analysis suggests that the higher the average number of triplets, the more likely a cooperative ESS is to emerge and the bigger the size of its basin of attraction. On the contrary, for a constant average number of triplets, the higher average degree, the less likely a cooperative ESS is to emerge. Moreover, we can numerically calculate the minimum average number of triplets for each average degree in order to have an ESS in the area of norm establishment and compare it with different network topologies (Figure 4).
The map applies to both analytic and simulated gradient landscapes. The axes represent the average boldness and vengefulness of the population as its strategic characteristics. For each point, we measure the direction and speed of population drift. For analytical landscapes, we will also pinpoint the expected location of the evolutionary stable states. For simulation landscapes, we will be measuring the time that the simulation spends in each of the two key regions: norm emergence and norm collapse zones. Sample maps for different network topologies are shown in Figure 7.
Minimal interconnectedness necessary for a cooperative evolutionary stable state to exist in the simplified analytical model for any given average degree of the network, compared to the expected interconnectedness of different network topologies with radius 1. Default metanorms parameters are assumed.
Simulation
The results derived from the previous section are suggestive but we should keep in mind that they could have been obtained as consequences of simplifying assumptions not directly from the model explained in section two, since we abstracted the evolutionary mechanisms and the details of network topology, imposed continuity on agent properties, and worked only in terms of expected behavior. We need to verify if the suggested hypotheses in the equation-based approach can be generalized to the original metanorms game. Since that model, and especially when it is played on networks, is very complex, we have to resort to extensive simulation to gain insights on how it evolves. All simulations can be replicated with the source code of the model provided at http://josema.galan.name/models.
In the experiment designed to verify the behavior of the metanorms mechanism, we use the same payoff matrix, mutation rate and number of rounds per generation as in Axelrod's original paper (see Table 1).
We used the network generation algorithms mentioned in the previous section to create a sample of 6000 networks. The density of the sampled networks is plotted in Figure 5, projected onto the average degree against clustering coefficient of the network and the square root of the interconnectedness spaces. Note that clustering, interconnectedness and the average degree are not independent variables.
The distribution of the sampled network topologies, projected onto two dimensional views of key network statistics. The left panel shows sampled network topologies projected onto the square root of interconnectedness and the average degree of the network, while the right panel describes the sample density in the average degree and the clustering coefficient space.
The results obtained from simulations allow us to perform two types of analyses, an analysis of the long-run limiting behavior of metanorms, and an analysis of the dynamics to compare with the expected dynamics predicted by the simplified theoretical model.
To study the influence of the network topology on the long-run behavior of the model, recall that when mutation rate is greater than zero, the metanorms game on networks is a time-homogeneous Markov chain (THMC) in which the limiting distribution coincides with the occupancy distribution as the long-run fraction of the time that the THMC spends in each state. Therefore, we can approximate the limiting distribution by computing the frequency of simulation in each state. We have defined the following zones:
- Norm Collapse: the simulation is in states where the average boldness is at least 6/7 and the average vengefulness is no more than 1/7.
- Norm Establishment: the simulation is in states where average boldness is no more than 1/7 and average vengefulness is at least 6/7.
We have computed the time that a single simulation is in either zone. In Figure 6 we have measured the long-run fraction of time, averaged over runs and over network topologies that the simulation is in cooperative norm emergence and collapse zones as a function of average degree and root square of interconnectedness and as function of average degree and clustering coefficient of the network. First, these results suggest that the ESS obtained analytically in the simplified model are in fact the only ESS in the system since the time that the simulation is out of these two zones is not significant.
Proportion of time that the simulation spends in the norm collapse and emergence zones as a function of key network statistics using similar projections as those in Figure 5. Color codes the fraction of simulation time spent in each zone computed for each bin. Time spent outside either zone is insignificant.
Second, we observe that on average the influence of average degree, the number of triplets and clustering coefficient behave as predicted analytically. The higher the average number of triplets the more time the simulation spends in norm establishment zone; on the contrary, for a given average number of triplets, the higher average degree of the network, the lower the probability of finding the simulation in the norm establishment zone. These results suggest that an important part of the limiting behavior of the game can be explained by two simple statistics of the network topology.
Last, we have found some variance in the results. For example, we do find the norm establishment in sparser networks that are analytically predicted not to reach the norm establishment zone. This indicates that agent and network heterogeneity, for example local “clumps” denser than the whole network by chance, and the specifics of the evolutionary mechanism also play important roles in norm establishment as they may allow for “seeding” the cooperative norm in the network [67], [85], [86].
We have also analyzed data from simulation to determine the match between gradient maps obtained in the simplified mathematical analysis and simulation model dynamics. The first column of Figure 7 represents the mathematically predicted dynamics, whereas the second column presents simulated speed and direction of population drift. We observe that in terms of speed and direction of trajectories the predicted dynamics match, especially for those cases where there is only one clear ESS or two ESSs. The interesting result appears when there is only one predicted ESS but we are close to having another in the zone of norm establishment. The observed dynamics show possible quasi-stable states in norm emergence earlier than predicted by the equation approach. This effect may be the consequence of mutation rate and variance of selection mechanism that make it difficult to escape from the norm establishment zone [53].
For simulated results, the mutation rate was set to 0.01. Color codes the speeds of movement of the population, either computed analytically or measured from the simulation with blue being the slowest and red the fastest. Figure 3 contains the legend for the graphs. In panel B, simulated population spends 95% of time in norm collapse zone. In panel D, the proportion of time in the norm collapse zone drops to 50%. In panel E, the simulation spends 95% of time in the norm emergence zone and 5% in the norm collapse zone.
Discussion
We have adapted the theoretical model of metanorms to guide agent interactions on static networks. Our analytical and computational results show that the interaction structure influences the effectiveness of the metanorms mechanism. In particular, we identified the average degree, clustering coefficient and interconnectedness as the average number of triplets per agent as key aspects that contribute to sustaining or collapsing norms of cooperation in networked populations. Higher clustering coefficient and average number of triplets increases cooperative behavior, suggesting that translating bilateral to trilateral interaction promotes cooperation. Comparing the results of our simplified mathematical analysis with those of computational modeling, we have also shown that some evolutionary details influence in the model dynamics that stabilize the zone of norm establishment.
The networks used for the analysis have been numerous and diverse, nevertheless we have not analyzed all possible configurations. Some recent studies [67], [85]–[87] have proved that community structure [88], subsets of nodes that are relatively densely connected to each other but sparsely connected to other dense groups, can be also an important parameter in the behavior of games in networks, although the network generators used in our analysis are not particularly designed to take into account this effect. Further research may clarify the effect of more complex topologies, particular evolutionary details or scale on the effectiveness of metanorms as mechanism to sustain cooperation in social dilemmas.
Acknowledgments
The authors wish to thank Rob Axtell, Luis Izquierdo, Sean Luke and other members of the Department of Computational Social Science at George Mason University and the INSISOC Group, for their help and for many extremely useful discussions.
Author Contributions
Conceived and designed the experiments: JMG MML SMR. Performed the experiments: JMG MML SMR. Analyzed the data: JMG MML SMR. Contributed reagents/materials/analysis tools: JMG MML SMR. Wrote the paper: JMG MML SMR.
References
- 1. Dawes RM, Messick DM (2000) Social Dilemmas. International Journal of Psychology 35(2): 111–116.
- 2. Fehr E, Gächter S (2002) Altruistic punishment in humans. Nature 415(6868): 137–140.
- 3. Fehr E, Gächter S (2000) Cooperation and punishment in public goods experiments. American Economic Review 90(4): 980–994.
- 4.
Cornes R, Sandler T (1986) The theory of externalities, public goods and club goods. Cambridge: Cambridge University Press.
- 5. Samuelson PA (1954) The Pure Theory of Public Expenditure. Review of Economics and Statistics 36(4): 387–389.
- 6.
Olson M (1965) The logic of collective action. Public goods and the theory of groups. Cambridge, Mass: Harvard University Press.
- 7. Ostrom E (1998) A behavioral approach to the rational choice theory of collective action presidential address, American political science association, 1997. American Political Science Review 92(1): 1–22.
- 8. Heckathorn DD (1998) Collective action, social dilemmas and ideology. Rationality and Society 10(4): 451–479.
- 9.
Greig D, Travisano M (2004) The Prisoner's Dilemma and polymorphism in yeast SUC genes. Biological Sciences B:. 271. Proceedings of the Royal Society. https://doi.org/10.1098/rsbl.2003.0083
- 10. Clutton-Brock TH, O'Riain MJ, Brotherton PNM, Gaynor D, Kansky R, et al. (1999) Selfish sentinels in cooperative mammals. Science 284(5420): 1640–1644.
- 11. Kollock P (1998) Social dilemmas: The anatomy of cooperation. Annual Review of Sociology 24: 183–214.
- 12. Brewer MB, Kramer RM (1986) Choice Behavior in Social Dilemmas. Effects of Social Identity, Group Size, and Decision Framing. Journal of Personality and Social Psychology 50(3): 543–549.
- 13. Kramer RM, Brewer MB (1984) Effects of group identity on resource use in a simulated commons dilemma. Journal of Personality and Social Psychology 46(5): 1044–1057.
- 14. Ostrom E, Walker JM, Gardner R (1992) Covenants with and without a Sword:Self-governance Is Possible. American Political Science Review 86: 404–417.
- 15. Bogaert S, Boone C, Declerck C (2008) Social value orientation and cooperation in social dilemmas: A review and conceptual model. British Journal of Social Psychology 47(3): 453–480.
- 16. Bicchieri C (2002) Covenants without swords: Group identity, norms, and communication in social dilemmas. Rationality and Society 14(2): 192–228.
- 17.
Izquierdo SS, Izquierdo LR, Gotts NM (2008) Reinforcement learning dynamics in social dilemmas. Journal of Artificial Societies and Social Simulation 11(2): 1. Available: http://jasss.soc.surrey.ac.uk/11/2/1.html.
- 18. Izquierdo SS, Izquierdo LR, Vega-Redondo F (2010) The option to leave: Conditional dissociation in the evolution of cooperation. Journal of Theoretical Biology 267(1): 76–84.
- 19.
Axelrod RM (1984) The Evolution of Cooperation. New York: Basic Books.
- 20. Carpenter JP, Matthews PH, Ong'ong'a O (2004) Why punish? Social reciprocity and the enforcement of prosocial norms. Journal of Evolutionary Economics 14(4): 407–429.
- 21. Rand DG, Dreber A, Ellingsen T, Fudenberg D, Nowak MA (2009) Positive Interactions Promote Public Cooperation. Science 325(5945): 1272–1275.
- 22. Szolnoki A, Perc M (2010) Reward and cooperation in the spatial public goods game. Europhysics Letters 92(3): 38003.
- 23. Rapoport A, Bornstein G, Erev I (1989) Intergroup Competition for Public Goods: Effects of Unequal Resources and Relative Group Size. Journal of Personality and Social Psychology 56(5): 748–756.
- 24.
Isaac RM, Walker JM (1988) Group Size Effects in Public Goods Provision: The Voluntary Contributions Mechanism. The Quarterly Journal of Economics 103(1): 179-199. Available: http://www.jstor.org/stable/1882648.
- 25. Denant-Boemont L, Masclet D, Noussair CN (2007) Punishment, counterpunishment and sanction enforcement in a social dilemma experiment. Economic Theory 33(1): 145–167.
- 26. Helbing D, Szolnoki A, Perc M, Szabo G (2010) Punish, but not too hard: How costly punishment spreads in the spatial public goods game. New Journal of Physics 12:
- 27. Helbing D, Szolnoki A, Perc M, Szabo G (2010) Evolutionary establishment of moral and double moral standards through spatial interactions. PLoS Computational Biology 6(4).
- 28. Helbing D, Szolnoki A, Perc M, Szabo G (2010) Defector-accelerated cooperativeness and punishment in public goods games with mutations. Physical Review E 81(5): 057104.
- 29. Szolnoki A, Szabo G, Perc M (2011) Phase diagrams for the spatial public goods game with pool punishment. Physical Review E 83(3): 036101.
- 30. Riolo RL, Cohen MD, Axelrod R (2001) Evolution of cooperation without reciprocity. Nature 414(6862): 441–443.
- 31. Gürerk Ö, Irlenbusch B, Rockenbach B (2006) The competitive advantage of sanctioning institutions. Science 312(5770): 108–111.
- 32. Hardin G (1968) The Tragedy of the Commons. Science 162(3859): 1243–1248.
- 33. Boyd R, Richerson PJ (1992) Punishment Allows the Evolution of Cooperation (or Anything Else) in Sizable Groups. Ethology and Sociobiology 13(3): 171–195.
- 34. Axelrod RM (1986) An Evolutionary Approach to Norms. American Political Science Review 80(4): 1095–1111.
- 35. Elster J (1989) Social norms and economic theory. Journal of Economic Perspectives 3(4): 99–117.
- 36.
Hechter M, Opp KD (2001) Social norms. New York: Russell Sage Foundation.
- 37. Helbing D, Johansson A (2010) Cooperation, Norms, and Revolutions: A Unified Game-Theoretical Approach. PLoS ONE 5(10): e12530.
- 38.
Coleman JS (1990) Foundations of social theory. Cambridge, Massachusetts: Belknap Press of Harvard University Press.
- 39.
Young HP (2008) Social Norms. In: Durlauf SN, Blume LE, editors. New Palgrave Dictionary of Economics. London: Macmillan.
- 40. Bendor J, Swistak P (2001) The evolution of norms. American Journal of Sociology 10(6): 1493–1545.
- 41. Yamagishi T (1986) The Provision of a Sanctioning System as a Public Good. Journal of Personality and Social Psychology 51(1): 110–116.
- 42. Cinyabuguma M, Page T, Putterman L (2006) Can second-order punishment deter perverse punishment? Experimental Economics 9(3): 265–279.
- 43. Cinyabuguma M, Page T, Putterman L (2005) Cooperation under the threat of expulsion in a public goods experiment. Journal of Public Economics 89(8): 1421–1435.
- 44. Smith EA, Bliege Bird RL (2000) Turtle hunting and tombstone opening: Public generosity as costly signaling. Evolution and Human Behavior 21(4): 245–261.
- 45. Gintis H, Smith EA, Bowles S (2001) Costly signaling and cooperation. Journal of Theoretical Biology 213(1): 103–119.
- 46. Heckathorn DD (1989) Collective Action and the Second-Order Free-Rider Problem. Rationality and Society 1(1): 78–100.
- 47. Henrich J, Boyd R (2001) Why people punish defectors: Weak conformist transmission can stabilize costly enforcement of norms in cooperative dilemmas. Journal of Theoretical Biology 208(1): 79–89.
- 48. Boyd R, Gintis H, Bowles S, Richerson PJ (2003) The evolution of altruistic punishment. Proceedings of the National Academy of Sciences of the United States of America 100(6): 3531–3535.
- 49. Gintis H (2000) Strong reciprocity and human sociality. Journal of Theoretical Biology 206(2): 169–179.
- 50. Panchanathan K, Boyd R (2004) Indirect reciprocity can stabilize cooperation without the second-order free rider problem. Nature 432(7016): 499–502.
- 51.
Elster J (2006) Altruistic behavior and altruistic motivations. In: Kolm S-G, Ythier J, editors. Handbook of the economics of giving, altruism, and reciprocity. Amsterdam: Elsevier. pp. 183–223.
- 52. Linares Martínez F (2007) El problema de la emergencia de normas sociales en la acción colectiva: una aproximación analítica. Revista Internacional de Sociología 65(46): 131–160.
- 53.
Galan JM, Izquierdo LR (2005) Appearances Can Be Deceiving: Lessons Learned Re-Implementing Axelrod's ‘Evolutionary Approach to Norms’. Journal of Artificial Societies and Social Simulation 8(3): 2. Available: http://jasss.soc.surrey.ac.uk/8/3/2.html.
- 54. Newman MEJ (2003) The structure and function of complex networks. SIAM Review 45(2): 167–256.
- 55. Opp KD (2002) When do norms emerge by human design and when by the unintended consequences of human action? The example of the no-smoking norm. Rationality and Society 14(2): 131–158.
- 56. Grujic J, Fosco C, Araujo L, Cuesta JA, Sánchez A (2010) Social Experiments in the Mesoscale: Humans Playing a Spatial Prisoner's Dilemma. PLoS ONE 5(11): e13749.
- 57. Horne C (2007) Explaining norm enforcement. Rationality and Society 19(2): 139–170.
- 58. Horne C (2004) Collective Benefits, Exchange Interests, and Norm Enforcement. Social Forces 82(3): 1037–1062.
- 59. Horne C, Cutlip A (2002) Sanctioning costs and norm enforcement: An experimental test. Rationality and Society 14(3): 285–307.
- 60. Horne C (2001) The enforcement of norms: Group cohesion and Meta-norms. Social Psychology Quarterly 64(3): 253–266.
- 61. Horne C (2008) Norm enforcement in heterogeneous groups: Sanctioning by majorities and isolated minorities. Rationality and Society 20(2): 147–172.
- 62.
Galán JM, Izquierdo LR, Izquierdo SS, Santos JI, del Olmo R, et al. (2009) Errors and artefacts in agent-based modelling. Journal of Artificial Societies and Social Simulation 12(1):1. Available: http://jasss.soc.surrey.ac.uk/12/1/1.html.
- 63.
Izquierdo LR, Izquierdo SS, Galán JM, Santos JI (2009) Techniques to understand computer simulations: Markov chain analysis . Journal of Artificial Societies and Social Simulation 12(1): 6. Available: http://jasss.soc.surrey.ac.uk/12/1/6.html.
- 64. Szabo G, Fath G (2007) Evolutionary games on graphs. Physics Reports 446(4-6): 97–216.
- 65.
Hauert C (2006) Spatial effects in social dilemmas. Journal of Theoretical Biology. 240(4). : 627–636. https://doi.org/10.1016/j.jtbi.2005.10.024 |
- 66. Ohtsuki H, Hauert C, Lieberman E, Nowak MA (2006) A simple rule for the evolution of cooperation on graphs and social networks. Nature 441(7092): 502–505.
- 67. Lozano S, Arenas A, Sánchez A (2008) Mesoscopic structure conditions the emergence of cooperation on social networks. PLoS ONE 3(4): e1892.
- 68. Wu B, Zhou D, Fu F, Luo Q, Wang L, et al. (2010) Evolution of cooperation on stochastic dynamical networks. PLoS ONE 5(6): e11187.
- 69. Perc M, Szolnoki A (2010) Coevolutionary games-A mini review. BioSystems 99(2): 109–125.
- 70. Poncela J, Gómez-Gardeñes J, Floría LM, Moreno Y (2007) Robustness of cooperation in the evolutionary prisoner's dilemma on complex networks. New Journal of Physics 9(6): 184.
- 71. Perc M (2009) Evolution of cooperation on scale-free networks subject to error and attack. New Journal of Physics 11:
- 72. Szolnoki A, Perc M (2009) Resolving social dilemmas on evolving random networks. Europhysics Letters 86(3).
- 73. Szolnoki A, Perc M (2009) Emergence of multilevel selection in the prisoner's dilemma game on coevolving random networks. New Journal of Physics 11:
- 74. Szolnoki A, Perc M, Danku Z (2008) Towards effective payoffs in the prisoner's dilemma game on scale-free networks. Physica A: Statistical Mechanics and its Applications 387(8-9): 2075–2082.
- 75. Szolnoki A, Perc M, Danku Z (2008) Making new connections towards cooperation in the prisoner's dilemma game. Europhysics Letters 84(5): 50007.
- 76. Roca CP, Lozano S, Arenas A, Sánchez A (2010) Topological traps control flow on real networks: The case of coordination failures. PLoS ONE 5(12): e15210.
- 77. Gómez-Gardeñes J, Romance M, Criado R, Vilone D, Sánchez A (2011) Evolutionary games defined at the network mesoscale: The Public Goods game. Chaos 21(1): 016113.
- 78. Barabasi AL, Albert R (1999) Emergence of scaling in random networks. Science 286(5439): 509–512.
- 79.
Watts DJ (1999) Small worlds: the dynamics of networks between order and randomness. In: Princeton , editor. N.J: Princeton University Press.
- 80. Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small-world’ networks. Nature 393(6684): 440–442.
- 81.
De Jong KA (2006) Evolutionary computation. A unified approach. Cambridge, Mass: MIT Press.
- 82. Maynard Smith J, Price G (1973) The Logic of Animal Conflict. Nature 246(2): 15–18.
- 83.
Weibull JW (1995) Evolutionary Game Theory. Cambridge, MA: MIT Press.
- 84.
Colman AM (1995) Game Theory and its Applications in the Social and Biological Sciences. Oxford, UK: Butterworth-Heinemann.
- 85. Lozano S, Arenas A, Sánchez A (2008) Community connectivity and heterogeneity: Clues and insights on cooperation on social networks. Journal of Economic Interaction and Coordination 3(2): 183–199.
- 86. Poza DJ, Santos JI, Galán JM, López-Paredes A (2011) Mesoscopic Effects in an Agent-Based Bargaining Model in Regular Lattices. PLoS ONE 6(3): e17661.
- 87. Perc M, Szolnoki A, Szabo G (2008) Restricted connections among distinguished players support cooperation. Physical Review E 78(6): 066101.
- 88. Lancichinetti A, Saramäku J, Kivelä M, Fortunato S (2010) Characterizing the community structure of complex networks. PLoS ONE 5(8): e11976.