Darwin believed that there are two kinds of biological selection, natural selection and sexual selection. Natural selection is for traits that confer success the individual's battle with predators and search for prey. Sexual selection is for traits that increase an individual's chance for successful mating. Darwin was particularly struck by the fact that some traits that clearly have fitness costs in the natural selection setting seem to have fitness benefits in the sexual selection setting. For instance, the peacock's gorgeous and abundant plumage is costly to produce and renders the bird vulnerable to predators but desirable to peahens.
The issue of sexual selection was especially important to Darwin because natural selection alone appears unable to explain central characteristics of human beings. For instance, hunter-gatherer tribes rarely have languages in which counting uses numbers beyond one, two, and three, yet humans are capable of prodigious mathematical feats. More generally, of what possible survival value is there in a brain that is extremely costly to maintain, much of whose power is used in singing, dancing, creating art and music, and in other tasks far removed from the daily grind of self-preservation? Lord Russel Wallace chided Darwin at length for maintaining that humans are subject to the same biological laws as other species, and urged him to recognize the intervention of a Divine hand in the creation of our species. Darwin used sexual selection to answer Lord Wallace, saying "He who admits the principle of sexual selection will be led to the remarkable conclusion that the nervous system not only regulates most of the existing function of the body, but has indirectly influenced the progressive development of various bodily structures and of certain mental qualities. Courage, pugnacity, perseverance strength and size of body, weapons of all kinds, musical organs, both vocal and instrumental, bright colours and ornamental appendages, have all be indirectly gained by the one sex or the other, through the exertion of choice, the influence of love and jealousy, and the appreciation of the beautiful in sound, colour or form, and these powers of mind manifestly depend on the development of the brain." Darwin (1879, p. 687.)
Concerning singing and language, Darwin was even more insistent on the importance of sexual selection: "some early progenitor of man probably first used his voice in producing true musical cadences, that is singing, as do some of the gibbon-apes at the present day; and we may conclude from a widely-spread analogy, that this power would have been especially exerted during the courtship of the sexes." Darwin (1879, p. 109)
The mechanism for sexual selection for male traits was outlined by Darwin as follows. A female mutant prefers a certain male ornament, and preferentially mates with males with that ornament. Her female offspring inherit her taste for the ornament, and her male offspring inherit the ornament. Thus this preferential mating scheme is perpetrated in to the next, and all succeeding, generations. Called "runaway selection," this dynamic was analytically specified by the great Ronald Fisher, and there are currently several models of this type in the literature. This is the model of sexual selection, characterized by coy females who instigate intense competition of males for her favors, the males having no role in the sustenance of their offspring and hence of no value except reproductive value, that Roughgarden says is wrong. It its place, Roughgarden proposes a dynamic of "social selection," in which males and females cooperate harmoniously in nurturing and raising their young. Her great example is nesting birds, who work furiously and jointly to tend to their young. Males and females in nesting species both care about the quality of their partners, including not only their ability to compete in reproducing, but also foraging, defending, and provisioning a nest.
Joan Roughgarden is a very famous and accomplished biologist. In recent years, however, she has turned to a virulent form of ideological conflict-aversion that greatly interferes with her ability to get her (important) message across. It is typical of scientists to build on the successes of previous generations (Newton claimed that "if I have seen further, it is by standing on the shoulders of giants"). Roughgarden will have nothing of this. Traditional population biology is incurably infected with patriarchal bias, and must be swept away. The truth for Roughgarden is simply the opposite of the traditional theory; whence the title of the book, in contrast with Richard Dawkins' famous The Selfish Gene. In fact, both conflict and cooperation, as well as selfish and altruistic behavior, are inextricably intertwined aspects of the behavior of many vertebrate species, and the contemporary task is to augment and not simply deny traditional population biology.
Social selection is, as Roughgarden asserts, the more general category of which sexual selection is a subcategory. What is social selection (this is my story---Roughgarden may consider me an enemy, for all I know, but mine is the true story, I assert)? Every sexually reproducing species in which members interact in frequent and routine ways develop standardized patterns of interaction that involved both conflict and cooperation. These patterns are built into their genes and take the form of standardized structures and standardized behaviors. Structural standardization is called "niche-construction," and accounts for the beaver dam, the termite mound, the beehive and so on. See, for instance, Kevin N. Laland, F. John Odling-Smee and Marcus W. Feldman, "Evolutionary Consequences of Niche Construction and Their Implications for Ecology", Proceedings of the National Academy of Sciences 96 (1999):10242-10247; Kevin N. Laland, F. John Odling-Smee and Marcus W. Feldman, "Group Selection: A Niche Construction Perspective", Journal of Consciousness Studies 7,1/2 (2000):221-224; Kevin N. Laland and Marcus W. Feldman, Niche Construction (Princeton: Princeton University Press, 2004); F. John Odling-Smee, Kevin N. Laland and Marcus W. Feldman, Niche Construction: The Neglected Process in Evolution (Princeton: Princeton University Press, 2003). The importance of standardized structures is that these structures then become the environment within which the species evolves. This process is perhaps most dramatic in humans, for whom production techniques and their related cultural forms become the basis for further genetic evolution. This is the so-called gene-culture coevolution. See, for instance, Robin M. Dunbar, "Coevolution of Neocortical Size, Group Size and Language in Humans", Behavioral and Brain Sciences 16,4 (1993):681-735; William H. Durham, Coevolution: Genes, Culture, and Human Diversity (Stanford: Stanford University Press, 1991); Herbert Gintis, "Gene-culture Coevolution and the Nature of Human Sociality", Proceedings of the Royal Society B 366 (2011):878-888; Robert Boyd and Peter J. Richerson, Culture and the Evolutionary Process (Chicago: University of Chicago Press, 1985).
The social behavior side of social selection is much less well developed that the social institutional side, but it includes the typical pattern of male-female mating and care of offspring, as well as the pattern of mate characteristics typical of males and females of that species. Darwin could only think of fitness-reducing characteristics, such as the peacock's feathers, or the attractive female's physical weakness and fragility. But in fact, the characteristics of social selection develop according to the laws of natural selection, just as every other aspect of the constitution of a species. Moreover, it has been recognized for many years that the runaway selection model favored by Darwin and Fisher is not mathematically tenable (see my analysis in Game Theory Evolving (Princeton: Princeton University Press, 2009). The widely preferred theory is the costly signaling theory of Amotz Zahavi and William Hamilton (see Amotz Zahavi, "Mate Selection---A Selection for Handicap", Journal of Theoretical Biology 53 (1975):205-214; Amotz Zahavi and Avishay Zahavi, The Handicap Principle: A Missing Piece of Darwin's Puzzle (New York: Oxford University Press, 1997); William D. Hamilton and M. Zuk, "Heritable True Fitness and Bright Birds: a Role for Parasites?", Science 218 (1982):384-387). Costly signaling theory nicely embraces the dual conflict/cooperation nature of social interaction.
According to the handicap principle, females are indeed choosy, but they choose mates who have "good genes" and hence will contribute to hearty offspring. For instance, a peacock with a heavy parasitic load does not have the resources to produce beautiful plumage. A female does well to choose a mate with light parasitic load, because her offspring will inherit the immunity to pests enjoyed by her mate. To make a long story short, virtually every case of female choice is an instance of costly signaling. Female choice is thus fitness enhancing and evolves by natural selection. Of course, there are times when the cues used by the female as a signal of mate fitness are imperfect signals and can be exploited by males for their own reproductive ends. But in general we can expect mate choice and cooperation in rearing offspring to an intimate aspect of the fitness of a species, and that the strategic interaction of mates will be some pattern of cooperation and competition, as is the case of all cooperative enterprises (see Samuel Bowles and Herbert Gintis, A Cooperative Species: Human Reciprocity and its Evolution (Princeton: Princeton University Press, 2011).
Where does Roughgarden stand on all this? First, she completely rejects costly signaling theory, reciting its failures but not its successes. This is quite disingenuous; Roughgarden's social selection hypothesis is immeasurably strengthened when seen as extending and correcting costly signaling theory. Moreover, costly signaling theory does support some of traditional sexual selection theory, while contradicting other parts of the theory. Similarly, Roughgarden cannot give a cogent explanation of social selection theory because it in fact sometimes supports traditional sexual selection and sometimes does not.
This book is thus very frustrating. There is virtually no chance Roughgarden's message will be accepted by traditional population biologists because they will not get beyond her whole-cloth rejection of classical sexual selection. The layman will see the morality play of the good guys (Roughgarden and allies) against the bad guys (supporters of conflict/competition theories of all types), but they will be confused by the mass of evidence, only half of which supports her position. This book is aching to be rewritten from a balanced perspective.