Geometries
The following is based heavily on the 1975 NASA Summer Study on Space Settlements design (see Links), with further assistance from The Lewis One Space Colony design study by Al Globus (also see Links), plus some personal thoughts.
Shape relationships
Before considering the relative virtues of the various possible shapes or geometries for space habitations to give pseudogravity, a consideration of their relationships. Looking at Figure 1, start with what is probably the simplest shape, a, the simple vessel connected to a countermass. It is interesting to note that, aside from fairground rides, this is similar to the only pseudogravity device that we have currently built and used, the centrifuges used for testing pilots (and astronauts) tolerance of high 'g' forces such as might be experienced in combat or at launch. From a we progress to b by replacing the countermass with a vessel that matches the first, to give a dumbbell.
Figure 1. Relationships amongst various possible geometries for space habitations (assuming segments joined where possible). Diagram explained in text.
From b we can either project the dumbbell along its axis of rotation to give c, project it around the axis of rotation to give d, the torus, or join its two ends with a continuous pressure vessel to give f, a cylinder rotating about a short axis. d in its turn can be projected along its axis of rotation to give e, the extended torus. f can be projected along its axis of rotation to give g or around its axis of rotations to give h, a disc, and h can be projected along its axis of rotations to give i, a cylinder rotating about its long axis. Alternatively the disc, i, can be projected around an axis perpendicular to its axis of rotation to give the sphere, j, and j can be projected along its axis of rotation to give k, a cylinder with hemispherical end caps.
Figure 2 shows a similar set of relationships, but with the various shapes kept segmented rather than forming continuous bodies when projected.
Figure 2. Relationships amongst various possible geometries for space habitations (assuming segments kept separate where possible). Diagram explained in text.
Virtues (or otherwise) of the various shapes
The NASA Summer study designs were all based on shapes that could efficiently contain pressure, such a as spheres, tori and cylinders with hemispherical end caps. The possibilities considered here include such shapes and add others that may be more or less practical for reasons discussed in what follows.
a and l, have the virtue of simplicity and, therefore, low cost. One of the earliest practical plans for a pseudogravity habitat in space was that by Convair Astronautics for an essentially similar asymmetric design based on living inside a spent rocket (Atlas type - see Artists' Impressions) with a countermass, of 'heavy equipment', at the other end from the habitation (in some ways this was more of an asymmetric version of f or s). The possibility of using the countermass as a small, crude, but highly shielded, temporary refuge in the event of solar flares may make this asymmetric design more attractive for early habitations, since the walls of the living section itself are likely to be only thick enough to contain the pressure and rotation stresses involved and would not provide significant shielding in such early, exploratory, designs.
The dumbbell (b and m) is probably more efficient in terms of living space for a particular total mass. However, the cylinder rotating about its short axis (f and s) would allow easier axis to the centre (presumably used for docking) and would also allow multiple floors with different 'g' levels (which might be useful or fun). We do know that we can build pressurised habitable structures of this type (e.g. airliner fuselages) so that might be somewhat reassuring for early habitations.
The simple torus (d and q) was the design chosen by the NASA Summer study and proposed by Von Braun (and seen in various science fiction films). It has the advantage of allowing a large radius of rotation, to give pseudo gravity with less risk of nausea (see Puking or bouncing) while keeping the effective diameter of the pressure vessel small, allowing a lighter structure to resist pressure. However, the advantage of this lighter structure can only be fully realised if the habitat radiation shielding is kept as a separate shell. If the radiation shield is part of the habitat wall proper, then the strength and weight of the wall would go up anyway, regardless of the pressure vessel diameter, negating the advantage mentioned above. The segmented torus (o) would have the benefit of compartmentation to contain damage, disease, etc. and smaller compartments could probably be built to contain pressure more efficiently, but the psychological benefits of the longer sightlines of the simple torus would be lost. It may also be more difficult to give the same level of mechanical strength per unit of weight with such a structure. Perhaps the greatest benefit, however, to this and any other such segmented structure (i.e. most of those in Figure 2) would be that they could be assembled gradually from smaller unit parts, such an approach having been shown by Mir, for example, to be politically and economically more feasible than going for one big effort (and spend) in one go.
The axially extended torus, e, as described by Hess' Demeter design for example (see What rings are for) provides one of the more efficient designs in terms of habitable area per unit mass. The mass of the inner surface of the torus (or ceiling) would to some extent be offset by the reduced mass of the end walls and by the reduced atmosphere requirement (reducing requirement for volatiles as well as reducing mass) compared with a cylinder. The design would also share with the simple torus the ease of introducing sunlight via the 'inside' ceiling. This characteristic would be shared with the segmented design, r, although it would be rather more problematic with the maximally segmented design, p, because of the number of internal supports that may be needed. O'Neill chose something similar to, but not exactly the same as, r for the agricultural area of his Bernal Sphere Island One design.
The axially extended dumbbell design, c, or its uniform thickness equivalent, g, have some efficiency and sightline advantages over their non-extended equivalents and something like these has been touched on in science fiction (e.g. in the film 2010), but they do not seem to provide overwhelming advantages compared with other designs. g in particular does not seem like an efficient design for containing pressure or for providing shielded volume. The segmented equivalents, n and t, provide the general compartmentation benefits already mentioned, but those aside they are not particularly inspiring.
The disc (h and u) shares the problems of g in terms of inefficiency in containing pressure and shielding volume, but its projection, the cylinder, i, is probably one of the most favoured designs of all, providing long sightlines, lots of constant 'g' area, efficient shielding etc.. It is the design chosen by Clark for his 'Rama' science fiction and by Globus for the 'Lewis One' design (see Links). In terms of pressure containment the flat ends are less efficient than hemispherical end caps as used in k. However, as Globus has pointed out, the inclusion of shielding mass in the equation makes the flat ended cylinder more attractive (see below).
The cylinder with hemispherical end caps, k, is another of the most favoured designs as in, for example, the O'Neill Island Three study. It has the advantage over a flat ended cylinder that it is a 'natural' shape for pressure containment (i.e. it's the sort of shape you can make with party balloon), but it has the disadvantage that the 'land' area of the end caps themselves, while it may be aesthetically attractive and usable for recreation, is not going to be at the same pseudogravity level as the rest of the inner surface and does not, therefore, represent real living area. However, since the wall area of the hemisphere has to be shielded to the same level as the rest of the colony wall, the shield mass would be greater than the combination of shield and structural mass for a flat ended cylinder, despite the latter needing greater structural strength to give rigidity to contain pressure.
The sphere, j and w, is a shape that represents maximum possible strength for mass and the most efficient shape for shielding of the enclosed volume. It is the shape chosen by O'Neill's group for their Island One study. However, the usable area is obviously limited since, with constantly varying radius of rotation as one moves along the axis of rotation, the pseudogravity level would only be relatively constant for a narrow 'equatorial' strip. This makes the design somewhat inefficient in terms of pseudogravity area for unit mass. The same argument applies to the multiple sphere, x.
Summary
To summarise, the drivers of the decision about what shapes to choose for space habitats will vary depending on the uses to which the habitat will be put, the likely length of stay for the inhabitants, the degree to which the habitat may need to be translocated and the maturity of the industrial system in place to construct and support it. However, typical drivers will probably include the need for the radius to be large enough to minimise rotation rate for a given level of pseudogravity (see Puking or bouncing), the need to minimise surface area to minimise radiation shielding mass, the need to minimise structural mass (and thus expense) for a given habitable area and the need to contain and resist internal pressure efficiently. The possible role of segmentation/compartmentation has been mentioned above and to its virtues should possibly be added that of the control of gyroscopic effects by contra-rotation of different segments such as might be possible with designs like x or r (see Big Gyros).
Front running designs for habitats are probably the simple asymmetric or symmetric dumbbell or cylinder rotating on its short axis (a/ l, m/b and f/s) to start with, going up in size via the torus (q/d) to larger more permanent colonies such as the cylinders (with or without hemispherical end caps, k or i) or the axially extended torus (e), possibly with some degree of segmentation if it is meant to be moved much (like r but less so). Between cylinders and axially extended tori, at the largest sizes the cylinder has the benefit that there is less risk of anything falling onto the inhabitants because there is no ceiling. The absence of ceiling may also allow longer sightlines (depending on the presence of clouds or otherwise). However, the ceiling of an axially extended torus may be psychologically more acceptable than seeing inhabited surfaces of other parts of the cylinder apparently suspended several miles above. It would also allow for the incorporation of sprinkler heads for 'rain' which may be more reliable than that produced by 'weather' within a colony. Either design would allow various options for bringing in sunlight or for artificial lighting.