N-body simulations
N-body simulations are ones which treat the Universe as a collection of particles, under the influence of gravitational forces through the Vlasov-Poisson equation. The Hybrid/Hardware-Accelerated Cosmology Code (HACC) efficiently solves this at cosmological scales, using a long-ranged spectral solver based off the SWFFT parallel FFT code (https://git.cels.anl.gov/hacc/SWFFT), and a short-ranged tree-based solver, optimized for a number of supercomputer architectures. See https://arxiv.org/abs/1410.2805 for more details.
Gravity-Only vs. Hydrodynamic
The most straightforward simulation is a gravity-only simulation. In this simulation, we only model the gravitational forces in the Universe. It is tempting to think of such a simulation as a dark matter-only simulation but this is not quite accurate, especially when you are comparing to our physical Universe. More accurately, the simulation models our Universe (and all of its associated mass) if the only force was gravity. Such a Universe does not form stars and galaxies, but it does form halos and large-scale structure.
In hydrodynamic simulations, we additionally model the physical processes of gases and stars.
It's reasonable to ask the following quesetion: Why run gravity-only simulations at all? There are a few answers to this. First, hydrodynamic simulations are much more computationally demanding, requiring around 10x the processing power of an equivalent gravity-only simulation.
Particles
The smallest unit of mass in a cosmological simulation is known as a particle. This is not a particle in the literal physics sense, but rather a collection of mass on the order of solar masses.
In gravity-only simulation, particles come in exactly one flavor: gravity-only particles. In hydrodynamic simulations we get several, such as dark matter, gas, and stars. Dark matter and gravity only particles are always the same throughout the course of the simulation. Star and gas particles though can change mass due to the physical processes that govern them.
Hydrodynamics Solver
To solve for the effect of hydrodynamics we use a Lagrangian method, in other words a method which treats particles as fluid elements. For that we use a high-order SPH scheme called Conservative Reproducing Kernel SPH (CRK-SPH, https://arxiv.org/abs/1605.00725). This reformulates the fluid equations to enforce conservation of mass, linear momentum and energy, while maintaining approximate angular momentum conservation. Adding this 'non-radiative' hydrodynamics does come at a cost however, in particular in addition to solving a larger set of equations, you also now need much higher time-resolution in the form of a sub-cycler within each timestep, which runs on GPU systems to resolve the hydrodynamics.
Subgrid Modeling
Astrophysical processes typically exist at much smaller scales and finer time intervals than can be resolved from large scale simulations, making it impossible to simulate them from first principals. Instead sub-grid models, approximate parameterized solutions, are used and calibrated to match observations. Typical subgrid models include star formation, metal enrichment, AGN and supernova feedback, as well as radiative and metal-line cooling. These models require extremely high time-resolution, for which we use an adaptive integration scheme based on local conditions.
Halo Finding
Particles in a simulation collect into structures known as halos. The process of locating these halos and measuring their properties is known as halo finding. In the halo catalogs available on this portal, there are two main measurements of a given halo's properties: friends-of-friends (FoF) and spherical overdensity (SO or SOD).
FoF Halo Finding
The starting point for finding halos in HACC simulations is the Friends-of-Friends algorithm. Start by selecting a particle from the simulation. Locate any other particles within a specified distance of the original particle and mark them. These are known as the friends of the original particle. Then, go through all of these friends and perform the same marking procedure. The particles marked during this process are known as the friends-of-friends of the original particle.
If you continue this process, you will eventually reach a point where every friend of every marked particles is also a marked particle. This collection of marked particles is an FoF Halo. We can now measure properties of this collection of marked particles, such as its total mass or bulk velocity.
You should take note that this algorithm treats all particles equivalently. For a multi-species simulation this poses a problem, therefore we only run FoF finders on dark matter particles, and exclude other particle types from the measurement.
SOD Halo Measurements
The starting point of spherical overdensity halo measurements is the FoF halos from the previous step.
The SOD measurement starts at the minimum potential point (center) of the FoF halo. Draw a sphere around the center and measure the average density inside of it. That is, take the total mass of all particles within the sphere and divide by the sphere's volume. Increase the radius of the sphere until the density drops below a particular threshold. Everything within the sphere is considred part of your SoD Halo.
There are many different choices that can be made for the threshold density. In the halo catalogs available on this portal, the "sod_halo_mass" column represents the radius at which the average density drops below 200 times the critical density of the universe, also known as . However there are many other measurements available in the catalog, such as .
For hydrodynamic simulations, SOD masses contain all of the particles within this radius, not just the dark matter particles, and use the total density of all species in the computation. This makes SOD measurements the typical default mass measurement for hydro simulations.
Galaxy finding
To find galaxies in the simulations, we look at all star particles on a given simulation rank, and use a generalization of the friends-of-friends algorithm called DBSCAN, where instead of finiding all particles with a single neighbour (or friend), you select all star particles with at least 10 neighbours within 50kpc in proper units, and determine the center using the minimum potential point. We then compute galaxy properties in a way analogous to SOD measurements, using a fixed 50kpc aperture (again in proper units). We use the aperture measurements for our galaxy properties but retain the DBSCAN information.