Reading Data with the OpenCosmo Toolkit

Once a query is complete, you can download the results to your local computer or another machine. Catalog, particle, and map queries return HDF5 files in the OpenCosmo data format, which this page covers; prepackaged analysis tasks return zip archives of plots and tables, and visualization tasks return Parquet files. The OpenCosmo Toolkit reads and writes files in the OpenCosmo data format and supports further filtering, selection, and computation on the data. If you are new to the toolkit, we recommend starting with the example website; for a full discussion of its capabilities, see the documentation. This page gives a basic overview.

Opening Data

Opening data with the toolkit is straightforward:

import opencosmo as oc

ds = oc.open("my_data.hdf5")

For simple catalog queries, ds is an OpenCosmo Dataset. The dataset does not hold the file's contents in memory; it provides an interface to them. To retrieve all of the data, call get_data:

data = ds.get_data()

This returns an Astropy table.

For more complex queries (such as particle queries), the file may contain more than one dataset. In these cases ds will be a collection, such as a StructureCollection, rather than a single dataset. See the collections documentation for more information.

Units

Columns in OpenCosmo datasets carry Astropy units. By default, data is presented in the "comoving" unit convention: coordinates and velocities are comoving, and factors of the reduced Hubble parameter $hh$ are absorbed into the values (masses in $M_{\odot }M\_\backslash odot$ rather than $M_{\odot }\mathrm{/}hM\_\backslash odot/h$, for example). The with_units method converts to other conventions, including "physical" and "scalefree":

ds = ds.with_units("physical")

See the units documentation for details.

Accessing Simulation Information

OpenCosmo datasets and collections carry metadata about their origin, such as the cosmology the simulation was run with and the box size. The cosmology is available directly through the cosmology attribute:

cosmology = ds.cosmology

and the simulation parameters through:

sim_params = ds.simulation

Performing Additional Queries

A portal query often returns more data than a particular analysis needs, or you may want to partition the data in a way specific to your work. The toolkit makes further filtering straightforward:

min_mass_filter = oc.col("fof_halo_mass") > 1e14

ds = ds.filter(min_mass_filter)
data = ds.get_data()

The filter call returns a new OpenCosmo dataset. Operations on OpenCosmo datasets always return a new dataset rather than modifying the existing one, so several datasets can be derived from a single parent:

min_mass_filter = oc.col("fof_halo_mass") > 1e14
max_mass_filter = oc.col("fof_halo_mass") < 1e14

ds_high_mass = ds.filter(min_mass_filter)
ds_low_mass = ds.filter(max_mass_filter)

Selecting Subsets of Columns

Most analyses use only a subset of the available columns, and reading the rest costs time and memory. The select method restricts a dataset to the columns you need:

ds = ds.select(("fof_halo_mass", "sod_halo_mass", "sod_halo_cdelta"))

Getting Rows and Sorting

It is often useful to test an analysis on a small number of objects before applying it to the full dataset. The take method returns a subset of rows; for example, one hundred random halos from a catalog:

ds = ds.take(100, at="random")

To select a specific subset instead, such as the one hundred most massive halos, combine take with sort_by:

ds = ds.sort_by("fof_halo_mass", invert=True).take(100, at="start")

Like numpy and astropy, OpenCosmo sorts in ascending order by default; setting invert=True sorts in descending order, so the example above keeps the most massive halos.

Combining Columns

Columns can be combined into new columns relevant to your analysis:

fof_halo_px = oc.col("fof_halo_mass")*oc.col("fof_halo_com_vx")
ds = ds.with_new_columns(fof_halo_px = fof_halo_px)

The new dataset contains a column named fof_halo_px that behaves like any other column. Because the toolkit is lazy, with_new_columns does not immediately compute the column in memory; the values are computed when the data is requested. If a subsequent filter removes a large number of rows, the new column is only evaluated for the rows that pass the filter — a substantial saving in memory and computation for large datasets.

Some derived columns are too complex to express with simple arithmetic on existing columns. For those cases, use the evaluate method; see the full documentation.

Structure Collections

If you performed a halo particle query (or a galaxy query with halo information included), OpenCosmo opens the file as a StructureCollection rather than a Dataset. A StructureCollection holds multiple related datasets grouped into structures (halos or galaxies) — for example, a halo catalog together with each halo's particles.

The StructureCollection automatically associates particles with their halos. For example, you can iterate over all halos with their particles:

collection = oc.open("my_collection.hdf5")
for halo in collection.halos():
    halo_properties = halo["halo_properties"]
    dm_particles = halo["dm_particles"]
    particle_dx = halo_properties["fof_halo_center_x"] - dm_particles.select("x").get_data()

In each iteration, halo_properties is a dictionary of the halo's properties and dm_particles is an OpenCosmo dataset containing that halo's particle data.

As with datasets, filter, take, select, and with_new_columns work on structure collections, with some differences arising from the fact that a collection contains multiple datasets. See the StructureCollection documentation for more information.

Next Steps

The OpenCosmo Toolkit provides substantial additional functionality not covered in this overview, including automatic parallelization with MPI. See the full documentation for more.

We are always working to improve the toolkit. If you encounter a bug or would like a feature that would help your science, please open an issue on the GitHub repository.