HETDEX opens Massive Cosmic Dataset to Scientists, Novices, AI and the public

The HETDEX database contains a whopping 600 million spectra (i.e. the light broken apart into its colours) for a period of history known as Cosmic Noon, 10 billion to 12 billion years ago. “This is a spectral map of the universe. It turns every point of light into a barcode of physics,” said Erin Mentuch Cooper, HETDEX data manager and lead author on the paper announcing the release. “The real excitement is what happens when thousands of astronomers start exploring it.”

One of them is Dr Meredith Powell at AIP, who has both helped on data quality tests and scientifically studies active galactic nuclei (AGN). “I am looking at trends between AGN properties and their surrounding galaxy environments,” says Powell. “For probing how supermassive black holes grew and affected their cosmic neighborhoods when the universe was at its busiest, the untargeted HETDEX dataset is fantastic,” she adds.

From 2017 to 2024, the Hobby-Eberly Telescope at McDonald Observatory surveyed a region of night sky equivalent to 2,000 full Moons, creating a map of the distant universe. HETDEX is using that map to solve the riddle of dark energy, the unknown substance causing our universe to expand more and more quickly over time. To do this, it is charting the location of over a million early galaxies. However, it has also gathered data on all of the space in between.

The HETDEX survey could be so large, because the instrument features 35,000 optical fibres that result in as many spectra. “The AIP developed and built the specialized optical fibre cables that feed the light from the 10m large telescope to the set of spectrographs,” explains Dr Andreas Kelz, head of the spectroscopic instrumentation group at AIP. “In this way, the instrumentation can measure tens of thousands of spectra at once which finally allows to map the distribution of galaxies across enormous cosmic volumes.”

The database consists of 431,000 data cubes that map information into three-dimensional space. When measured on the sky, each is roughly one thirtieth the size of the full Moon. Most correspond to regions around the Big Dipper and Orion. In addition to raw data, the release also contains a catalogue of every object HETDEX has found so far: over one million distant galaxies, half a million nearby star-forming galaxies, 18,000 supermassive blackholes, and over 150,000 stars. Scientists, students, and citizen researchers can download customized subsets of data based on sky location.

While the release is based on half a petabyte of data, the team was able to process it down to a more manageable 10 terabytes. It also developed extensive tutorials and tools to help users – both human and AI - to make the most of this massive, complex dataset. “It's been so important for me to make it as accessible as possible,” said Mentuch Cooper. “We’ve turned more than half a billion spectra into something you can actually explore. It’s like compressing a universe of information into something you can hold in your hands.”

Due to the depth of the HETDEX database, using Artificial Intelligence (AI) is expected to play a major role in sorting through it all. HETDEX also used automated methods to comb through its observations and identify possible early galaxies. In parallel, more than 24,000 citizen scientists helped confirm the presence of these galaxies through the Dark Energy Explorers program.

Today’s release marks the first time the full HETDEX dataset and survey catalogue have been made available together. While the core survey is now complete, observations are ongoing, calibrations continue to improve, and supplementary releases are expected for the future.