Project Leader

RAMSES is a typical computational intensive application used by astrophysicists to study the formation of galaxies. RAMSES is used, among other things, to simulate the evolution of a collisionless, self-gravitating fluid called "dark matter" through cosmic time. Individual trajectories of macro-particles are integrated using a state-of-the-art " N body solver ", coupled to a finite volume Euler solver, based on the Adaptive Mesh Refinement technics. The computational space is decomposed among the available processors using a mesh partitionning strategy based on the Peano--Hilbert cell ordering. Cosmological simulations are usually divided into two main categories. Large scale periodic boxes requiring massively parallel computers are performed on very long elapsed time (usually several months). The second category stands for much faster small scale "zoom simulations". One of the particularity of the HORIZON project is that it allows the re-simulation of some areas of interest for astronomers.

Figure 1: Re-simulation on a supercluster of galaxies to increase the resolution

For example in Figure 1, a supercluster of galaxies has been chosen to be re-simulated at a higher resolution (highest number of particules) taking the initial information and the boundary conditions from the larger box (of lower resolution). This is the latter category we are interested in.

Figure 2: Time sequence (from left to right) of the projected density field in a cosmological simulation (large scale periodic box).

Performing a zoom simulation requires two steps: the first step consists of using RAMSES on a low resolution set of initial conditions i.e. with a small number of particles) to obtain at the end of the simulation a catalog of "dark matter halos", seen in Figure 2 as high-density peaks, containing each halo position, mass and velocity. A small region is selected around each halo of the catalog, for which we can start the second step of the "zoom" method. This idea is to resimulate this specific halo at a much better resolution. For that, we add in the Lagrangian volume of the chosen halo a lot more particles, in order to obtain more accurate results. Similar "zoom simulations" are performed in parallel for each entry of the halo catalog and represent the main resource consuming part of the project.

RAMSES simulations are started from specific initial conditions, containing the initial particle masses, positions and velocities. These initial conditions are read from Fortran binary files, generated using a modified version of the Grafic code. This application generates Gaussian random fields at different resolution levels, consistent with current observational data obtained by the WMAP satellite observing the cosmic microwave background radiation. Two types of initial conditions can be generated with Grafic:

• Single level: this is the "standard" way of generating initial conditions. The resulting files are used to perform the first,low-resolution simulation, from which the halo catalog is extracted.
• Multiple levels: this initial conditions are used for the "zoom simulation" The resulting files consist of multiple, nested boxes of smaller and smaller dimensions, as for Russian dolls. The smallest box is centered around the halo region, for which we have locally a very high accuracy thanks to a much larger number of particles.

The result of the simulation is a set of "snaphots". Given a list of time steps (or expansion factor), RAMSES outputs the current state of the universe (i.e. the different parameters of each particules) in Fortran binary files.

These files need post-processing with Galics softwares: HaloMaker, TreeMaker and GalaxyMaker. These three softwares are meant to be used sequentially, each of them producing different kinds of information:

• HaloMaker: detects dark matter halos present in RAMSES output files, and creates a catalog of halos
• TreeMaker: given the catalog of halos, TreeMaker builds a merger tree: it follows the position, the mass, the velocity of the different particules present in the halos through cosmic time
• GalaxyMaker: applies a semi-analytical model to the results of TreeMaker to form galaxies, and creates a catalog of galaxies

Figure 3: workflow of the simulation

Figure 3 shows the sequence of softwares used to realise a whole simulation.

Grid'5000 is the French Research Grid. It is composed of 9 sites spread all over France, each with 100 to 1000 PCs, connected by the RENATER Education and Research Network (1Gb/s or 10Gb/s). For our experiments, we deployed a DIET platform on 5 sites (6 clusters).

• 1 MA deployed on a single node, along with omniORB, the monitoring tools, and the client
• 6 LA: one per cluster (2 in Lyon, and 1 in Lille, Nancy, Toulouse and Sophia)
• 11 SEDs: two per cluster (one cluster of Lyon had only one SeD due to reservation restrictions), each controlling 16 machines (AMD Opterons 246, 248, 250, 252 and 275)

We studied the possibility of computing a lot of low-resolution simulations. The client requests a 128^3 particles 100Mpc.h^-1 simulation (first part). When it receives the results, it requests simultaneously 100 sub-simulations (second part). As each server cannot compute more than one simulation at the same time, we won't be able to have more than 11 parallel computations at the same time.

The experiment (including both the first and the second part of the simulation) lasted 16h 18min 43s (1h 15min 11s for the first part and an average of 1h 24min 1s for the second part).

The benefit of running the simulation in parallel on different clusters is clearly visible: it would take more than 141h to run the 101 simulation sequentially.

Website:

• http://www.projet-horizon.fr/

License type:

CeCILL.

The needs:

• Language(s):

  Fortran 90, C/C++ for the client and servers

• Compiler(s):

  gfortran, gcc

• Library(ies):

  In case of parallel computations : mpi

• System(s):

  Linux, Unix

• Memory:

  For a 128^3 particles 100Mpc.h^-1 simulation : 2GB.

• Disk:

  For a 128^3 particles 100Mpc.h^-1 simulation : between 26 GB and ... depending on the number of backups

• Mean execution time:

  For a 128^3 particles 100Mpc.h^-1 simulation : around 1h15 for the first part and around 1h20 for the second part.

• Number of processors:
  • Minimum: 1
  • Maximum: 5000
• Use of a Database:

  No.

• Other specific needs:

  NFS partition to store files