http://www.sfu.ca/physics/cosmology/defrost
Copyright (C) 2007–2012 Andrei Frolov <frolov@sfu.ca>
Distributed under the terms of the GNU General Public License
If you use this code for your research, please cite
arXiv:0809.4904
Included in this distribution are:
README - this file
Makefile - build script
defrost.f90 - DEFROST source code
Download and install required libraries (FFTW, optionally SILO/HDF5).
Edit Makefile for your Fortran compiler settings and library locations.
Edit defrost.f90 specifying your reheating model (number of scalar
fields, initial conditions, potential and its derivatives), configure
the solver (box size, spatial and time resolutions, etc) and output
settings. Build defrost binary by saying make and run it, optionally
saving the log defrost > LOG.
CAUTION: Being a 3D PDE solver, DEFROST needs a lot of resources. Active memory footprint while running 256^3 simulation is around 512Mb, and the execution speed on a dual Xeon 5160 machine (64 bit, 4 cores at 3.0GHz, quad-channel DDR2 RAM) is 0.32s per time step (for 2 field model, second order integrator). If you enable 3D output, a lot of data will get dumped to your disk. The size of a single 256^3 frame containing all variables is about 1Gb.
DEFROST is written in Fortran–90 (with Fortran preprocessor used).
The code is developed and optimized for Intel Fortran compiler (ifort).
GNU Fortran compilers (g77, g95, and gfortran) are not supported.
DEFROST uses FFTW library for Fourier transforms.
Download and compile FFTW version 3.0 or later (tested with FFTW 3.1.2),
including Fortran wrappers and threaded FFT support. Fedora users can
install pre-compiled binaries by saying yum install fftw fftw-devel.
DEFROST output is instrumented for interactive 3D visualization with LLNL VisIt. DEFROST can dump 3D data in native VisIt format (using SILO library) or raw box-of-values format (simple to read and write, but missing some metadata and requiring many files per frame as opposed to a single SILO database).
If you would like to use VisIt native SILO format (recommended), you
will need to install SILO and HDF5 libraries. Download and install
HDF5 library first, which SILO uses as a
back-end. Fedora users can install pre-compiled binaries by saying
yum install hdf5 hdf5-devel. Download and install SILO library next. Getting older versions of
SILO to build against shared HDF5 library required patching,
but this is not a problem anymore.
On a Mac, all the required libraries are available from MacPorts.
The provided Makefile is configured for optimized parallel build using
Intel Fortran compiler, threaded FFTs and SILO output. Unfortunately,
GNU Fortran compiler produces (very) much slower code and cannot handle
parallelism, so the support for it was dropped. Compiling with other
Fortran compilers should be possible, but was not tested.
Adjust FFLAGS to optimize for your machine (in particular, -x flag
selects target CPU). As DEFROST uses Fortran preprocessor, -fpp flag
is necessary for it to compile. Floating point numbers are set to double
precision by default (-r8 flag), but the code can be compiled for
single precision (-r4) to decrease memory footprint, or quad precision
(-r16) to improve roundoff errors (at 10x runtime penalty). Regardless
of solver precision, the bindings to FFTW and SILO will use double
precision floats.
DEFROST solver code is fully parallel, and can be build as such using
either automatic parallelizer (-parallel) or OpenMP (-openmp).
Automatic parallelizer will produce slightly more efficient code.
Threaded FFTs will be used if THREADS variable is set; comment it out
if you don’t want that (or you don’t have libfftw3_threads).
By default, DEFROST uses automatic array variables, which get allocated
on stack. This can easily exceed default process limits set by the shell
(especially if OpenMP is used). If you get immediate segmentation fault
on startup without any error messages, try overriding stack size limit
(ulimit -s unlimited in bash), or use dynamically allocated arrays (at
small performance penalty) by defining DYNAMIC_ARRAYS in Makefile.
Changing compiler memory model by using -mcmodel=large flag might be
better instead.
Finally, give locations to search for header files and libraries (using
-I... and -L... flags correspondingly), if your FFTW and SILO/HDF5
libraries are not in the standard locations. If you do not have (or want
to use) SILO libraries, comment out SILO variable, and DEFROST will
compile for raw output instead.
There is a number of parameters defined in DEFROST source code which control the solver and the output behavior, and are intended to be modified by user. They are documented inline. I tried to separate reheating model from numerical implementation as much as practical, but if you want to change the model, you will have to edit the source code directly. Version 1.x code is described in arXiv:0809.4904. Version 2.x uses new (and awesome) symplectic PDE integrators we developed, and will be documented in upcoming publication.
The 3D field evolution equations are discretized on a n^3 uniform grid
with spacing dx. Time step dt has to be small enough to satisfy
Courant’s conditions, and to resolve oscillations of all the fields.
Parameter m (padded grid size) deserves special explanation. Inner
evolution loop of DEFROST is highly optimized, and runs almost at the
memory bandwidth speed. If array access pattern stride is commensurate
with cache page size, memory access performance can be affected (this is
known as “cache thrashing”). In the worst case, the effect can be as
much as 2x slowdown. Padding array by a few empty elements avoids it.
Try setting m+1 prime, or a product of a few large primes.
DEFROST has flexible output options (controlled by output$...
parameters) and lazy calculation policy (only stuff you want gets
calculated). Disabling unneeded output can save execution time and disk
space. Select quantities {bov/psd/cdf} and variables
{fld/vel/rho/prs/pot} you want output, and select curve data
format {gnu/vis}. The output is dumped every nt time steps, and 3D
image can be downsampled (using nearest neighbour) to save disk space.
By default, dilution due to expansion is scaled out. All output (except
execution log to stdout) can be disabled by setting output parameter
to .false.. For fast logging of expansion history only, calls to
checkpt() inside the main evolution loop can be omitted.
For performance reasons, scalar field potential and its derivatives had
to be inlined separately inside the evolution loop in ...step()
routines. In addition, human-readable potential string is output in
head() routine. These are defined in the model section using
preprocessor macros. Don’t forget that potential enters indirectly into
initial conditions (both for homogeneous field components and
fluctuations), so they will need to be adjusted as well.
This is easy. Say make and defrost.
DEFROST will print expansion history and average density and pressure on
stdout as it goes along. Say defrost > LOG if you want to keep it.
DEFROST can output spectra and distributions of select variables
(enabled by default). The file CDF contains cumulative distribution
function, and file PSD contains power spectral density (logarithmic in
base 10, in arbitrary units). Both files are in plain text format, with
brief information header (lines starting with comment symbol ‘#’),
followed by data. Values for all the fields appear on the same line in
order specified in the header, separated by white space. Blocks of
values at different time (frames) are separated by two empty lines.
This data format is directly used by gnuplot, and should be easy to convert for your favourite plotting program. Some gnuplot examples:
plot 'LOG' using 1:2 with lines # expansion history
plot 'LOG' using 1:($5/$4) with lines # equation of state
plot 'LOG' using 1:(($4/3-$3**2)*$2**2) w l # flatness violation
plot 'PSD' i 0:512:16 u 2:((10**$3)*$2**4) w l # inflaton spectra
DEFROST can output 3D snapshots of select variables periodically during
the evolution (disabled by default due to high storage requirements, set
output$bov to enable). The output is instrumented for interactive 3D
visualization with LLNL VisIt. Check
examples on DEFROST website or the image gallery on VisIt website to see
what you can do with it. If you compiled DEFROST with SILO support, you
can just open frame-*.silo database to access DEFROST simulation data.
Otherwise, you will have to open databases for each variable separately
(they are named xxx-*.bov). Set output$vis to export statistics to
VisIt as well.
VisIt can make a movie of your 3D visualization. Set up your plots, tidy
up using Controls > Annotation..., and go to File > Save Movie...
wizard. Movie encoder shipped with VisIt distribution is not very good,
so I recommend saving movie frames as images (in a lossless format such
as PNG), and encoding using stand-alone encoder. I highly recommend
mplayer/mencoder with lavc codec (builtin)
or x264 codec. If you
have deep pockets and want GUI and lots of presets, Adobe Media Encoder (ships
with CS5.5) might be an option.
Mencoder is a very capable program, but can be intimidating to use because of command line interface and multitude of options. Grab my presets to see encoding settings I use. The encoding itself is best done in two passes:
mencoder -profile x264 -x264encopts pass=1 mf://"/tmp/movie_*.png" -o test.avi
mencoder -profile x264 -x264encopts pass=2 mf://"/tmp/movie_*.png" -o test.avi
After encoding is done, test the playback with mplayer test.avi. If
you want to play the movie from other applications (e.g. to include it
in a PowerPoint presentation), you will need to install system-wide
MPEG4/H264 codecs. I recommend Combined Community Codec Pack on Windows, and Perian
for QuickTime support on Mac.
Bug reports, patches, suggestions, flames, etc. should go to the author. Have fun! ^_^
Andrei Frolov <frolov@sfu.ca> – $Id$