3.2.1. Obtaining and building CTSM and LILAC¶

This section describes the process for obtaining and building the CTSM library and its dependencies, and linking to these libraries in an atmosphere model’s build.

Important

This documentation only applies to the process where you are building CTSM with LILAC for use in an atmosphere model that has not been integrated with CESM or CIME. If you are using CTSM within CESM, or running CTSM in land-only mode with a data atmosphere, then you should refer to the general CTSM user’s guide as well as the CIME documentation.

3.2.1.1. Quick start example / overview¶

The basic process for obtaining and building CTSM is the following:

Obtain CTSM by running:

git clone https://github.com/ESCOMP/CTSM.git
cd CTSM
./bin/git-fleximod update

Then build CTSM and its dependencies. On a machine that has been ported to CIME, the command will look like this (example given for NCAR’s derecho machine):

./lilac/build_ctsm /glade/scratch/$USER/ctsm_build_dir --machine derecho --compiler intel

and then, before building the atmosphere model:

source /glade/scratch/$USER/ctsm_build_dir/ctsm_build_environment.sh

On a machine that has not been ported to CIME, you will need to provide some additional information. Run ./lilac/build_ctsm -h for details, but the basic command will look like this:

./lilac/build_ctsm ~/ctsm_build_dir --os Darwin --compiler gnu --netcdf-path /usr/local --esmf-mkfile-path /Users/sacks/ESMF/esmf8.0.0/lib/libO/Darwin.gfortranclang.64.mpich3.default/esmf.mk --max-mpitasks-per-node 4 --no-pnetcdf

In both cases, you will then need to include the necessary information in the include and link lines of the atmosphere model’s build. For a Makefile-based build, this can be done by including the file /PATH/TO/CTSM/BUILD/ctsm.mk in the atmosphere model’s build scripts, then adding CTSM_INCLUDES to the include line and CTSM_LIBS to the link line.

Further details on these steps are given below.

3.2.1.2. Prerequisites¶

Building CTSM requires:

a Unix-like operating system (Linux, AIX, OS X, etc.)
git version 1.8 or newer
python3
- Note that some scripts in the workflow look for ‘python3’ and others look for ‘python’. So python should be available under both of these names (although it is okay for python to refer to version 2.7.x).
perl version 5
a GNU version of the make tool (gmake)
CMake
Fortran and C compilers
- See https://github.com/escomp/cesm#details-on-fortran-compiler-versions for information on compiler versions known to work with CESM, and thus CTSM.
LAPACK and BLAS libraries
a NetCDF library version 4.3 or newer built with the same compiler you will use for CTSM
- a PNetCDF library is optional
a functioning MPI environment
- typically, this includes compiler wrappers like mpif90 and mpicc
ESMF version 8 or later
- ESMF is needed for LILAC (and for CESM3 and later)

3.2.1.3. Obtaining CTSM¶

CTSM and its dependencies (excluding the prerequisites noted above) can be obtained with:

git clone https://github.com/ESCOMP/CTSM.git
cd CTSM
./bin/git-fleximod update

By default, this will put you on the master branch of CTSM, which is the main development branch. You can checkout a different branch or tag using git checkout; be sure to rerun ./bin/git-fleximod update after doing so.

For more details, see https://github.com/ESCOMP/CTSM/wiki/Quick-start-to-CTSM-development-with-git

3.2.1.4. Building CTSM and its dependencies¶

3.2.1.4.1. Overview¶

CTSM provides a build script, lilac/build_ctsm, for building CTSM and its dependencies. (The dependencies built with this build script include various libraries that are packaged with CIME. This does not build the prerequisites noted above: it is assumed that those are already built on your machine.)

There are two possible workflows for building CTSM and its dependencies. The first works if you are using a machine that has been ported to CIME; the second works if you are using a machine that has not been ported to CIME. Both workflows are described below. If you are using a machine that has not been ported to CIME, it is possible to do a complete CIME port and then use the first workflow (by following the CIME porting guide), but unless you need to do so for other reasons (such as running CESM, or running CTSM in a land-only configuration forced by a data atmosphere, using the CIME scripting infrastructure), it is generally simpler to use the second workflow below: A full CIME port requires many settings that are not needed for just building CTSM.

There is a third usage where you simply want to rebuild after making some source code changes to CTSM. This is also documented below.

All of these workflows use CIME’s build system behind the scenes. Typically, you will not need to be aware of any of those details, but if problems arise, you may want to consult the CIME documentation.

3.2.1.4.2. Building on a CIME-supported machine¶

If you are using a machine that has been ported to CIME (for example, NCAR’s derecho machine), then you do not need to specify much information to build_ctsm. In addition, in this case, CIME will load the appropriate modules and set the appropriate environment variables at build time, so you do not need to do anything to set up your environment ahead of time. Building CTSM with LILAC requires ESMF. ESMF is an optional CIME dependency before CESM3 and not optional for CESM3. NCAR’s derecho machine DOES provide ESMF. For other machines, you may need to add this to your CIME port. Please see esmf.org for download and build instructions.

To build CTSM and its dependencies in this case, run:

./lilac/build_ctsm /PATH/TO/CTSM/BUILD --machine MACHINE --compiler COMPILER

where you should fill in the capitalized arguments with appropriate values for your machine.

Note

The given directory (/PATH/TO/CTSM/BUILD) must not exist. This directory is created for you by the build script.

Some other options to build_ctsm are supported in this case (but many are not, since they are only applicable to the non-CIME-supported machine workflow); run ./lilac/build_ctsm -h for details.

Important

If PNetCDF (parallel NetCDF) is not available on this machine, you will need to add the option --no-pnetcdf.

If you plan to run with OpenMP threading-based parallelization, or hybrid MPI/OpenMP, then it is important to add --build-with-openmp.

Besides the build files themselves, build_ctsm creates the following important files that are needed for the build of the atmosphere model:

/PATH/TO/CTSM/BUILD/ctsm.mk: This Makefile-formatted file gives variables that should be set in the atmosphere model’s build. See below for information on how to use this file.
/PATH/TO/CTSM/BUILD/ctsm_build_environment.sh or /PATH/TO/CTSM/BUILD/ctsm_build_environment.csh: These files specify the build environment that CIME used to build CTSM and its dependencies. Before building the atmosphere model, you should source the appropriate file (based on your shell - use the .sh file for bash and similar shells, and the .csh file for tcsh and similar shells). This will ensure that the atmosphere model is built with the same compiler and library versions as CTSM. For example, with bash: source /PATH/TO/CTSM/BUILD/ctsm_build_environment.sh.

3.2.1.4.3. Building on a machine that has not been ported to CIME¶

If you are using a machine thata has not been ported to CIME, then you need to specify additional information to build_ctsm that is needed by the build system. Before building CTSM, you should load any modules and/or set any environment variables required by the atmosphere model or CTSM builds, including all of the prerequisites noted above.

The minimal amount of information needed is given by the following:

./lilac/build_ctsm /PATH/TO/CTSM/BUILD --os OS --compiler COMPILER --netcdf-path NETCDF_PATH --esmf-mkfile-path ESMF_MKFILE_PATH --max-mpitasks-per-node MAX_MPITASKS_PER_NODE --pnetcdf-path PNETCDF_PATH

where you should fill in the capitalized arguments with appropriate values for your machine. Run ./lilac/build_ctsm -h for details on these arguments, as well as documentation of additional, optional arguments. Some of these optional arguments may be needed for successful compilation, while others (such as --pnetcdf-path) may be needed for good model performance.

Note

The given directory (/PATH/TO/CTSM/BUILD) must not exist. This directory is created for you by the build script.

Important

If PNetCDF (parallel NetCDF) is not available on your machine/compiler, you should use the option --no-pnetcdf instead of --pnetcdf-path. You must specify exactly one of those two options.

If you plan to run with OpenMP threading-based parallelization, or hybrid MPI/OpenMP, then it is important to add --build-with-openmp.

Example usage for a Mac (a simple case) is:

./lilac/build_ctsm ~/ctsm_build_dir --os Darwin --compiler gnu --netcdf-path /usr/local --esmf-mkfile-path /Users/sacks/ESMF/esmf8.0.0/lib/libO/Darwin.gfortranclang.64.mpich3.default/esmf.mk --max-mpitasks-per-node 4 --no-pnetcdf

Example usage for NCAR’s derecho machine (a more complex case) is:

module purge
module load ncarenv/1.3 python/3.7.9 cmake intel/19.1.1 esmf_libs mkl
module use /glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/intel/19.1.1/
module load esmf-8.2.0b23-ncdfio-mpt-O mpt/2.22 netcdf-mpi/4.8.0 pnetcdf/1.12.2 ncarcompilers/0.5.0

./lilac/build_ctsm /glade/scratch/$USER/ctsm_build_dir --os linux --compiler intel --netcdf-path '$ENV{NETCDF}' --pio-filesystem-hints gpfs --pnetcdf-path '$ENV{PNETCDF}' --esmf-mkfile-path '$ENV{ESMFMKFILE}' --max-mpitasks-per-node 36 --extra-cflags '-xCORE_AVX2 -no-fma' --extra-fflags '-xCORE_AVX2 -no-fma'

(It’s better to use the alternative process for a CIME-supported machine in this case, but the above illustrates what would be needed for a machine similar to this that has not been ported to CIME.)

Besides the build files themselves, build_ctsm creates an important file that is needed for the build of the atmosphere model: /PATH/TO/CTSM/BUILD/ctsm.mk. This Makefile-formatted file gives variables that should be set in the atmosphere model’s build. See below for information on how to use this file.

3.2.1.4.4. Rebuilding after changing CTSM source code¶

To rebuild after changing CTSM source code, you should follow one of the above workflows, but the build_ctsm command will simply be:

./lilac/build_ctsm /PATH/TO/CTSM/BUILD --rebuild

where /PATH/TO/CTSM/BUILD should point to the same directory you originally used.

3.2.1.5. Including CTSM in the atmosphere model’s build¶

Once you have successfully built CTSM and its dependencies, you will need to add various paths to the compilation and link lines when building your atmosphere model. For a Makefile-based build system, we facilitate this by producing a file, /PATH/TO/CTSM/BUILD/ctsm.mk, which you can include in your own build script. (We do not yet produce an equivalent for CMake or other build systems.)

There are two important variables defined in this file:

CTSM_INCLUDES: This variable should be included in the compilation line for the atmosphere model’s source files. It lists all paths that need to be included in these compilations so that the compiler can find the appropriate Fortran module files.
CTSM_LIBS: This variable should be included in the link line when creating the final executable. It lists paths and library names that need to be included in the link step. Note: This may not include all of the libraries that are prerequisites, such as LAPACK, BLAS and NetCDF. If your atmosphere doesn’t already require these, you may need to add appropriate information to your atmosphere model’s link line. However, it should already include all required link information for ESMF.

Other variables in this file do not need to be included directly in the atmosphere model’s build (they are just intermediate variables used to create CTSM_INCLUDES and CTSM_LIBS).

For example, for the WRF build, we do the following: If building with CTSM, then we expect that the user has set an environment variable:

export WRF_CTSM_MKFILE=/PATH/TO/CTSM/BUILD/ctsm.mk

If that environment variable exists, then the configure script adds the following to the Makefile-based build:

Includes the ctsm.mk file (like include ${WRF_CTSM_MKFILE})
Adds a CPP definition, -DWRF_USE_CTSM, which is used to do conditional compilation of the CTSM-LILAC interface code
Adds $(CTSM_INCLUDES) to its variable INCLUDE_MODULES
Adds $(CTSM_LIBS) to its variable LIB