The procedure for the parallelization of POM is the implementation of a message-passing code using two-dimensional data decomposition of the horizontal domain. This approach ensures the portability across a large variety of parallel machines, and allows to maintain the same numerical algorithms used in the serial code. The horizontal global domain is thus partitioned into two-dimensional local domains using a Cartesian decomposition, and the vertical dimension is not divided. Each local domain (i.e. each processor) integrates independently the code on the local domain. The computation applied to each local domain is the same as that applied to the entire global domain with the serial code. The horizontal arrays assigned to each local domain are expanded by one grid point in each horizontal dimension, creating a halo of ghost cells (Fig. 1). The two- and three-dimensional arrays in these ghost cells are exchanged between neighbor local domains using the MPI standard interface (http://www.mcs.anl.gov/research/projects/mpich2/).

 

Fig. 1. Example of data decomposition scheme for a global size of 14 × 17, local sizes of 6 × 7, and 9 processors. Crosses represent the boundaries of the global domain, shadow areas are the ghost cells (or local boundaries) and arrows indicate the communication between local domains to exchange variables at the ghost cells.

In order to assess the performance of sbPOM, we used the seamount test case (Ezer et al., 2002; Mellor et al., 1998) which is made available with the code. It is a stratified Taylor column problem which simulates the flow across a seamount. The test case has 1026 × 770 global grid points and 31 vertical sigma levels. The simulation time is measured while varying the number of processors on a Blue Gene/L massively parallel supercomputer housed in the New York Center for Computational Sciences (http://www.newyorkccs.org/). The problem size for the test case is too large to run on one or two processors due to memory allocation. Simulations are therefore run with a number of processors ranging from 4 to 2048 (2K), and the efficiency is measured with respect to the four processors, t4/tnproc·(nproc/4), where tnproc is the simulation time when using nproc processors. The efficiency is very high even for 2K processors (∼0.8) (Fig. 2).

 

Fig. 2. Performance for the test case. (a) Time (seconds per simulation day) as a function of the number of processors. (b) Parallel efficiency relative to performance on four processors. The black line (in both panels) is efficiency equal to one.