Oasis: A high-level/high-performance open source Navier–Stokes solver

https://doi.org/10.1016/j.cpc.2014.10.026Get rights and content

Abstract

Oasis is a high-level/high-performance finite element Navier–Stokes solver written from scratch in Python using building blocks from the FEniCS project (fenicsproject.org). The solver is unstructured and targets large-scale applications in complex geometries on massively parallel clusters. Oasis utilizes MPI and interfaces, through FEniCS, to the linear algebra backend PETSc. Oasis advocates a high-level, programmable user interface through the creation of highly flexible Python modules for new problems. Through the high-level Python interface the user is placed in complete control of every aspect of the solver. A version of the solver, that is using piecewise linear elements for both velocity and pressure, is shown to reproduce very well the classical, spectral, turbulent channel simulations of Moser et al. (1999). The computational speed is strongly dominated by the iterative solvers provided by the linear algebra backend, which is arguably the best performance any similar implicit solver using PETSc may hope for. Higher order accuracy is also demonstrated and new solvers may be easily added within the same framework.

Program summary

Program title: Oasis

Catalogue identifier: AEUW_v1_0

Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEUW_v1_0.html

Program obtainable from: CPC Program Library, Queen’s University, Belfast, N. Ireland

Licensing provisions: GNU Lesser GPL version 3 or any later version

No. of lines in distributed program, including test data, etc.: 3491

No. of bytes in distributed program, including test data, etc.: 266924

Distribution format: tar.gz

Programming language: Python/C++.

Computer: Any single laptop computer or cluster.

Operating system: Any (Linux, OSX, Windows).

RAM: a few Megabytes to several hundred Gigabytes

Classification: 12.

External routines: FEniCS 1.3.0 (www.fenicsproject.org, that in turn depends on a number of external libraries like MPI, PETSc, Epetra, Boost and ParMetis)

Nature of problem:

Incompressible, Newtonian fluid flow.

Solution method:

The finite element method.

Unusual features:

FEniCS automatically generates and compiles low-level C++ code based on high-level Python code.

Running time:

The example provided takes a couple of minutes on a single processor.

Introduction

The Navier–Stokes equations describe the flow of incompressible, Newtonian fluids. The equations are transient, nonlinear and velocity is non-trivially coupled with pressure. A lot of research has been devoted to finding efficient ways of linearizing, coupling and solving these equations. Many commercial solvers for Computational Fluid Dynamics (CFD) are available, and, due to the complexity of the high-level implementations (usually Fortran or C), users are often operating these solvers through a Graphical User Interface (GUI). To implement a generic, unstructured Navier–Stokes solver from scratch in a low-level language like C or Fortran is a considerable and time-consuming task involving tens of thousands of lines of error prone code that require much maintenance. Nowadays, as will be shown in this paper, the use of new and modern high-level software tools enables developers to cut the size of programs down to a few hundred lines and development times to hours.

The implementation of any unstructured (Eulerian) CFD-solver requires a computational mesh. For most CFD software packages today the mesh is generated by a third-party software like, e.g., the open source projects VMTK  [1], Gmsh  [2] or Cubit  [3]. To solve the governing equations on this computational mesh, the equations must be linearized and discretized such that a solution can be found for a certain (large) set of degrees of freedom. Large systems of linear equations need to be assembled and subsequently solved by appropriate direct or iterative methods. Like for mesh generation, basic linear algebra, with matrix/vector storage and operations, is nowadays most commonly outsourced to third-party software packages like PETSc  [4] and Trilinos  [5] (see, e.g.,  [6], [7], [8]).

With both mesh generation and linear algebra outsourced, the main job of CFD solvers boils down to linearization, discretization and assembly of the linear system of equations. This is by no means a trivial task as it requires, e.g., maps from computational cells to global degrees of freedom and connectivity of cells, facets and vertices. For parallel performance it is also necessary to distribute the mesh between processors and set up for inter-communication between compute nodes. Fortunately, much of the Message Passing Interface (MPI) is already handled by the providers of basic linear algebra. When it comes down to the actual discretization, the most common approaches are probably the finite volume method, which is very popular for fluid flow, finite differences or the finite element method.

FEniCS  [9] is a generic open source software framework that aims at automating the discretization of differential equations through the finite element method. FEniCS takes full advantage of specialized, reliable and robust third-party providers of computational software and interfaces to both PETSc and Trilinos for linear algebra and several third-party mesh generators. FEniCS utilizes the Unified Form Language (UFL,  [10]) and the FEniCS Form Compiler (FFC,  [11]) to automatically generate low-level C++ code that efficiently evaluates any equation formulated as a finite element variational form. The FEniCS user has to provide the high-level variational form that is to be solved, but does not need to actually perform any coding on the level of the computational cell, or element. A choice is made of finite element basis functions, and code is then generated for the form accordingly. There is a large library of possible finite elements to choose from and they may be combined both implicitly in a coupled manner or explicitly in a segregated manner—all at the same level of complexity to the user. The user never has to see the generated low-level code, but, this being an open source project, the code is wide open for inspection and even manual fine-tuning and optimization is possible.

In this paper we will describe the Navier–Stokes solver Oasis, that is written from scratch in Python, using building blocks from FEniCS and the PETSc backend. Our goal with this paper is to describe a code that is (i) short and easily understood, (ii) easily configured and (iii) as fast and accurate as state-of-the-art Navier–Stokes solvers developed entirely in low-level languages.

We assume that the reader has some basic knowledge of how to write simple solvers for partial differential equations using the FEniCS framework. Otherwise, reference is given to the online FEniCS tutorial  [12].

Section snippets

Fractional step algorithm

In Oasis we are solving the incompressible Navier–Stokes equations, optionally complemented with any number of passive or reactive scalars. The governing equations are thus ut+(u)u=ν2up+f,u=0,cαt+ucα=Dα2cα+fα, where u(x,t) is the velocity vector, ν the kinematic viscosity, p(x,t) the fluid pressure, cα(x,t) is the concentration of species α and Dα its diffusivity. Any volumetric forces (like buoyancy) are denoted by f(x,t) and chemical reaction rates (or other scalar sources) by fα

Variational formulations for the fractional step solver

The governing PDEs (4), (5), (6), (7) are discretized with the finite element method in space on a bounded domain ΩRd, with 2d3, and the boundary Ω. Trial and test spaces for the velocity components are defined as V={vH1(Ω):v=u0onΩ},Vˆ={vH1(Ω):v=0onΩ}, where u0 is a prescribed velocity component on part Ω of the boundary and H1(Ω) is the Sobolev space containing functions v such that v2 and |v|2 have finite integrals over Ω. Both the scalars and pressure use the same H1(Ω) space

Oasis

We now have all the variational forms that together constitute a fractional step solver for the Navier–Stokes equations, complemented with any number of scalar fields. We will now describe how the fractional step algorithm has been implemented in Oasis and discuss the design of the solver package. For installation of the software, see the user manual  [19]. Note that this paper refers to version 1.3 of the Oasis solver, which in turn is consistent with version 1.3 of FEniCS.

High-performance implementation

The naive solver described in the previous section is very easy to implement and understand, but for obvious reasons it is not very fast. For example, the entire coefficient matrix is reassembled each time step (see Fig. 4), even though it is only the convection term that changes in time. We will now explain how the same incremental pressure correction solver can be implemented efficiently, at the cost of losing some intuitiveness. The implementation of the high-performance solver described in

Verification of implementation

The fractional step algorithm implemented in NSfracStep is targeting transient flows in large-scale applications, with turbulent as well as laminar or transitional flow. It is not intended to be used as a steady state solver.4Oasis has previously been used to study, e.g., blood flow in highly complex intracranial aneurysms  [21], [22], where the results compare very well with, e.g., the spectral element code

Concluding notes on performance

The computational speed of any implicit, large-scale Navier–Stokes solver is determined by many competing factors, but most likely it will be limited by hardware and by routines for setting up (assembly) and solving for its linear algebra subsystems. In Oasis, and many comparable Navier–Stokes solvers, the linear algebra is performed through routines provided by a backend (here PETSc) and are thus arguably beyond our control. Accepting that we cannot do better than limits imposed by hardware

Acknowledgment

This work has been supported by a Center of Excellence grant from the Research Council of Norway to the Center for Biomedical Computing at Simula Research Laboratory.

References (33)

  • M.S. Alnæs et al.

    Unified form language: A domain-specific language for weak formulations of partial differential equations

    ACM Trans. Math. Software

    (2014)
  • R.C. Kirby et al.

    A compiler for variational forms

    ACM Trans. Math. Software

    (2006)
  • FEniCS tutorial. URL:...
  • M.A. Christon et al.

    Computational predictability of time-dependent natural convection flows in enclosures (including a benchmark solution)

    Internat. J. Numer. Methods Fluids

    (2012)
  • R.I. Issa

    Solution of the implicitly discretized fluid flow equations by operator-splitting

    J. Comput. Phys.

    (1985)
  • Ansys-fluent. URL:...
  • Cited by (78)

    View all citing articles on Scopus

    This paper and its associated computer program are available via the Computer Physics Communication homepage on ScienceDirect (http://www.sciencedirect.com/science/journal/00104655).

    View full text