Code Structure

This page is a map of the SU2 source code for new developers. It shows:

• where things live;
• how a simulation runs through the classes of SU2_CFD;
• which files to change for common tasks.

For the full class documentation, see the Code Reference (Doxygen). Class names on this page link to their Doxygen page.

• Top-level folders
• Inside SU2_CFD
• How SU2_CFD runs
• Main classes
• Linear algebra
• Containers and toolboxes
• Parallelism: MPI, OpenMP and vectorization
• Algorithmic differentiation
• Where do I change…

Top-level folders

Inside SU2_CFD

Headers are in SU2_CFD/include/<folder> and sources are in SU2_CFD/src/<folder>. The folders gradients and numerics_simd exist only under include.

SU2_CFD/src/SU2_CFD.cpp holds main(). SU2_CFD/src/python_wrapper_structure.cpp holds the driver functions that the Python wrapper uses.

How SU2_CFD runs

The tree below shows the calls of one SU2_CFD run, from main() down to the numerics.

main()                                         SU2_CFD/src/SU2_CFD.cpp
 |-- CConfig: read the config file (number of zones, solver kind)
 |-- create the driver:
 |     -d (dry run)            -> CDummyDriver
 |     single zone             -> CSinglezoneDriver / CDiscAdjSinglezoneDriver
 |     multizone               -> CMultizoneDriver  / CDiscAdjMultizoneDriver
 |     harmonic balance        -> CHBDriver
 |   The CDriver constructor creates everything:
 |     PreprocessInput, PreprocessOutput, InitializeGeometry, InitializeSolver,
 |     InitializeNumerics, InitializeIntegration, PreprocessIteration, ...
 |
 |-- driver->StartSolver()
 |     loop over time steps (CSinglezoneDriver):
 |       Preprocess -> Run -> Postprocess -> Update -> Monitor -> Output
 |       Run() calls CIteration::Solve()
 |         loop over inner iterations:
 |           CIteration::Iterate()                  e.g. CFluidIteration
 |             CIntegration::MultiGrid_Iteration()  flow (CMultiGridIntegration)
 |             CIntegration::SingleGrid_Iteration() turbulence, species, ... (CSingleGridIntegration)
 |               CSolver::Preprocessing()           primitives, gradients, limiters
 |               CSolver::SetTime_Step()
 |               CIntegration::Space_Integration()
 |                 CSolver::Upwind_Residual() or Centered_Residual()
 |                 CSolver::Viscous_Residual(), Source_Residual()
 |                   -> CNumerics::ComputeResidual() on each edge or point
 |                 CSolver::BC_...() on each marker
 |               CIntegration::Time_Integration()
 |                 CSolver::ImplicitEuler_Iteration() -> CSysSolve with CSysMatrix, CSysVector
 |               CSolver::Postprocessing()
 |           CIteration::Monitor()  -> COutput::SetHistoryOutput()
 |           CIteration::Output()   -> COutput::SetResultFiles()
 |
 |-- driver->Finalize()

Notes:

• CMultiGridIntegration::MultiGrid_Iteration runs the multigrid cycle. On each grid level it does the same steps as above: preprocessing, space and time integration, postprocessing.
• CFluidIteration::Iterate solves the flow first. Then CommonAuxiliarySolvers runs SingleGrid_Iteration for the transition, turbulence and species solvers. The coupling is segregated: each solver is solved on its own, one after the other.
• CIntegration::Space_Integration and CIntegration::Time_Integration are in SU2_CFD/src/integration/CIntegration.cpp. Read them to see exactly which solver functions are called, and in which order.
• The discrete adjoint drivers (CDiscAdjSinglezoneDriver, CDiscAdjMultizoneDriver) record one primal iteration on the AD tape. The tape is the stored list of operations that AD later runs backwards to get derivatives. The drivers then evaluate the tape many times (AD::ComputeAdjoint) to solve the adjoint fixed-point problem.

The containers in the driver

The driver holds arrays of pointers to all objects. It passes them down to the iterations and integrations.

The solver indices and term indices are defined in Common/include/option_structure.hpp.

Main classes

Drivers

• CDriverBase: common base of the SU2_CFD and SU2_DEF drivers. It also holds the functions that the Python wrapper uses.
• CDriver: creates geometry, solvers, numerics, integrations, iterations and outputs.
• CSinglezoneDriver, CMultizoneDriver: primal problems. The multizone driver couples the zones with block Gauss-Seidel or Jacobi iterations. It transfers data between zones through CInterface objects.
• CDiscAdjSinglezoneDriver, CDiscAdjMultizoneDriver: discrete adjoint problems.
• CHBDriver (child of CFluidDriver): harmonic balance.
• CDummyDriver: dry run (SU2_CFD -d). It prints the available output fields.

Iterations

CIteration runs one iteration of one physics in one zone (Preprocess, Iterate, Postprocess, Monitor, Output, Solve). CIterationFactory creates the children:

• CFluidIteration, with the children CTurboIteration, CPBFluidIteration (pressure-based incompressible), CFEMFluidIteration, CHeatIteration and CAdjFluidIteration (continuous adjoint).
• CFEAIteration: structural analysis.
• CDiscAdjFluidIteration, CDiscAdjHeatIteration, CDiscAdjFEAIteration: discrete adjoint.

Integrations

CIntegration holds Space_Integration and Time_Integration. CIntegrationFactory picks the child from the solver type (INTEGRATION_TYPE in SU2_CFD/include/solvers/CSolverFactory.hpp):

• CMultiGridIntegration: finite volume flow solvers (also with MGLEVEL= 0).
• CSingleGridIntegration: turbulence, transition, species, heat and radiation solvers.
• CNewtonIntegration: flow solvers with NEWTON_KRYLOV= YES (primal only).
• CStructuralIntegration: structural (FEA) solver.
• CFEM_DG_Integration: discontinuous Galerkin flow solver.
• The discrete adjoint solvers use the base CIntegration.

Solvers

CSolver solves one set of equations on one grid level. It:

• computes the residuals;
• applies the boundary conditions (BC_... functions);
• owns the linear system (Jacobian, LinSysRes, LinSysSol, System).

CSolverFactory (SU2_CFD/src/solvers/CSolverFactory.cpp) decides which solvers to create for each SOLVER option.

Main families:

• Finite volume flow: CFVMFlowSolverBase is a template with the shared code. Its children are CEulerSolver / CNSSolver (compressible), CIncEulerSolver / CIncNSSolver (incompressible) and CNEMOEulerSolver / CNEMONSSolver (non-equilibrium).
• Scalar transport: CScalarSolver (template), with:
  • CTurbSolver → CTurbSASolver, CTurbSSTSolver, CTransLMSolver;
  • CSpeciesSolver → CSpeciesFlameletSolver;
  • CHeatSolver;
  • CPoissonSolver.
• Others: CRadP1Solver, CFEASolver, CMeshSolver (mesh deformation), CFEM_DG_EulerSolver / CFEM_DG_NSSolver, CBaselineSolver.
• Adjoint: CDiscAdjSolver (discrete adjoint of any solver), CDiscAdjFEASolver, CDiscAdjMeshSolver, and the continuous adjoint solvers CAdjEulerSolver, CAdjNSSolver, CAdjTurbSolver.
• CTemplateSolver: an empty starting point for a new solver.

Variables

CVariable stores the data of all points of one solver in one object (solver->GetNodes()). Each quantity is its own container, indexed by point: Solution, Solution_Old, Gradient, Limiter, Delta_Time, … The matrices are row-major (su2activematrix), so the values of one point sit next to each other in memory. You access a value by point index, for example nodes->GetSolution(iPoint, iVar).

The children follow the solvers:

• CFlowVariable → CEulerVariable → CNSVariable;
• CIncEulerVariable → CIncNSVariable;
• CScalarVariable → CTurbVariable → CTurbSAVariable, CTurbSSTVariable;
• and so on.

Numerics

CNumerics computes a flux across one edge, or a source term at one point. ComputeResidual(config) returns a ResidualType with the residual and the Jacobians for the two points of the edge.

The solver uses it in this way:

1. It loops over the edges or points.
2. It sets the inputs of the numerics object.
3. It calls ComputeResidual.
4. It adds the result to LinSysRes and Jacobian.

CDriver::InitializeNumerics (SU2_CFD/src/drivers/CDriver.cpp) creates the numerics objects. It uses the helpers InstantiateTurbulentNumerics, InstantiateTransitionNumerics and InstantiateSpeciesNumerics.

Geometry

CGeometry holds the mesh and the dual grid. The dual grid is the set of control volumes, one around each mesh point, that the edge-based finite volume method uses. The main members are:

• nodes (CPoint): coordinates, volumes and neighbors of all points.
• edges (CEdge): the two points and the normal of each edge.
• vertex[iMarker][iVertex] (CVertex): boundary points and their normals.
• InitiateComms / CompleteComms: halo exchange between MPI ranks (see MPI below).

Children:

• CPhysicalGeometry: the mesh read from file, partitioned with ParMETIS. The file is in SU2 or CGNS format; the readers are in Common/include/geometry/meshreader.
• CMultiGridGeometry: the coarse levels. Each one is made by agglomeration, which merges neighboring control volumes of the finer level into larger ones.
• CDummyGeometry: empty geometry for the dry run.
• CMeshFEM_DG: geometry of the DG solver (Common/include/fem).

Grid movement (CSurfaceMovement, CVolumetricMovement, CFreeFormDefBox) is in Common/include/grid_movement. Interpolation between non-matching zone interfaces (CInterpolator and its children) is in Common/include/interface_interpolation.

Configuration

CConfig (Common/include/CConfig.hpp, Common/src/CConfig.cpp) reads the configuration file and stores every option. There is one CConfig per zone. There is also a driver_config, read from the main configuration file. It holds the settings of the driver, for example for multizone problems.

• CConfig::SetConfig_Options declares the options with addDoubleOption, addBoolOption, addEnumOption, addStringListOption, …
• The enums, and the maps from option strings to enum values, are in Common/include/option_structure.hpp. The option parser classes are in Common/include/option_structure.inl.
• CConfig::SetPostprocessing and CConfig::SetMarkers compute the derived settings.

See Adding a Config Option and the Option Reference.

Output

COutput writes the screen output, the history file, and the volume and surface files. Each solver kind has a child class, created by COutputFactory. Examples are CFlowCompOutput, CFlowIncOutput, CHeatOutput and CElasticityOutput. CMultizoneOutput combines the zones of a multizone run. The file writers (Paraview, Tecplot, CGNS, CSV, SU2 restart, …) are in SU2_CFD/include/output/filewriter.

See Adding Output Fields for the details.

Linear algebra

Files: Common/include/linear_algebra and Common/src/linear_algebra.

• CSysMatrix: block sparse matrix, with one block per pair of neighbor points. Implicit solvers use it for the Jacobian.
• CSysVector: block vector, for the residual and the solution update.
• CSysSolve: Krylov solvers (FGMRES_LinSolver, RFGMRES_LinSolver, FGCRODR_LinSolver, BCGSTAB_LinSolver, CG_LinSolver, Smoother_LinSolver). Solve reads the method and the preconditioner from the configuration. For the primal flow these are LINEAR_SOLVER and LINEAR_SOLVER_PREC. The mesh, adjoint and other problems have their own options.
• CPreconditioner and its children CJacobiPreconditioner, CILUPreconditioner, CLU_SGSPreconditioner, CLineletPreconditioner, CPastixPreconditioner. CPreconditioner::Create builds the chosen one.
• CMatrixVectorProduct: the product used inside the Krylov solvers.

Each solver has the members Jacobian (CSysMatrix<su2mixedfloat>), LinSysRes, LinSysSol and System (CSysSolve<su2mixedfloat>). su2mixedfloat is float when SU2 is built with -Denable-mixedprec=true. See also Linear Solvers and Preconditioners.

Containers and toolboxes

Containers (Common/include/containers):

• C2DContainer (C2DContainer.hpp): aligned 2D storage. The same file defines the common type aliases:
  • su2vector<T>, su2matrix<T> (row-major), ColMajorMatrix<T>;
  • su2activevector, su2activematrix (of su2double);
  • su2passivevector, su2passivematrix (of passivedouble).
• container_decorators.hpp: C3DDoubleMatrix (also called CVectorOfMatrix, used for gradients) and CMatrixView.
• CLookUpTable, CFileReaderLUT, CTrapezoidalMap: look-up tables. The flamelet fluid model (CFluidFlamelet) and the data-driven fluid model (CDataDrivenFluid) use them.
• CVertexMap: maps indices from a large range (for example all points) to a smaller one (for example the boundary vertices).
• CFastFindAndEraseQueue: a queue that can also remove items from the middle quickly.
• CPyWrapperMatrixView: gives the Python wrapper access to solver data such as su2activematrix.

Toolboxes (Common/include/toolboxes):

• geometry_toolbox.hpp (vector operations);
• graph_toolbox.hpp (sparse patterns, coloring);
• printing_toolbox.hpp (PrintingToolbox::CTablePrinter);
• ndflattener.hpp, CSquareMatrixCM, CSymmetricMatrix, C1DInterpolation, CLinearPartitioner, CQuasiNewtonInvLeastSquares;
• the verification solutions in toolboxes/MMS.

Parallelism: MPI, OpenMP and vectorization

Files: Common/include/parallelization.

MPI. Use the SU2_MPI wrapper from mpi_structure.hpp, never MPI directly. SU2_MPI is CBaseMPIWrapper, or CMediMPIWrapper (MeDiPack) in AD builds. This way MPI calls work with su2double.

SU2 partitions the mesh at start-up. Each rank also stores halo points: copies of the points next to its partition that belong to other ranks. To update the halo values, call InitiateComms / CompleteComms of CGeometry and CSolver. An MPI_QUANTITIES value says what to send.

OpenMP. Do not include omp.h. Include omp_structure.hpp and use its macros. They compile to nothing when OpenMP is off (-Dwith-omp=false), and to OpDiLib calls in AD builds:

• SU2_OMP_PARALLEL … END_SU2_OMP_PARALLEL, SU2_OMP_PARALLEL_(ARGS)
• SU2_OMP_FOR_STAT(CHUNK), SU2_OMP_FOR_DYN(CHUNK), SU2_OMP_FOR_(ARGS) … END_SU2_OMP_FOR
• SU2_OMP_MASTER … END_SU2_OMP_MASTER, SU2_OMP_CRITICAL … END_SU2_OMP_CRITICAL, SU2_OMP_BARRIER, SU2_OMP_ATOMIC
• SU2_OMP_SAFE_GLOBAL_ACCESS(...): run code on one thread, with barriers around it
• SU2_OMP_SIMD
• helpers: omp_get_num_threads(), omp_get_thread_num(), roundUpDiv, atomicAdd

When the solver supports it (CSolver::GetHasHybridParallel()), the integrations open one parallel region for a whole iteration.

Edge loops write to both points of an edge. Two threads could then write to the same point at the same time (a race condition). To avoid this, SU2 groups the edges by color (CGeometry::GetEdgeColoring, option EDGE_COLORING_GROUP_SIZE). If the coloring is poor, the flow and scalar solvers first store the edge fluxes, then sum them for each point (ReducerStrategy, EdgeFluxes).

Set the number of threads per rank with SU2_CFD -t <threads>.

Vectorization. SIMD (single instruction, multiple data) instructions let the CPU apply one operation to several values at once.

• vectorization.hpp defines simd::Array, a short fixed-size array with arithmetic operators. The compiler turns its loops into SIMD instructions. special_vectorization.hpp has explicit versions for some instruction sets.
• CNumericsSIMD (SU2_CFD/include/numerics_simd) uses it to compute the fluxes of several edges at once.
• CEulerSolver uses the vectorized versions for all centered schemes. It also uses them for the ROE and MSW upwind schemes, with an ideal gas and no low-Mach correction. See CEulerSolver::Centered_Residual and CEulerSolver::Upwind_Residual.
• The build option cpu-arch (default native) sets the target instruction set.

Algorithmic differentiation

SU2 uses operator overloading AD with CoDiPack (externals/codi).

• su2double (Common/include/code_config.hpp) is the type for all values that may need derivatives. What it is depends on the build:
  • in a normal build, it is passivedouble (normally double);
  • with -Denable-autodiff=true (builds SU2_CFD_AD, SU2_DOT_AD), it is a CoDiPack reverse type;
  • with -Denable-directdiff=true (builds SU2_CFD_DIRECTDIFF), it is a forward type.
• passivedouble is never differentiated. Use SU2_TYPE::GetValue(x) to get the plain value of a su2double.
• The AD namespace (Common/include/basic_types/ad_structure.hpp) wraps the tape:
  • StartRecording, StopRecording, RegisterInput, RegisterOutput, ComputeAdjoint, ClearAdjoints;
  • preaccumulation (StartPreacc, SetPreaccIn, SetPreaccOut, EndPreacc). Preaccumulation stores only the Jacobian of a code section on the tape, instead of all its operations. This makes the tape smaller.

  The functions do nothing in non-AD builds, so the code can always call them.

• MeDiPack (externals/medi) differentiates MPI calls. OpDiLib (externals/opdi) differentiates OpenMP regions.

See Advanced AD Techniques for more.

Where do I change…

New boundary condition

1. Add the kind to enum BC_TYPE in Common/include/option_structure.hpp.
2. In Common/src/CConfig.cpp, add the marker option in CConfig::SetConfig_Options, and set Marker_CfgFile_KindBC in CConfig::SetMarkers.
3. Add a virtual BC_... function to CSolver (SU2_CFD/include/solvers/CSolver.hpp).
4. Implement it in the solvers that need it, for example CEulerSolver, CIncEulerSolver, and the scalar solvers through CScalarSolver.
5. Call it from the switch on GetMarker_All_KindBC in CIntegration::Space_Integration.

New convective scheme

1. Add the value to enum class UPWIND (or CENTERED) and to Upwind_Map in option_structure.hpp.
2. Write the numerics class in SU2_CFD/include/numerics/flow/convection/ and SU2_CFD/src/numerics/flow/convection/.
3. Create it in CDriver::InitializeNumerics (switch on GetKind_Upwind_Flow()).
4. Optional, for a compressible ideal gas only: write a vectorized version in SU2_CFD/include/numerics_simd/flow/convection/upwind.hpp. Create it in createUpwindIdealNumerics (CNumericsSIMD.cpp) and select it in CEulerSolver::Upwind_Residual.

New turbulence model source term

• The source terms are in SU2_CFD/include/numerics/turbulent/turb_sources.hpp: CSourceBase_TurbSA and SAFactory for the Spalart-Allmaras (SA) model, CSourcePieceWise_TurbSST for the SST (shear stress transport) model.
• CDriver::InstantiateTurbulentNumerics creates them. CTurbSASolver::Source_Residual / CTurbSSTSolver::Source_Residual use them.
• Users choose the model variants with SA_OPTIONS / SST_OPTIONS (enum class SA_OPTIONS, SST_OPTIONS in option_structure.hpp).

New output field

Change the output class of the solver in SU2_CFD/src/output/, or CFlowOutput for turbulence and species fields. See Adding Output Fields.

New config option

Add the option in CConfig::SetConfig_Options, a member and a getter in CConfig.hpp, and an entry in config_template.cfg. See Adding a Config Option.

New solver

Start from CTemplateSolver (SU2_CFD/include/solvers/CTemplateSolver.hpp) and the template numerics (CConvective_Template, CViscous_Template, CSource_Template in SU2_CFD/include/numerics/template.hpp). Then add:

1. a value in enum class MAIN_SOLVER and Solver_Map (option_structure.hpp);
2. the solver in CSolverFactory::CreateSolverContainer and CreateSubSolver, with its INTEGRATION_TYPE;
3. its numerics in CDriver::InitializeNumerics;
4. an iteration in CIterationFactory::CreateIteration;
5. an output class in COutputFactory::CreateOutput.

Tests and style

When you change code, add or update a regression test (see Running Regression Tests) or a unit test (see Writing Unit Tests). Follow the Style Guide.

Last checked against SU2 develop 11c177b904.