- Generated by
1.9.8
|
Basilisk CFD
Adaptive Cartesian mesh PDE framework
|
Go to the source code of this file.
Data Structures | |
| struct | MyUniform |
| struct | Shader |
| struct | kh_INT_s |
| struct | GridGPU |
Macros | |
| #define | gpu_grid ((GridGPU *)grid) |
| #define | str_append(dst, ...) str_append_array (dst, (const char *[]){__VA_ARGS__, NULL}) |
| #define | xstr(a) str(a) |
| #define | str(a) #a |
| #define | IS_EXTERNAL_CONSTANT(g) ((g)->constant && (g)->type == sym_INT && !(g)->data) |
| #define | EXTERNAL_NAME(g) (g)->global == 2 ? "_loc_" : "", (g)->name, (g)->reduct ? "_in_" : "" |
| #define | reset(...) reset_gpu (__VA_ARGS__) |
| #define | init_grid(n) gpu_init_grid(n) |
| #define | free_grid() gpu_free_grid() |
Typedefs | |
| typedef struct kh_INT_s | kh_INT_t |
Variables | |
| bool | on_cpu = false |
| static char | glsl_preproc [] |
| static scalar * | apply_bc_list |
| static int | bc_period_x = -1 |
| static int | bc_period_y = -1 |
| attribute | |
The stored attibute tracks where the up-to-date field is stored: | |
| double(* | boundary_right )(Point, Point, scalar, bool *) |
| double(* | boundary_top )(Point, Point, scalar, bool *) |
| double(* | boundary_bottom )(Point, Point, scalar, bool *) |
| #define free_grid | ( | void | ) | gpu_free_grid() |
| #define reset | ( | ... | ) | reset_gpu (__VA_ARGS__) |
| #define str_append | ( | dst, | |
| ... | |||
| ) | str_append_array (dst, (const char *[]){__VA_ARGS__, NULL}) |
| trace char * build_shader | ( | External * | externals, |
| const ForeachData * | loop, | ||
| const RegionParameters * | region, | ||
| const GLuint | nwg[2] | ||
| ) |
|
static |
|
static |
If this is a foreach_point() iteration, we draw a single point
This is a region
| bool gpu_end_stencil | ( | ForeachData * | loop, |
| const RegionParameters * | region, | ||
| External * | externals, | ||
| const char * | kernel | ||
| ) |
Definition at line 2045 of file grid.h.
Referenced by foreach_stencil_generic().
|
static |
|
static |
Definition at line 1130 of file grid.h.
Referenced by gpu_reduction().
|
static |
|
static |
We will directly apply boundary conditions to fields marked 'dirty' by automatic stencils.
We also apply boundary stencils so that input/output are also set properly for boundary conditions which may use external fields.
We make sure all fields marked dirty are also outputs.
This can be required if boundary conditions have been modified between loops.
For the Intel driver, it looks like the next line is necessary to ensure proper synchronisation of the compute shader and fragment shader (for example when using output_ppm() for interactive display). The nvidia driver somehow does not need this...
| attribute |
The stored attibute tracks where the up-to-date field is stored:
0: on both the CPU and GPU (i.e. synchronized). 1: on the CPU.
|
static |
Definition at line 408 of file grid.h.
Referenced by gpu_reduction().
The files in this directory implement Cartesian and Multigrid grids on Graphics Processing Units (GPUs). The ultimate goal is to allow running any Basilisk solver on GPUs without any modification to the original source code.
To do so the Basilisk preprocessor automatically generates "computation kernels" for each loop iterator. These kernels are then dynamically compiled (at runtime) by the OpenGL Shading Language (GLSL) compiler which is part of the (OpenGL) graphics card driver. If compilation is successful, the corresponding loop is performed on the GPU, otherwise the CPU is used. If this hybrid GPU/CPU hybrid mode of operation is used, synchronisation between the GPU and CPU memory is necessary and is done automatically.
OpenGL is an open standard (unlike e.g. CUDA) and is widely supported by graphics cards (with the notable exception of Apple graphics cards and some high-end "professional" Nvidia cards).
As described above, from a Basilisk perspective GPUs are just another type of grid. Selecting a "GPU grid" can simply be done using either
in the source code, or using the -grid command line option of qcc like this
The standard Basilisk Makefile also includes the handy recipe
which will compile and run code.c using the gpu/multigrid grid.
Note that for all this to work properly you first need to install the Basilisk GPU libraries.
Basilisk uses the GLFW library to configure and access the graphics card and OpenGL (version >= 4.3) for the rest. These libraries and the associated Basilisk libraries can be easily installed on Debian-like systems using
Note that you will also need the appropriate graphics card drivers (often proprietary for Nvidia). Note also that (reasonably high-end) laptop computers often have two graphics cards: a low-power, slow one and a high-power, fast one. To check which one you are currently using you can use something like
On my Dell XPS laptop I can switch to the (proprietary driver of the) fast Nvidia graphics card using
There are several test cases for GPUs you can try. For example
If this worked, you can then try a more interesting example
and also
GPUs are fast compared to CPUs because they use specialised hardware which relies on highly-parallel (tens of thousands of execution threads) asynchronous accesses to fast video memory channels. This imposes strong constraints on programs which can run efficiently on these systems, in particular regarding memory allocation and accesses. These constraints are reflected in the programming languages usable on GPUs, for example the OpenGL Shading Language (GLSL) which underlies the GPU grid in Basilisk.
GLSL is mostly a subset of C99 and the good news is that this subset happens to be what is used within most foreach loops in Basilisk (this is not a coincidence...). Thus, in many cases, simple and efficient Basilisk code will also run transparently and efficiently on GPUs.
There are obvious cases where foreach loops will not run on GPUs (see the next section). In theses cases, Basilisk will automatically switch to running the loop on the CPU and will synchronize the CPU and GPU memories. Note that this also means that the memory (i.e. scalar fields etc.) for a program is always allocated twice: once on the CPU and once on the GPU.
As an example, consider the following simple code
this can be run on the CPU using e.g.
If we now run on the GPU using
we get
Basilisk warns us that "printf" is not known in GLSL (at line 9) and that, as a consequence, the loop at line 8 (i.e. the second loop which includes "printf") was run on the CPU. Note that the first message is a "GLSL: error" but that the code still ran OK on the CPU. Note also that this error happened at runtime and not during compilation. That's because foreach loops are compiled dynamically at runtime by the graphics card driver.
Since GPUs have a very limited access to the operating system (i.e. only through the OpenGL interface) we cannot expect the loop including "printf" (or any other output) to run on the GPU. Note also that the second loop should be "serial" rather than parallel (see Parallel Programming). So we need to modify the code to
If we now recompile and run with make test.gpu.tst, the GLSL error and warnings are gone since we explicitely specified that the second loop should run on the CPU (and in serial mode).
Another way to specify that a given loop should run on the CPU (either in serial or parallel mode) is to use
Similarly one could use foreach (gpu) to force running on the GPU, in which case the GLSL warning above would become an error. This can be useful when debugging GPU codes and used in combination with the -cpu compilation flag which will force loops to run on the CPU by default.
In C99 variable-size arrays can be defined simply using for example
Since this relies on dynamic memory allocation on the stack, this is not possible in general in GLSL. The only cases where this will work is if the size of the array can be computed "statically" i.e. at the time the GLSL kernel is compiled. Furthermore, the GLSL compiler is strict (or not very clever) and a code looking like
will fail with an error like
To fix this one needs to write instead
Note that the size of the array must be a constant, but only at the time when the GLSL kernel is compiled. This allows using variable-size arrays also in GLSL, provided their size is constant within the kernel. For example the following code will work fine on the GPU, even if n changes between calls to func2().
Finally, using variable-sized arrays as function parameters, as done in func1() above, is not allowed in GLSL. To work around this strong limitation, the kernel preprocessor will expand calls to functions using variable-size arrays (using the macro engine). Note that this means that the function must respect the constraints applying to macros, in particular they can return only at the end of the function.
Inputs/Outputs: The only possible direct output on GPUs is the screen (see output_ppm on GPUs). All other inputs or outputs must go through the CPU. Complex memory allocation and access: There is no notion of "memory stack" on GPUs, all memory requirements are static and must be defined at compilation time. This means that variable/dynamical arrays, dynamic memory allocation (malloc etc.) and pointers do not exist on GPUs (and in GLSL). Using any of these in a foreach loop will give you a GLSL error as above. Limited support for function pointers: function pointers are fundamentally different from memory pointers. Basilisk includes limited support for function pointers i.e. what is sufficient for field attributes* implementing custom boundary conditions or coarsening/refinement functions. Using external libraries: GPUs cannot (obviously) use functions defined in external (CPU) libraries.
Aside from the fundamental constraints above, the current implementation also has the following limitations, some of which will be lifted in the (not-too-distant) future. In rough order of "lifting priority" these are:
Only 2D Cartesian and Multigrid grids for now: 3D multigrid will follow easily, quadtrees and octrees are more difficult. The maximum size of any scalar field is limited to what can be indexed using a 32-bits unsigned integer i.e. 2^32^ floats or 16 GB. Boundary conditions have only been implemented for 3x3 stencils. At this stage only a few solvers have been tested. Other solvers may or may not work. In particular surface tension will not work yet because the estimation of curvature relies on code which is not portable to GPUs. Only simple external types (int, bool, float, double, coord etc.) are currently supported: for example custom typedefs are not supported yet for external variables. Loop-local variables and functions can use (locally-defined) custom types. Loop-local lists of scalars have very limited support (but are not used much anyway): external loops support is OK. Double precision (64-bits floats) is supported by Basilisk (use ‘CFLAGS=’-DDOUBLE_PRECISION'`) but depends on the (often limited) support by graphics cards and their drivers. Note also that using single precision can have an important impact on the convergence and accuracy of multigrid solvers.
To convince yourself that GPUs are worth the trouble, see the GPUBenchmarks": speedups of one to two orders of magnitude compared to CPUs are achievable. To maximize performance, here are a few tips and observations: Make sure that you are using the correct graphics card and driver (see @ref "installation" "glxinfo" above). GPUs are highly parallel so will only provide speedups for large enough simulations (e.g. larger than 128^2^), increasingly so as resolution is increased. Frequent CPU/GPU memory synchronisation will kill performance. Be careful to control how often you output data for example, much more so than when running on CPUs. An exception is <a href="output.h" target="_blank" >graphical outputs</a> which are much cheaper on GPUs and can be done often with little overhead. Loops done on the CPU within e.g. timestep iterations will generally kill performance. Use <a href="/src/README.trace" target="_blank" >built-in profiling</a> to check where time is spent. Use the <tt>-DTRACE=3</tt> compilation flag to get profiling information at the level of foreach loops. @subsection autotoc_md149 Bugs Trying to use more than one shader storage buffer (SSBO) with Intel drivers does not seem work. This limits the amount of available video memory to a single SSBO with a maximum size which is usually 2GB (or less). @subsection autotoc_md150 See also <a href="/src/test/READMEgpu-tests" target="_blank" >Test cases on GPUs</a> @ref "/home/runner/work/basilisk-docs/basilisk-docs/basilisk/src/grid/gpu/Benchmarks.md" "GPU benchmarks" <a href="/src/ast/kernels.c" target="_blank" >Computation kernels
1.9.8