What is GPU memory used for and how to free it?

Question

NVIDIA

Almost all of the video card's memory is used when there are no processes.
I work with Jupyter Notebook, I haven't restarted the kernel yet.

nvidia-smi shows Memory usage: 8001MiB / 8118MiB

Model training crashes with an out of memory error.

What is the memory clogged with and how to free it?

docker run --gpus all --rm nvidia/cuda nvidia-smi
Sat Feb 22 11:19:19 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:09:00.0 Off |                  N/A |
|  0%   47C    P2    41W / 200W |   8001MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Answer 1

Restarting the kernel of my Jupyter Notebook freed up all the GPU memory.