Admin: New server guide

2026-04-15T10:00:38Z

New server guide

New page

**Warning: Installing and configuring software on a GPU server carries inherent risks. Incorrectly configured drivers or libraries can lead to system instability, data loss, or performance degradation. Proceed with caution and ensure you have backups of any critical data.**

## Installing TensorFlow on a GPU Server

This guide details the process of setting up TensorFlow with GPU acceleration on a Linux server, leveraging CUDA and cuDNN for optimal performance. This allows you to significantly speed up your machine learning model training and inference tasks.

### Prerequisites

Before you begin, ensure you have the following:

* A Linux server with an NVIDIA GPU installed. You can rent powerful GPU servers at [Immers Cloud](https://en.immers.cloud/signup/r/20241007-8310688-334/) with options ranging from $0.23/hr for inference to $4.74/hr for H200 instances.
* SSH access to your server with root or sudo privileges.
* Basic familiarity with the Linux command line.
* An internet connection on your server to download necessary packages.
* Consider the TensorFlow version you intend to use, as it dictates compatible CUDA and cuDNN versions. Check the [TensorFlow build configurations](https://www.tensorflow.org/install/source#gpu) for specific version requirements.

### Step 1: Install NVIDIA Drivers

The foundation for GPU acceleration is the correct NVIDIA driver.

1. **Check for existing drivers:**
```bash
nvidia-smi
```
If this command returns information about your GPU, a driver is likely already installed. Note the driver version.

2. **Install drivers (if not present or to update):** The method varies by Linux distribution. For Ubuntu/Debian:
```bash
sudo apt update
sudo apt install nvidia-driver-<version> # Replace <version> with the recommended version for your GPU
```
For CentOS/RHEL:
```bash
sudo yum update
sudo yum install epel-release
sudo yum install nvidia-driver-latest-dkms # Or a specific version
```
**Note:** It's often recommended to use the driver version provided by your distribution's repositories for better compatibility.

3. **Reboot your server:**
```bash
sudo reboot
```
After rebooting, run `nvidia-smi` again to confirm the driver is loaded correctly.

### Step 2: Install CUDA Toolkit

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model created by NVIDIA. It enables software to use NVIDIA GPUs for general-purpose processing.

1. **Download the CUDA Toolkit:** Visit the [NVIDIA CUDA Toolkit Archive](https://developer.nvidia.com/cuda-toolkit-archive) and select the version compatible with your desired TensorFlow version. Download the installer for your Linux distribution. For example, for CUDA 11.8 on Ubuntu:
```bash
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
chmod +x cuda_11.8.0_520.61.05_linux.run
```

2. **Run the installer:**
```bash
sudo sh cuda_11.8.0_520.61.05_linux.run
```
Follow the on-screen prompts. You can typically accept the defaults, but ensure you **do not** install the driver if you've already installed a newer or compatible one in Step 1.

3. **Update environment variables:** Add CUDA to your PATH and LD_LIBRARY_PATH.
```bash
echo 'export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
source ~/.bashrc
```
Replace `/usr/local/cuda-11.8/` with your actual CUDA installation path if different.

4. **Verify CUDA installation:**
```bash
nvcc --version
```
This should display the installed CUDA version.

### Step 3: Install cuDNN

cuDNN (CUDA Deep Neural Network library) is a GPU-accelerated library of primitives for deep neural networks. It significantly speeds up deep learning operations.

1. **Download cuDNN:** You need to register as an NVIDIA Developer to download cuDNN from the [NVIDIA cuDNN Download Page](https://developer.nvidia.com/cudnn). Choose the version compatible with your CUDA Toolkit version. Download the "cuDNN Library for Linux (x86_64)" zip file.

2. **Extract and copy files:** Upload the downloaded cuDNN zip file to your server. Then, extract and copy the files to your CUDA installation directory.
```bash
# Assuming cuDNN is downloaded to ~/Downloads/cudnn-linux-x86_64-8.x.x.x_cudaX.X-archive.tar.xz
tar -xvf ~/Downloads/cudnn-linux-x86_64-8.x.x.x_cudaX.X-archive.tar.xz
sudo cp cudnn-linux-x86_64-8.x.x.x_cudaX.X/include/cudnn*.h /usr/local/cuda-11.8/include/
sudo cp cudnn-linux-x86_64-8.x.x.x_cudaX.X/lib/libcudnn* /usr/local/cuda-11.8/lib64/
sudo chmod a+r /usr/local/cuda-11.8/include/cudnn*.h /usr/local/cuda-11.8/lib64/libcudnn*
```
Adjust paths and filenames according to your downloaded cuDNN version and CUDA installation.

### Step 4: Install TensorFlow with GPU Support

Now you can install TensorFlow. It's highly recommended to use a Python virtual environment.

1. **Install Python and pip (if not already installed):**
```bash
sudo apt update
sudo apt install python3 python3-pip python3-venv
```

2. **Create and activate a virtual environment:**
```bash
python3 -m venv tf_gpu_env
source tf_gpu_env/bin/activate
```

3. **Install TensorFlow:** Install the `tensorflow-gpu` package.
```bash
pip install tensorflow[and-cuda]
```
For older TensorFlow versions, you might need to install `tensorflow-gpu` directly:
```bash
pip install tensorflow-gpu
```
This command will download and install TensorFlow and its dependencies.

### Step 5: Verify TensorFlow GPU Installation

Test if TensorFlow can detect and use your GPU.

1. **Launch Python interpreter:**
```bash
python
```

2. **Run the verification script:**
```python
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
if tf.config.list_physical_devices('GPU'):
print("TensorFlow is using the GPU!")
else:
print("TensorFlow is NOT using the GPU.")
```

3. **Exit Python:**
```python
exit()
```
If the output shows "Num GPUs Available: 1" (or more, depending on your server) and "TensorFlow is using the GPU!", your setup is successful.

### Troubleshooting

* **`nvidia-smi` shows no devices or errors:** Ensure your NVIDIA drivers are correctly installed and loaded. Check the output of `dmesg` for driver-related errors.
* **`nvcc --version` command not found:** Verify that your CUDA toolkit installation path is correctly added to your `PATH` environment variable and that you've sourced your `.bashrc` file.
* **TensorFlow cannot detect GPU:**
* **Version Mismatch:** The most common issue is incompatibility between TensorFlow, CUDA, and cuDNN versions. Double-check the [TensorFlow build configurations](https://www.tensorflow.org/install/source#gpu) for the exact required versions.
* **Environment Variables:** Ensure `LD_LIBRARY_PATH` correctly points to your CUDA `lib64` directory.
* **cuDNN Installation:** Confirm that the cuDNN files were copied to the correct CUDA directories (`include` and `lib64`).
* **Permission Denied errors:** Ensure you are running commands with `sudo` when necessary, especially for driver installations and file copying.

### Related Articles

* [[GPU Server Management]]
* [[Optimizing Deep Learning Workloads]]
* [[Introduction to CUDA]]

<br>

---

**Disclosure:** This article may contain affiliate links. If you click on a link and make a purchase, we may receive a commission at no extra cost to you. This helps support our content creation.

[[Category:AI and GPU]]
[[Category:Server Administration]]
[[Category:Deep Learning]]

Installing TensorFlow on GPU Server - Revision history

Admin: New server guide