Skip to main content

Job Resource Management

note

This feature is supported in the Enterprise Plan only.

In just a few steps, you'll have a GitLab Runner up and running. Let's get started!

How to Request Nvidia GPU

To leverage GPUs for your jobs in Puzl, you need to specify the KUBERNETES_GPU_REQUEST variable in your .gitlab-ci.yml:

variables:
  KUBERNETES_GPU_REQUEST: 2

How to Distribute GPUs Across Containers

Puzl allows you to distribute GPU resources across multiple containers within a single job. By default, every container in a job will have access to all the requested GPUs. However, you can restrict or specify which GPUs are visible to each container using the NVIDIA_VISIBLE_DEVICES variable.

For example, in the scenario below:

variables:
  KUBERNETES_GPU_REQUEST: 3
  NVIDIA_VISIBLE_DEVICES: "0,1"
services:
  - name: nvidia/gpu-inference
    variables:
      NVIDIA_VISIBLE_DEVICES: "2"

The main build container will have access to GPUs with indexes 0 and 1, while the nvidia/gpu-inference service container will only have access to GPU with index 2.

Ensuring Correct GPU Allocation

There are a few things to remember when distributing GPUs across containers:

  1. If you want to ensure a container does not have access to any GPUs, set the NVIDIA_VISIBLE_DEVICES variable to an empty string "" for that container.

  2. The sum of GPUs indicated in the NVIDIA_VISIBLE_DEVICES environment variables across all containers should match the total GPUs requested in KUBERNETES_GPU_REQUEST.

How to Request Exclusive Cores

To request exclusive cores, you need:

  1. Set spec.pipelines.resources.dropContainerResourceLimits in your GitLabRunnerClaim resource to false.
  2. Set the CPU and memory request variables with the same values as their corresponding limit variables in your .gitlab-ci.yml:
variables:
  # In this set of variables, you are requesting the amount of resources you need for your build container. 
  # https://docs.gitlab.com/runner/executors/kubernetes.html#overwrite-container-resources
  KUBERNETES_CPU_REQUEST: "3"
  KUBERNETES_CPU_LIMIT: "3"
  KUBERNETES_MEMORY_REQUEST: "8Gi"
  KUBERNETES_MEMORY_LIMIT: "8Gi"

  # Setting 1 CPU and 1 GB of memory for helper container is enough in the most cases.
  KUBERNETES_HELPER_CPU_REQUEST: "1"
  KUBERNETES_HELPER_CPU_LIMIT: "1"
  KUBERNETES_HELPER_MEMORY_REQUEST: "1Gi"
  KUBERNETES_HELPER_MEMORY_LIMIT: "1Gi"

  # You cannot designate resources for each specific service. 
  # Please, avoid setting excessively low values for memory and CPU for services, 
  # as doing so can result in performance issues with caching or out-of-memory errors due to insufficient resources.
  # Find more information about services here: https://docs.gitlab.com/ee/ci/services/
  KUBERNETES_SERVICE_CPU_REQUEST: "2"
  KUBERNETES_SERVICE_CPU_LIMIT: "2"
  KUBERNETES_SERVICE_MEMORY_REQUEST: "4Gi"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "4Gi"

You can specify the resource allocations above for any individual job.

Notice

To allocate your job with exclusive CPU cores, you must always request both CPU and memory resources in the job config.