We Are Going To Discuss About **CUDA out of memory error, cannot reduce batch size**. So lets Start this Python Article.

## CUDA out of memory error, cannot reduce batch size

**How to solve CUDA out of memory error, cannot reduce batch size**As long as a single sample can fit into GPU memory, you do not have to reduce the

*effective*batch size: you can do gradient accumulation.

Instead of updating the weights after every iteration (based on gradients computed from a too-small mini-batch) you can*accumulate*the gradients for several mini-batches and only when seeing enough examples, only then updating the weights.

This is nicely explained in this video.

Effectively, your training code would look something like this.

Suppose your large batch size is`large_batch`

, but can only fit`small_batch`

into GPU memory, such that`large_batch = small_batch * k`

.

Then you want to update the weights every`k`

iterations:`train_data = DataLoader(train_set, batch_size=small_batch, ...) opt.zero_grad() # this signifies the start of a large_batch for i, (x, y) in train_data: pred = model(x) loss = criterion(pred, y) loss.backward() # gradeints computed for small_batch if (i+1) % k == 0 or (i+1) == len(train_data): opt.step() # update the weights only after accumulating k small batches opt.zero_grad() # reset gradients for accumulation for the next large_batch`

**CUDA out of memory error, cannot reduce batch size**As long as a single sample can fit into GPU memory, you do not have to reduce the

*effective*batch size: you can do gradient accumulation.

Instead of updating the weights after every iteration (based on gradients computed from a too-small mini-batch) you can*accumulate*the gradients for several mini-batches and only when seeing enough examples, only then updating the weights.

This is nicely explained in this video.

Effectively, your training code would look something like this.

Suppose your large batch size is`large_batch`

, but can only fit`small_batch`

into GPU memory, such that`large_batch = small_batch * k`

.

Then you want to update the weights every`k`

iterations:`train_data = DataLoader(train_set, batch_size=small_batch, ...) opt.zero_grad() # this signifies the start of a large_batch for i, (x, y) in train_data: pred = model(x) loss = criterion(pred, y) loss.backward() # gradeints computed for small_batch if (i+1) % k == 0 or (i+1) == len(train_data): opt.step() # update the weights only after accumulating k small batches opt.zero_grad() # reset gradients for accumulation for the next large_batch`

## Solution 1

As long as a single sample can fit into GPU memory, you do not have to reduce the *effective* batch size: you can do gradient accumulation.

Instead of updating the weights after every iteration (based on gradients computed from a too-small mini-batch) you can *accumulate* the gradients for several mini-batches and only when seeing enough examples, only then updating the weights.

This is nicely explained in this video.

Effectively, your training code would look something like this.

Suppose your large batch size is `large_batch`

, but can only fit `small_batch`

into GPU memory, such that `large_batch = small_batch * k`

.

Then you want to update the weights every `k`

iterations:

```
train_data = DataLoader(train_set, batch_size=small_batch, ...)
opt.zero_grad() # this signifies the start of a large_batch
for i, (x, y) in train_data:
pred = model(x)
loss = criterion(pred, y)
loss.backward() # gradeints computed for small_batch
if (i+1) % k == 0 or (i+1) == len(train_data):
opt.step() # update the weights only after accumulating k small batches
opt.zero_grad() # reset gradients for accumulation for the next large_batch
```

Original Author Shai Of This Content

## Solution 2

Shai’s answer is suitable, but I want to offer another solution. Recently, I’ve been observing awesome results from Nvidia AMP – Automatic Mixed Precision, which is a nice combination of the advantages of fp16 vs fp32. A positive side effect is that it significantly speeds up training as well.

It’s only a single line of code in tensorflow: `opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)`

More details here

You can also stack AMP with Shai’s solution.

Original Author Stanley Zheng Of This Content

## Conclusion

So This is all About **This Tutorial.** Hope This Tutorial Helped You. Thank You.