CUDA assertion error binary_cross_entropy loss

2022-09-02 20:25:24 浏览数 (2)

A CUDA assertion error pops up when setting --no_lsgan. It seems it's because there are negative values thrown into the nn.BCELoss(). Get's fixed applying nn.BCEWithLogitsLoss() instead.

代码语言:javascript复制
(...)
/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::tuple<float, float, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [16,0,0], thread: [31,0,0] Assertion `input >= 0. && input <= 1.` failed.
CUDA error after cudaEventDestroy in future dtor: device-side assert triggeredTraceback (most recent call last):
  File "train.py", line 56, in <module>
    Variable(data['image']), Variable(data['feat']), infer=save_fake)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 66, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/blanca/project/wip/pix2pixHD-master/models/pix2pixHD_model.py", line 158, in forward
    loss_D_fake = self.criterionGAN(pred_fake_pool, False)
  File "/blanca/project/wip/pix2pixHD-master/models/networks.py", line 110, in __call__
    loss  = self.loss(pred, target_tensor)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 372, in forward
    size_average=self.size_average)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1179, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, size_average)
RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THC/generic/THCStorage.c line=184 error=59 : device-side assert triggered
terminate called after throwing an instance of 'std::runtime_error'
  what():  cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THC/generic/THCStorage.c:184
Aborted (core dumped)

0 人点赞