Loss values stay the same for every iteration, with extremely large image sizes

[ProGamerGov]

When using -image_size 2432, -image_size 2560, and -image_size 2816, with -backend cudnn, -optimizer adam, and -style_scale 0.5, the loss values seem to remain the same in every iteration. Lower image sizes don't seem to suffer from this issue.

I also used -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12, which is the most efficient set of parameters for multiple GPUs that I have come across thus far.

ubuntu@ip-Address:~/neural-style$ ./multires_1.sh
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 538683157
Successfully loaded models/VGG16_SOD_finetune.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8-SOD100: 1 1 4096 100
Setting up style layer          2       :       relu1_1
Setting up style layer          7       :       relu2_1
Setting up style layer          12      :       relu3_1
Setting up style layer          19      :       relu4_1
Setting up content layer        21      :       relu4_2
Setting up style layer          26      :       relu5_1
Capturing content targets
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> output]
  (1): nn.GPU(1) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (2): nn.GPU(2) @ nn.Sequential {
    [input -> (1) -> output]
    (1): nn.StyleLoss
  }
  (3): nn.GPU(3) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  }
  (4): nn.GPU(4) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.ReLU
    (2): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (5): nn.GPU(5) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (6): nn.GPU(6) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> output]
    (1): nn.StyleLoss
    (2): cudnn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
    (3): cudnn.ReLU
  }
  (7): nn.GPU(7) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (8): nn.GPU(8) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> output]
    (1): cudnn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
    (3): nn.StyleLoss
    (4): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (5): cudnn.ReLU
    (6): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (7): cudnn.ReLU
    (8): cudnn.SpatialMaxPooling(2x2, 2,2)
    (9): cudnn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
    (10): cudnn.ReLU
    (11): nn.StyleLoss
    (12): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (13): cudnn.ReLU
    (14): nn.ContentLoss
    (15): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (16): cudnn.ReLU
    (17): cudnn.SpatialMaxPooling(2x2, 2,2)
    (18): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (19): cudnn.ReLU
    (20): nn.StyleLoss
  }
}
Capturing style target 1
Capturing style target 2
Capturing style target 3
Capturing style target 4
Capturing style target 5
Capturing style target 6
Capturing style target 7
Capturing style target 8
Running optimization with ADAM
Iteration 50 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 100 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 150 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 200 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 538683157
Successfully loaded models/VGG16_SOD_finetune.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8-SOD100: 1 1 4096 100
Setting up style layer          2       :       relu1_1
Setting up style layer          7       :       relu2_1
Setting up style layer          12      :       relu3_1
Setting up style layer          19      :       relu4_1
Setting up content layer        21      :       relu4_2
Setting up style layer          26      :       relu5_1
Capturing content targets
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> output]
  (1): nn.GPU(1) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (2): nn.GPU(2) @ nn.Sequential {
    [input -> (1) -> output]
    (1): nn.StyleLoss
  }
  (3): nn.GPU(3) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  }
  (4): nn.GPU(4) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.ReLU
    (2): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (5): nn.GPU(5) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (6): nn.GPU(6) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> output]
    (1): nn.StyleLoss
    (2): cudnn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
    (3): cudnn.ReLU
  }
  (7): nn.GPU(7) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (8): nn.GPU(8) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> output]
    (1): cudnn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
    (3): nn.StyleLoss
    (4): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (5): cudnn.ReLU
    (6): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (7): cudnn.ReLU
    (8): cudnn.SpatialMaxPooling(2x2, 2,2)
    (9): cudnn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
    (10): cudnn.ReLU
    (11): nn.StyleLoss
    (12): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (13): cudnn.ReLU
    (14): nn.ContentLoss
    (15): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (16): cudnn.ReLU
    (17): cudnn.SpatialMaxPooling(2x2, 2,2)
    (18): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (19): cudnn.ReLU
    (20): nn.StyleLoss
  }
}
Capturing style target 1
Capturing style target 2
Capturing style target 3
Capturing style target 4
Capturing style target 5
Capturing style target 6
Capturing style target 7
Capturing style target 8
Running optimization with ADAM
Iteration 50 / 200
  Content 1 loss: 1840350.585938
  Style 1 loss: 2566.359043
  Style 2 loss: 3547471.069336
  Style 3 loss: 5368391.235352
  Style 4 loss: 355980.445862
  Style 5 loss: 13842.927933
  Total loss: 11128602.623463
Iteration 100 / 200
  Content 1 loss: 1840350.585938
  Style 1 loss: 2566.359043
  Style 2 loss: 3547471.069336
  Style 3 loss: 5368391.235352
  Style 4 loss: 355980.445862
  Style 5 loss: 13842.927933
  Total loss: 11128602.623463
Iteration 150 / 200
  Content 1 loss: 1840350.585938
  Style 1 loss: 2566.359043
  Style 2 loss: 3547471.069336
  Style 3 loss: 5368391.235352
  Style 4 loss: 355980.445862
  Style 5 loss: 13842.927933
  Total loss: 11128602.623463
Iteration 200 / 200
  Content 1 loss: 1840350.585938
  Style 1 loss: 2566.359043
  Style 2 loss: 3547471.069336
  Style 3 loss: 5368391.235352
  Style 4 loss: 355980.445862
  Style 5 loss: 13842.927933
  Total loss: 11128602.623463
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 538683157
Successfully loaded models/VGG16_SOD_finetune.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8-SOD100: 1 1 4096 100
Setting up style layer          2       :       relu1_1
Setting up style layer          7       :       relu2_1
Setting up style layer          12      :       relu3_1
Setting up style layer          19      :       relu4_1
Setting up content layer        21      :       relu4_2
Setting up style layer          26      :       relu5_1
Capturing content targets
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> output]
  (1): nn.GPU(1) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (2): nn.GPU(2) @ nn.Sequential {
    [input -> (1) -> output]
    (1): nn.StyleLoss
  }
  (3): nn.GPU(3) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  }
  (4): nn.GPU(4) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.ReLU
    (2): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (5): nn.GPU(5) @ nn.Sequential {
    [input -> (1) -> (2) -> output]
    (1): cudnn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
  }
  (6): nn.GPU(6) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> output]
    (1): nn.StyleLoss
    (2): cudnn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
    (3): cudnn.ReLU
  }
  (7): nn.GPU(7) @ nn.Sequential {
    [input -> (1) -> output]
    (1): cudnn.SpatialMaxPooling(2x2, 2,2)
  }
  (8): nn.GPU(8) @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> output]
    (1): cudnn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
    (3): nn.StyleLoss
    (4): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (5): cudnn.ReLU
    (6): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (7): cudnn.ReLU
    (8): cudnn.SpatialMaxPooling(2x2, 2,2)
    (9): cudnn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
    (10): cudnn.ReLU
    (11): nn.StyleLoss
    (12): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (13): cudnn.ReLU
    (14): nn.ContentLoss
    (15): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (16): cudnn.ReLU
    (17): cudnn.SpatialMaxPooling(2x2, 2,2)
    (18): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (19): cudnn.ReLU
    (20): nn.StyleLoss
  }
}
Capturing style target 1
Capturing style target 2
Capturing style target 3
Capturing style target 4
Capturing style target 5
Capturing style target 6
Capturing style target 7
Capturing style target 8
Running optimization with ADAM
Iteration 50 / 200
  Content 1 loss: 1613944.433594
  Style 1 loss: 3785.628319
  Style 2 loss: 5391063.720703
  Style 3 loss: 8514136.230469
  Style 4 loss: 540189.697266
  Style 5 loss: 18604.844570
  Total loss: 16081724.554920
Iteration 100 / 200
  Content 1 loss: 1613944.433594
  Style 1 loss: 3785.628319
  Style 2 loss: 5391063.720703
  Style 3 loss: 8514136.230469
  Style 4 loss: 540189.697266
  Style 5 loss: 18604.844570
  Total loss: 16081724.554920
Iteration 150 / 200
  Content 1 loss: 1613944.433594
  Style 1 loss: 3785.628319
  Style 2 loss: 5391063.720703
  Style 3 loss: 8514136.230469
  Style 4 loss: 540189.697266
  Style 5 loss: 18604.844570
  Total loss: 16081724.554920
Iteration 200 / 200
  Content 1 loss: 1613944.433594
  Style 1 loss: 3785.628319
  Style 2 loss: 5391063.720703
  Style 3 loss: 8514136.230469
  Style 4 loss: 540189.697266
  Style 5 loss: 18604.844570
  Total loss: 16081724.554920

What is happening here, and is it possible to fix this?

Here's the nvidia-smi output:

ubuntu@ip-Address:~$ nvidia-smi

Fri Oct 20 01:58:01 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48                 Driver Version: 367.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:00:17.0     Off |                    0 |
| N/A   62C    P0    63W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:00:18.0     Off |                    0 |
| N/A   47C    P0    71W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:00:19.0     Off |                    0 |
| N/A   67C    P0    61W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:00:1A.0     Off |                    0 |
| N/A   52C    P0    72W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 0000:00:1B.0     Off |                    0 |
| N/A   65C    P0    66W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 0000:00:1C.0     Off |                    0 |
| N/A   48C    P0    71W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 0000:00:1D.0     Off |                    0 |
| N/A   65C    P0    66W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 0000:00:1E.0     Off |                    0 |
| N/A   48C    P0    74W / 149W |      0MiB / 11439MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
ubuntu@ip-Address:~$

Edit:

This also happened with a second content/style image combo at the same image size values.

[htoyryla]

No idea about loss problem, but your allocation of layers to the GPUs looks strange to me. The lower conv layers have a GPU of their own, sometimes a GPU handles nothing but a single style loss module (GPU2) or a pooling layer (GPU7). One conv layer is split so that its ReLU is handled on a different GPU (GPU3 - GPU4). And finally the whole upper half of the model (seven conv layers with their ReLUs and style loss modules) is handled by a single GPU.

I don't know if this can cause problems, but my intuition would say put a conv layer together with its associated activation layer and loss module onto the same gpu always.

Other peculiarities in your output:
Style is captured 8 times while you only have 5 style layers. Don't know though if this is normal for multigpu use (which I don't have).
Nvidia-smi shows no GPU memory being used.

[ProGamerGov]

@htoyryla

Nvidia-smi shows no GPU memory being used.

The nvidia-smi was made after the issue when I didn't have neural_style.lua running. I have a bunch of saved nvidia-smi outputs here: https://gist.github.com/ProGamerGov/8f1d07d866700b159c4b15c4e2fda868

Style is captured 8 times while you only have 5 style layers.

As for the style images, they are all the same image, except they have been rotated, and reflected into different style images. I used these scripts that I made to create them.

Don't know though if this is normal for multigpu use (which I don't have).

I haven't really experimented with multigpu before, so I could have easily screwed something up with the resource allocation parameter. But I also can't seem to find examples of anyone creating images at the same sizes as I was, with Neural-Style, so it could be an issue that was hidden until now.

How would you suggest that I structure the -multigpu_strategy parameter for a VGG-16 model?

[htoyryla]

Anyway, my main point was the strange way you have allocated layers to gpus:

The lower conv layers have a GPU of their own, sometimes a GPU handles nothing but a single style loss module (GPU2) or a pooling layer (GPU7). One conv layer is split so that its ReLU is handled on a different GPU (GPU3 - GPU4). And finally the whole upper half of the model (seven conv layers with their ReLUs and style loss modules) is handled by a single GPU.

I don't know if this can cause problems, but my intuition would say put a conv layer together with its associated activation layer and loss module onto the same gpu always.

Keep in mind that when assigning layers to gpus, also the ReLUs (which actually form the output of a conv layer) and style loss layers (which calculate the style loss) are counted as separate layers. Likewise the pooling layers. I don't know why the way you allocated the layers would not work, but on the other hand, putting the layers that belong together onto the same gpu should be safer. And also, to spread the memory usage evenly among the gpus, if you want to maximize the image size.

[htoyryla]

How would you suggest that I structure the -multigpu_strategy parameter for a VGG-16 model?

You should figure that out yourself. Make a numbered list of all layers. Mark which layers belong together so as not to put them to different gpus. Then you can experiment allocating them to different gpus. Look at nvidia-smi to see which gpus use most memory.

I would assume that the conv layers use up most of the memory. ReLUs and loss layers belong naturally on the same gpu as the preceding conv layer, at least it makes no sense put them on another gpu. I would also put a pooling layer on the same gpu as the conv layer that precedes it.

[ProGamerGov]

@htoyryla Thanks for the replies.

I'm not sure how I didn't see that the terminal output under each GPU, lists what layers have been assigned to them. I also was working with a 1-12 set of values for the -multigpu_strategy parameter, instead of the actually 20 possible values that are used.

There are 4 loss module layers used:

	(3): nn.StyleLoss
	(11): nn.StyleLoss
	(14): nn.ContentLoss
    (20): nn.StyleLoss

7 Convolution and ReLU pairs:

    (1): cudnn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
    (2): cudnn.ReLU
	
	(4): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (5): cudnn.ReLU
	
	(6): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
    (7): cudnn.ReLU
	
	(9): cudnn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
    (10): cudnn.ReLU
	
	(12): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (13): cudnn.ReLU
	
	(15): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (16): cudnn.ReLU
	
	(18): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
    (19): cudnn.ReLU

And 2 pooling layers:

	(8): cudnn.SpatialMaxPooling(2x2, 2,2)
	(17): cudnn.SpatialMaxPooling(2x2, 2,2)

With 8 GPUs and what you theorized in terms of GPU usage per layer, ideally I think it would be best to stick all the Convolution/ReLU pairs onto their own layers for 7 GPUs, and the lower memory using pooling and loss module layers onto the 8th GPU. But due to how the -multigpu_strategy parameter works, the layers have to be listed sequentially.

With giving every Convolution/ReLU pair their own layer, a few are going to have to also share a loss module layer or a pooling layer. But this also seems to leave a final 8th GPU unused.

These two ideas though don't factor in any changes in memory usage between each of the 7 Convolution/ReLU pairs. But I assume that like how normally higher layers use more memory, the same holds true for the -multigpu_strategy parameter. I could counteract this by making some of the lower Convolution/ReLU pairs share the same GPU, but then I still have to potentially split of the higher Convolution/ReLU pairs in order to not waste some of the GPUs. Though if spiting the layers causes the issues, then I can't do this.

I was also considering that the issue could be caused by a limitation in the libraries, or code that Neural-Style uses. Though it makes more sense that it would be a library, instead of Neural-Style's code itself. If it's caused by the optim library, that means that the ADAM optimizer is related to the issue. If it's caused by the 'image' library, then that means at some point things fail when working with extremely large images. If Lua's limitations are the cause, then maybe doing a custom install would fix things. If it's the result of CUDA/cuDNN, then there's no way I could ever hope to fix things.

For these issues, I would need to test for all of these possibilities, and hope that I catch something which clearly shows were the fault starts.

[htoyryla]

Of course the layers have to be assigned sequentially, as the calculation is done sequentially. One GPU calculates the first layers and then its output is passed to the next GPU, until the whole network has been calculated.

Because of this, you need to plan this with a sequential list, looking at where to place the splits. Much of your speculation is beside the point because of the sequential nature of multigpu calculation.

Most naturally, the pooling and style loss layers belong together with the conv layer before. Put the conv layer, relu, style layers and pooling layer all together. Splitting a ReLU onto a different GPU makes no sense as usually ReLU is calculated in place (simply one simple additional calculation within the convlayer). You could put a style layer or a pooling layer to the next GPU too, if that helps you to divide the memory usage better.

If there is nothing to put to the 8th GPU then don't. It will not help much to move a single simple layer there, and it will only make things slower.

There may be a problem with some of the software used, but until you configure it in a way that makes sense, you cannot know.

[htoyryla]

@ProGamerGov you say you have 7 conv layers, but in the output in your first post there are 11 conv layers, of which 7 have been put onto gpu8. You have now been looking only at these higher layers.

To get the full numbered list of ALL layers, run neural-style once without multigpu; you can interrupt it as soon as the optimizer is started. Also remember that the layer numbering changes when you change the style and/or content layers.

Using the defaults for content and style layers, VGG16 looks like this (however including TVLoss which is not counted when specifying the strategy parameter, something that you need to take into account):

nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> (33) -> output]
  (1): nn.TVLoss
  (2): nn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
  (3): nn.ReLU
  (4): nn.StyleLoss
  (5): nn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  (6): nn.ReLU
  (7): nn.SpatialMaxPooling(2x2, 2,2)
  (8): nn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
  (9): nn.ReLU
  (10): nn.StyleLoss
  (11): nn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
  (12): nn.ReLU
  (13): nn.SpatialMaxPooling(2x2, 2,2)
  (14): nn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
  (15): nn.ReLU
  (16): nn.StyleLoss
  (17): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (18): nn.ReLU
  (19): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (20): nn.ReLU
  (21): nn.SpatialMaxPooling(2x2, 2,2)
  (22): nn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
  (23): nn.ReLU
  (24): nn.StyleLoss
  (25): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (26): nn.ReLU
  (27): nn.ContentLoss
  (28): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (29): nn.ReLU
  (30): nn.SpatialMaxPooling(2x2, 2,2)
  (31): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (32): nn.ReLU
  (33): nn.StyleLoss
}

[htoyryla]

If there is a software problem, it is probably not in the image library, as it is only used for loading, pre/deprocessing and saving images. In your case, it looks like that the losses have been calculated properly, which means the forward pass through the model works OK. The fact that the losses do not change could indicate that the gradients end up being zero and the input image stays the same. Why this happens can be debugged when you have the process properly configured otherwise, but you have to know what you are doing.

[ProGamerGov]

@htoyryla I came up with using:

-gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 3,6,12,15,20,26,31, -gpu 0,1,2,3,4,5,6 -multigpu_strategy 3,6,12,15,20,26, and a few other combinations.

As the default set of layers was:

Successfully loaded models/VGG16_SOD_finetune.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8-SOD100: 1 1 4096 100
Setting up style layer          2       :       relu1_1
Setting up style layer          7       :       relu2_1
Setting up style layer          12      :       relu3_1
Setting up style layer          19      :       relu4_1
Setting up content layer        21      :       relu4_2
Setting up style layer          26      :       relu5_1
Capturing content targets
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> output]
  (1): cudnn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
  (2): cudnn.ReLU
  (3): nn.StyleLoss
  (4): cudnn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
  (5): cudnn.ReLU
  (6): cudnn.SpatialMaxPooling(2x2, 2,2)
  (7): cudnn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
  (8): cudnn.ReLU
  (9): nn.StyleLoss
  (10): cudnn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
  (11): cudnn.ReLU
  (12): cudnn.SpatialMaxPooling(2x2, 2,2)
  (13): cudnn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
  (14): cudnn.ReLU
  (15): nn.StyleLoss
  (16): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (17): cudnn.ReLU
  (18): cudnn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
  (19): cudnn.ReLU
  (20): cudnn.SpatialMaxPooling(2x2, 2,2)
  (21): cudnn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
  (22): cudnn.ReLU
  (23): nn.StyleLoss
  (24): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (25): cudnn.ReLU
  (26): nn.ContentLoss
  (27): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (28): cudnn.ReLU
  (29): cudnn.SpatialMaxPooling(2x2, 2,2)
  (30): cudnn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
  (31): cudnn.ReLU
  (32): nn.StyleLoss
}

I also tested different combinations of parameters, in an effort to rule out any parameters as the cause. I could affect the loss values but they always remained the same between iterations. The issue also occurs with lbfgs and -lbfgs_num_correction. Using the NIN model with even up to 6-8k resolution (With a single GPU), still works fine, so this might mean that the issue may be related to how VGG and NIN models are interact with Neural-Style.

I printed out the tensor from the self.gradInput variable from the StyleLoss:updateGradInput(input, gradOutput) function, and the values seemed to be normal. Though trying to print the tensor in other areas resulted in error messages about Lua not having enough memory.

Could this issue simply be caused by something reaching it's limit in terms of size, and then not having an error message of any kind? The only thing that seems to be constant with this issue is the -image_size value being extremely large. Maybe the once the limit is reached, it fails to finish the calculations and is constantly retrying the same iteration 1 calculations again and again? That makes more sense, because if my -multigpu_strategy parameters were the only cause, then why would they still result in working loss values at lower image sizes?

Do you know anyway that I could test for a variable hitting it's limit and stopping, or failing the calculations?

Edit:

I can confirm that the issue does not occur when using the NIN model and the -multigpu_strategy parameter with any of the values that I have tested. The issue also occurs with VGG-19 models.

[htoyryla]

I didn't mean to say that your multi-gpu strategy was causing the error, just that your original values were dividing the layers to the gpus in a way that is hardly optimal; as well as in a way that has hardly been the intention of the developers, and therefore maybe never tested.

It could be now that a limit is reached which does not result in an error but, instead, some variables getting zero (similar to that you may sometimes get nans or infs). If the gradients become zero, the input will not be changed and each iteration will process the same image, resulting in the same losses.

This may be tricky to catch. Most of the calculations happen under the hood, inside torch and cuda. I would start monitoring the gradients from the loss modules. You tried that already, but I am not sure if a plain print inside a module assigned to run on a gpu is a good idea. What I did recently was add local variables to the loss modules, store the gradients there and then inside feval print them out. But even that will not probably help much, we may learn something but still not be able to solve the issue (which almost certainly happens under the hood).

BTW, you say self.gradInput (for the modules you checked) looked normal. Now, as the losses get evaluated properly, that is to be expected. What counts, however, is the gradients at the input of the model, which affect how the input image is to be modified. If THAT gradient gets zero, nothing will change from iteration to iteration.

Comes to my mind, have you tried different style and content weights? They affect the gradients, so maybe significantly larger weights might help. Another factor between the input and the model is the TVLoss layer. Experiment with different settings there too. Also, TVLoss module can be a good place to get the value of the total gradient.

[ProGamerGov]

@htoyryla Using a style weight of 75000 did not fix the issue. Neither did disabling the content loss module via -content_weight 0. For my previous tests, -tv_weight was set to 0. I also tested using -init random and -pooling avg, but both those parameters with their other options did not fix the issue either.

The -tv_weight value does appear to be capable of "fixing" the loss values, but at the cost of the negative effects of using the -tv_weight value. The -tv_weight value seems to have a specific cutoff point in terms of it's effectiveness at "fixing" the loss values.

Using -tv_weight 0.0000005:

Running optimization with ADAM
Iteration 1 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 2 / 200
  Content 1 loss: 1836153.125000
  Style 1 loss: 4058.315277
  Style 2 loss: 5598318.603516
  Style 3 loss: 7999315.429688
  Style 4 loss: 550134.429932
  Style 5 loss: 21249.456882
  Total loss: 16009229.360294
Iteration 3 / 200
  Content 1 loss: 1712206.250000
  Style 1 loss: 6884.845018
  Style 2 loss: 9333292.236328
  Style 3 loss: 13165092.773438
  Style 4 loss: 877727.874756
  Style 5 loss: 32724.154472
  Total loss: 25127928.134012
Iteration 4 / 200
  Content 1 loss: 1606906.347656
  Style 1 loss: 9842.310905
  Style 2 loss: 13070276.367188
  Style 3 loss: 17805824.707031
  Style 4 loss: 1193168.975830
  Style 5 loss: 43954.135895
  Total loss: 33729972.844505

Using -tv_weight 0:

Running optimization with ADAM
Iteration 1 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 2 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 3 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 4 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219

-tv_weight 0.00000000001:

Running optimization with ADAM
Iteration 1 / 200
  Content 1 loss: 1994813.281250
  Style 1 loss: 1589.992940
  Style 2 loss: 2065276.977539
  Style 3 loss: 2789657.592773
  Style 4 loss: 215494.812012
  Style 5 loss: 9914.423704
  Total loss: 7076747.080219
Iteration 2 / 200
  Content 1 loss: 1994724.609375
  Style 1 loss: 1591.018081
  Style 2 loss: 2066690.734863
  Style 3 loss: 2791444.885254
  Style 4 loss: 215617.721558
  Style 5 loss: 9919.698715
  Total loss: 7079988.667846
Iteration 3 / 200
  Content 1 loss: 1994599.609375
  Style 1 loss: 1592.466652
  Style 2 loss: 2068689.147949
  Style 3 loss: 2793970.825195
  Style 4 loss: 215791.442871
  Style 5 loss: 9927.155256
  Total loss: 7084570.647299
Iteration 4 / 200
  Content 1 loss: 1994446.484375
  Style 1 loss: 1594.235837
  Style 2 loss: 2071136.352539
  Style 3 loss: 2797064.941406
  Style 4 loss: 216004.257202
  Style 5 loss: 9936.286926
  Total loss: 7090182.558286
Iteration 5 / 200
  Content 1 loss: 1994269.531250
  Style 1 loss: 1596.276283
  Style 2 loss: 2073960.937500
  Style 3 loss: 2800638.610840
  Style 4 loss: 216249.984741
  Style 5 loss: 9946.829796
  Total loss: 7096662.170410

The actual cutoff for -tv_weight affecting things in the correct direction has 14 zero past the decimal point before the 1. Using 15 zeros creates some weird endless loop pattern in the values that occurs over multiple iterations. Using 16 zeros results in no change in terms of loss values.

Using -tv_weight with 14, 15, and 16 zeros, and their associated loss values: https://gist.github.com/ProGamerGov/7338462bea65c066b57ef5668003a15d

Based on my experiments, I think the issue must be occurring in the Style Loss functions, and/or the feval(x) function. The results of my testing also seem to support the theory of something in the code reaching it's limit.

---

What I did recently was add local variables to the loss modules, store the gradients there and then inside feval print them out. But even that will not probably help much, we may learn something but still not be able to solve the issue (which almost certainly happens under the hood).

Would it be possible for you to share the modified code you are referring to here?

[htoyryla]

I think it is significant that TVLoss has an effect on the issue. According to your results, if the TVLoss layer is omitted, then the input does not change (= total gradient at input is zero). When you include the TVLoss layer, then the input changes, but the style losses increase instead of decreasing.

You suspect the problem lies within StyleLoss or feval. It may be, but remember that the loss modules and feval are only a part of the process, adam in optim package is doing much of the work in determining how the input should change before the next iteration. One could track the gradients from the loss modules to see how they behave, and it would also help to see the total gradient before and after TVLoss.

The code in which I stored internal variables within a loss module belongs to this discussion #425 . An example can be found here https://gist.github.com/htoyryla/233a9d0857440d2a8bafe732ddeba325 . In the StyleLoss module I have created a variable dbg, and then during operation I store some variable there. At line 192 I then print out the values. Note, that in this case I was only interested in what happens at target capture. You would also want to print the values from eval.

Printing out gradients (large tensors) as such is hardly very informative. In my case I was looking for nans or infs, you probably should be looking for where the gradients get to zero. Taking min, max and mean of the gradient is probably more informative than printing out whole gradients.

"the theory of something in the code reaching it's limit" is IHMO very vague. My theory is that for some reason, the input does not change from iteration to iteration, which again suggests that the total gradient is zero. Therefore one should be looking for whether it is so and why it happens.

[ProGamerGov]

@htoyryla I modified Neural-Style to print a whole bunch of variables from the style loss function, and the feval(x) function here: https://gist.github.com/ProGamerGov/e09e450f5c7dbb72ccac22c6244fa2f3

Functioning Loss System and the feval(x) function's grad and x variables:

• -image_size 512

• -tv_weight 0

The -image_size issue feval(x) function's grad and x variables:

• -image_size 2432

• -tv_weight 0

The -image_size issue with the -tv_weight layer, and the feval(x) function's grad and x variables:

• -image_size 2432

• -tv_weight 0.0000005

---

Link to the album: https://imgur.com/a/0JnVT

---

The raw log files for iterations 0-200 (saved every 1 iteration) for the above experiments can be found here:

-image_size 2432 & -tv_weight 0: https://gist.github.com/ProGamerGov/29e89551413bfe8ff8022df3edf822bd

-image_size 2432 & -tv_weight 0.0000005: https://gist.github.com/ProGamerGov/887307613788bde39567e287247bd209

-image_size 512 & -tv_weight 0: https://gist.github.com/ProGamerGov/c3d850318d6d80eee63890beb74ff928

---

It seems that the total gradient is 0, but a -tv_weight parameter above a certain value will change things. I'm not sure really how I would dig in deeper to why the gradient is 0, but I think that might require printing the variables from the optim's package's code?

[ProGamerGov]

After testing the GramMatrix code from here: https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L489-L512, it looks like the same values over and over again with an image size of 2432. There aren't any obvious zeros or out of place values.

Maybe the backwards pass (self.crit:backward) might be the cause or related to the cause?

Edit, More Logs:

GramMatrix & -image_size 2432 & -tv_weight 0.0000005: https://gist.github.com/ProGamerGov/e703044aeee6f4f8404e1bea3ba22dee

GramMatrix & -image_size 2432 & -tv_weight 0: https://gist.github.com/ProGamerGov/9929e217b88dd76be8e93af2ef9e35ec

GramMatrix & -image_size 512 & -tv_weight 0: https://gist.github.com/ProGamerGov/c943491331eb422f525b1274f4e21c25

The modified neural_style.lua used to create the 3 above logs: https://gist.github.com/ProGamerGov/8eced5ca1e8cfc181a50bf5fa0be9669

---

This log uses -image_size 2432 & -tv_weight 0, and I tried to get more variables that I missed in the previous logs from the GramMatrix and Style Loss functions: https://gist.github.com/ProGamerGov/a6f7b47b84ad16b91d8a9977ab572212

The above log used this modified neural_style.lua: https://gist.github.com/ProGamerGov/d0ad22965d6d41001201b54310dfd3db

[htoyryla]

OK. So now we know that the total gradient really is zero. This was to be expected, too, because nothing was changing from iteration to iteration.

There is no point in examining what the forward pass gives (e.g. in Gram matrix). One needs to look at the gradients. In a styleloss module, the gradient is calculated as follows.

    local dG = self.crit:backward(self.G, self.target)
    dG:div(input:nElement())
    self.gradInput = self.gram:backward(input, dG)

dG is the difference between the current gram matrix output and the target. A backward pass through the gram matrix gives the gradient at (gram matrix) input.

In the latter part of the code, not shown above, the gradient is multiplied by style weight and then added to the gradient from the output, which is the way how the gradients are accumulated in the backward pass until we have the total gradient at the bottom which we then apply to the image before the next iteration.

Note also the second line, division by the number of elements in the feature maps. The idea is to make loss values from different layers more in range with each other. But it also makes the gradients smaller as the image size increases. It would be interesting to know what happens to these variables when the image size is increased until neural-style stops working. Do they gradually vanish into zero... if yes then we can modify the normalization (the 2nd line above), or do they vanish suddenly at some size.

However, it seems to me that StyleLoss alone cannot cause the problem. Even if StyleLoss would produce zero gradients for some reason, ContentLoss would still produce a nonzero gradient making the iterations to work. Unless, of course, one is using -init image and content loss is zero.

It is pity I can't try out this myself... an interesting issue. Your printouts are almost useless to me,
too much information instead of focusing on the meaningful points. I am doing some tests without GPU to see what happens to the gradients when the image size increases. It looks like they do decrease with image size. Note also that even if the gradient values may not look too small at this point, they will be passed on to the optimizer and we don't really know what can happen there.