Potential incompatibility with v0.3 pre-trained models using weight_norm(..., dim=None)

[farleylai]

Issue description

Calling load_state_dict() to load a pre-trained PyTorch-0.3 model that uses weight_norm(..., dim=None) leads to mismatched tensor XXX_g dimension error: torch.Size([]) while torch.Size([1]) expected.

Source issue

Code example

In the weight_norm.py:

def _norm(p, dim):
    """computes the norm over all dimensions except dim"""
    if dim is None:
         return p.norm()  # returns 0-dim scalar tensor in v0.4 but 1-dim in v0.3
    ...

return p.norm().reshape(1) feels like a backward compatible workaround to return a 1-dim tensor as previous versions.

System Info

Collecting environment information...
PyTorch version: 0.4.0
Is debug build: No
CUDA used to build PyTorch: 9.1.85

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.2.148
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 396.37
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4

Versions of relevant libraries:
[pip3] numpy (1.13.3)
[pip3] numpydoc (0.7.0)
[pip3] torch (0.4.0)
[pip3] torchfile (0.1.0)
[pip3] torchtext (0.2.3)
[pip3] torchvision (0.2.0)
[conda] cuda80 1.0 h205658b_0 pytorch
[conda] cuda90 1.0 h6433d27_0 pytorch
[conda] cuda91 1.0 h4c16780_0 soumith
[conda] magma-cuda80 2.1.0 5 soumith
[conda] magma-cuda91 2.3.0 1 soumith
[conda] nccl2 1.0 0 soumith
[conda] pytorch 0.4.0 py36_cuda9.1.85_cudnn7.1.2_1 [cuda91] pytorch
[conda] torchfile 0.1.0
[conda] torchtext 0.2.3
[conda] torchvision 0.2.0 py36h17b6947_1 pytorch

[zhangweifeng1218]

return p.norm().reshape(1)? Is this feasible? The pytorch 0.4 has fixed the bug that pytorch 0.4 now can load the pre-trained model using weight_norm of pytorch 0.3. But When I use weight_norm in pytorch 0.4, the model still can not be trained using multiple GPUs.

[ailzhang]

@zhangweifeng1218 Can you provide a minimal repro script with error on multi GPUs?