ValueError: Object arrays cannot be loaded when allow_pickle=False #158
Description
I am working on srgan in google colab and facing this isssue. I havs downloaded the pre-trained model vgg19.npy and saved it in "models" folder. The dataset for training and validation is stored in "myImages" folder.
Corresponding changes have been made in config.py:
...
## train set location
config.TRAIN.hr_img_path = 'myImages/DIV2K_train_HR/'
config.TRAIN.lr_img_path = 'myImages/DIV2K_train_LR_bicubic_X4/DIV2K_train_LR_bicubic/X4/'
config.VALID = edict()
## test set location
config.VALID.hr_img_path = 'myImages/DIV2K_valid_HR/'
config.VALID.lr_img_path = 'myImages/DIV2K_valid_LR_bicubic_X4/DIV2K_valid_LR_bicubic/X4/'
...
The error is:
2019-07-24 08:39:54.888009: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-24 08:39:54.971483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:54.972007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2019-07-24 08:39:54.981637: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-24 08:39:55.145234: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-24 08:39:55.220131: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-24 08:39:55.240930: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-24 08:39:55.428084: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-24 08:39:55.532085: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-24 08:39:55.871871: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-24 08:39:55.872255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:55.872801: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:55.873197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-24 08:39:55.887475: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-24 08:39:56.122111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:56.126218: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d01dc0 executing computations on platform CUDA. Devices:
2019-07-24 08:39:56.126268: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2019-07-24 08:39:56.210139: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2019-07-24 08:39:56.210370: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d02f40 executing computations on platform Host. Devices:
2019-07-24 08:39:56.210407: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2019-07-24 08:39:56.213122: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
**2019-07-24 08:39:56.213554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: **
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2019-07-24 08:39:56.213662: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-24 08:39:56.213687: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-07-24 08:39:56.213732: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-07-24 08:39:56.213755: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-07-24 08:39:56.213776: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-07-24 08:39:56.213800: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-07-24 08:39:56.213824: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-24 08:39:56.213943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:56.214390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:56.214773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-24 08:39:56.217825: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-07-24 08:39:56.219169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-24 08:39:56.219199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-07-24 08:39:56.219212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-07-24 08:39:56.224052: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:56.224490: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-24 08:39:56.227002: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2019-07-24 08:39:56.227103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14202 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
WARNING: Logging before flag parsing goes to stderr.
W0724 08:39:56.285700 139916578662272 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py:505: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
2019-07-24 08:39:59.343182: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-24 08:40:02.917466: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
File "train.py", line 357, in
train()
File "train.py", line 75, in train
VGG = tl.models.vgg19(pretrained=True, end_with='pool4', mode='static')
File "/usr/local/lib/python3.6/dist-packages/tensorlayer/models/vgg.py", line 317, in vgg19
restore_model(model, layer_type='vgg19')
File "/usr/local/lib/python3.6/dist-packages/tensorlayer/models/vgg.py", line 169, in restore_model
npz = np.load(os.path.join('models', model_saved_name[layer_type]), encoding='latin1').item()
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 447, in load
pickle_kwargs=pickle_kwargs)
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/format.py", line 696, in read_array
raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False
I don't understand any of it!