Kernel args patch to show zero_init buffer (#1809)

Updated kernel args print to indicate zero_init buffers, which explains elementwise kernels happening before fusion.

Changes from this

```
Reduction and semaphore buffers:
  Float [16]
  Long [1]
```

To

```
Reduction and semaphore buffers:
  Float [16] is_zero_initialized: 0
  Long [1] is_zero_initialized: 1
```

The is_zero_initialized: 1 on a given buffer means an extra init kernel would be needed.

Author: jjsjann123

torch/csrc/jit/codegen/cuda/README.md

@@ -187,7 +187,7 @@ There're a few debug dump that could be turned on via environment variables. Loo
 1. `dump_eff_bandwidth`: print out effective bandwidth of each generated kernel. This naively measure the kernel time divided by I/O buffer size and is a good/simple metric of performance for bandwidth bound kernels
 2. `cuda_kernel`: print out generated cuda kernels
 3. `launch_param`: print out launch config of generated kernels
-4. `print_args`: print out input output tensors of executed codegen kernels
+4. `kernel_args`: print out input/output/buffer tensors of all executed codegen kernels, note that for buffers, we indicate whether they are zero-initialized, which hints on an extra kernel to fill the tensor before codegen kernels.
 
 ### FAQs
 

torch/csrc/jit/codegen/cuda/executor.cpp

@@ -790,13 +790,15 @@ std::vector<at::Tensor> FusionExecutor::runFusion(
                 at::TensorOptions()
                     .dtype(executor_entry->buffer_types[i])
                     .device(options_.device)));
+            global_buffers.zero_init.push_back(true);
           } else {
             global_buffers.buffers.push_back(at::native::empty_cuda(
                 executor_entry->buffer_sizes[i],
                 executor_entry->buffer_types[i],
                 c10::nullopt,
                 options_.device,
                 c10::nullopt));
+            global_buffers.zero_init.push_back(false);
           }
         }
       }
@@ -984,9 +986,14 @@ std::vector<at::Tensor> FusionExecutor::runFusion(
                 << " (strides = " << output.strides() << ")" << std::endl;
     }
     std::cout << "Reduction and semaphore buffers:" << std::endl;
-    for (const auto& buffer : global_buffers.buffers) {
+    TORCH_INTERNAL_ASSERT(
+        global_buffers.buffers.size() == global_buffers.zero_init.size(),
+        "global_buffer buffer & zero_init container should have identical sizes");
+    for (const auto i : c10::irange(global_buffers.buffers.size())) {
+      const auto& buffer = global_buffers.buffers[i];
+      const auto& zero_init = global_buffers.zero_init[i];
       std::cout << "  " << buffer.scalar_type() << " " << buffer.sizes()
-                << std::endl;
+                << " is_zero_initialized: " << zero_init << std::endl;
     }
   }