Add a CSV output option to displaylist_benchmark_parser.py and add a README detailing how to run and process benchmarks on iOS/Android (flutter#31481)

Author: George Wright

testing/benchmark/README_displaylist.md

@@ -0,0 +1,68 @@
+# Running and Processing DisplayList Benchmarks
+
+The DisplayList benchmarks for Flutter are used to determine the relative
+cost of operations in order to assign scores to each op for the raster 
+cache’s cache admission algorithm.
+
+Due to the nature of benchmarking, these need to be run on actual devices in
+order to get representative results, and because of the more locked-down
+nature of the iOS and Android platforms it’s a little involved getting the
+benchmark suite to run.
+
+This document will detail the steps involved in getting the benchmarks to run
+on both iOS and Android, and how to process the resulting data.
+
+## iOS
+
+iOS does not allow you to run unsigned code or arbitrary executables, so the
+approach here is to build a dylib that contains the benchmarking code which
+will then be linked to a skeleton test app in Xcode. The dylib contains an
+exported C function, void RunBenchmarks(int argc, char **argv) that should
+be called from the skeleton test app to run the benchmarks.
+
+The dylib is not built by default and it will need to be specified as a
+target manually when calling ninja.
+
+The target name is ios_display_list_benchmarks, e.g.:
+
+    $ ninja -C out/ios_profile ios_display_list_benchmarks
+
+Once that dylib exists, the IosBenchmarks test app in flutter/testing/ios can
+be loaded in Xcode. Ensure that the team is set appropriately so the code can
+be signed and that FLUTTER_ENGINE matches the Flutter Engine build you wish to
+use (e.g. ios_profile).
+
+Once that is done, you can just hit the Run button and the JSON output will be
+sent to the Xcode console. Copy that elsewhere and save it as a .json file.
+
+Note: you may need to delete some errors from the console output that are
+unrelated to the JSON output.
+
+## Android
+
+On Android, even on non-rooted devices, it is possible to execute unsigned
+binaries using adb. As a result, there is a build target that will build a
+binary that can be pushed to device using adb and executed using adb shell.
+The only caveat is that the binary needs to be on a volume that isn’t mounted
+as noexec, which typically rules out the sd card. /data/local/tmp seems like
+an option that is typically available.
+
+The build target is called display_list_benchmarks and will create a binary
+called display_list_benchmarks in the root output directory
+(e.g. android_profile_arm64).
+
+    $ adb push out/android_profile_arm64/display_list_benchmarks /data/local/tmp/display_list_benchmarks
+    $ adb shell /data/local/tmp/display_list_benchmarks --benchmark_format=json | tee android-results.json
+
+
+The results in android-results.json can then be processed.
+
+## Processing Results
+
+There is a script in flutter/testing/benchmark called
+displaylist_benchmark_parser.py which will take the JSON file and output a PDF
+with graphs of all the benchmark series, as well as a CSV that can be imported
+into a spreadsheet for further analysis.
+
+This can then be manually analysed to determine the relative weightings for the
+raster cache’s cache admission algorithm.

testing/benchmark/displaylist_benchmark_parser.py

@@ -5,6 +5,7 @@
 # found in the LICENSE file.
 
 import argparse
+import csv
 import json
 import sys
 import matplotlib.pyplot as plt
@@ -20,6 +21,7 @@ def __init__(self, name, backend, timeUnit, drawCallCount):
     self.yLimit = 200
     self.timeUnit = timeUnit
     self.drawCallCount = drawCallCount
+    self.optionalValues = {}
 
   def __repr__(self):
     return 'Name: % s\nBackend: % s\nSeries: % s\nSeriesLabels: % s\n' % (self.name, self.backend, self.series, self.seriesLabels)
@@ -34,6 +36,12 @@ def addDataPoint(self, family, x, y):
     if y > self.yLimit:
       self.largeYValues = True
 
+  def addOptionalValue(self, name, x, y):
+    if name not in self.optionalValues:
+      self.optionalValues[name] = {}
+
+    self.optionalValues[name][x] = y
+
   def setFamilyLabel(self, family, label):
     # I'm not keying the main series dict off the family label
     # just in case we get data where the two aren't a 1:1 mapping
@@ -93,17 +101,41 @@ def plot(self):
 
     return figures
 
+  def writeCSV(self, writer):
+    # For now assume that all our series have the same x values
+    # this is true for now, but may differ in the future with benchmark changes
+    x_values = []
+    y_values = []
+    for family in self.series:
+      x_values = ['x'] + self.series[family]['x']
+      y_values.append([self.seriesLabels[family]] + self.series[family]['y'])
+
+    for name in self.optionalValues:
+      column = [name]
+      for key in self.optionalValues[name]:
+        column.append(self.optionalValues[name][key])
+      y_values.append(column)
+
+    writer.writerow([self.name, self.drawCallCount])
+    for line in range(len(x_values)):
+      row = [x_values[line]]
+      for series in range(len(y_values)):
+        row.append(y_values[series][line])
+      writer.writerow(row)
+
 def main():
   parser = argparse.ArgumentParser()
 
   parser.add_argument('filename', action='store',
       help='Path to the JSON output from Google Benchmark')
   parser.add_argument('-o', '--output-pdf', dest='outputPDF', action='store', default='output.pdf',
                       help='Filename to output the PDF of graphs to.')
+  parser.add_argument('-c', '--output-csv', dest='outputCSV', action='store', default='output.csv',
+                      help='Filename to output the CSV data to.')
 
   args = parser.parse_args()
   jsonData = parseJSON(args.filename)
-  return processBenchmarkData(jsonData, args.outputPDF)
+  return processBenchmarkData(jsonData, args.outputPDF, args.outputCSV)
 
 def error(message):
   print(message)
@@ -127,7 +159,7 @@ def extractAttributesLabel(benchmarkResult):
 
   return label[:-2]
 
-def processBenchmarkData(benchmarkJSON, outputPDF):
+def processBenchmarkData(benchmarkJSON, outputPDF, outputCSV):
   benchmarkResultsData = {}
 
   for benchmarkResult in benchmarkJSON:
@@ -169,18 +201,28 @@ def processBenchmarkData(benchmarkJSON, outputPDF):
     else:
       benchmarkDrawCallCount = -1
 
+    optional_keys = ['DrawCallCount_Varies', 'VerbCount', 'PointCount', 'VertexCount', 'GlyphCount']
+
     if benchmarkName not in benchmarkResultsData:
       benchmarkResultsData[benchmarkName] = BenchmarkResult(benchmarkName, benchmarkBackend, benchmarkUnit, benchmarkDrawCallCount)
 
+    for key in optional_keys:
+      if key in benchmarkResult:
+        benchmarkResultsData[benchmarkName].addOptionalValue(key, benchmarkSeededValue, benchmarkResult[key])
+
     benchmarkResultsData[benchmarkName].addDataPoint(benchmarkFamilyIndex, benchmarkSeededValue, benchmarkRealTime)
     benchmarkResultsData[benchmarkName].setFamilyLabel(benchmarkFamilyIndex, benchmarkFamilyLabel)
 
   pp = pdfp(outputPDF)
 
+  csv_file = open(outputCSV, 'w')
+  csv_writer = csv.writer(csv_file)
+
   for benchmark in benchmarkResultsData:
     figures = benchmarkResultsData[benchmark].plot()
     for fig in figures:
       pp.savefig(fig)
+    benchmarkResultsData[benchmark].writeCSV(csv_writer)
   pp.close()