External tool support (#1756)

* Add tool support

* Add tool support

* Updating Docs

* Updating Docs

* Updating CONTRIBUTING.md

* Updating CONTRIBUTING.md

* Adding more complete tool options.

* Adding more complete tool options.

* Update docs on Linux again.

* Update docs on Linux again.

* Review changes and associated docs updates

* Update README.md

* Remove unnecessary type declarations.

* Review Changes

Author: gspencergoog

CHANGELOG.md

@@ -1,3 +1,7 @@
+## 0.22.0
+* Added the ability to run external tools on a section of documentation and 
+  replace it with the output of the tool.
+
 ## 0.21.1
 * Fix a problem where category ordering specified in categories option
   was not obeyed.  Reintroduce categoryOrder option to solve this problem.

CONTRIBUTING.md

@@ -25,8 +25,16 @@ yet in the issue tracker, start by opening an issue. Thanks!
 2. When a change is user-facing, please add a new entry to the [changelog](https://github.com/dart-lang/dartdoc/blob/master/CHANGELOG.md)
 3. Please include a test for your change.  `dartdoc` has both `package:test`-style unittests as well as integration tests.  To run the unittests, use `dart test/all.dart`.  Most changes can be tested via a unittest, but some require modifying the [test_package](https://github.com/dart-lang/dartdoc/tree/master/testing/test_package) and regenerating its docs via `grind update-test-package-docs`.
 4. For major changes, run `grind compare-sdk-warnings` and `grind compare-flutter-warnings`, and include the summary results in your pull request.
-5.  Be sure to format your Dart code using `dartfmt -w`, otherwise travis will complain.
-6.  Post your change via a pull request for review and integration!
+5. Be sure to format your Dart code using `dartfmt -w`, otherwise travis will complain.
+6. Because there are generated versions of the dartdoc docs for stable and development versions of Dart,
+   you need to update the docs twice:
+   - Download and install the latest STABLE version of dart from [the Dart website](https://www.dartlang.org/tools/sdk).
+     (It's probably easiest to download a zip file and change your PATH to the extracted location's `bin` directory)
+   - Run `pub run grinder update-test-package-docs` to update the stable docs.
+   - Download and install the latest DEV version of dart from [the Dart website](https://www.dartlang.org/tools/sdk)
+     (It's probably easiest to download a zip file and change your PATH to the extracted location's `bin` directory)
+   - Run `pub run grinder update-test-package-docs` to update the dev docs.
+7. Post your change via a pull request for review and integration!
 
 ## Testing
 

README.md

@@ -221,15 +221,15 @@ You can specify links to videos inline that will be handled with a simple HTML5
 You can specify "macros", i.e. reusable pieces of documentation. For that, first specify a template
 anywhere in the comments, like:
 
-```
+```dart
 /// {@template template_name}
 /// Some shared docs
 /// {@endtemplate}
 ```
 
 and then you can insert it via `{@macro template_name}`, like
 
-```
+```dart
 /// Some comment
 /// {@macro template_name}
 /// More comments
@@ -240,6 +240,60 @@ dartdoc is currently documenting.  This can lead to inconsistent behavior betwee
 packages, especially if different command lines are used for dartdoc.  It is recommended to use collision-resistant
 naming for any macros by including the package name and/or library it is defined in within the name.
 
+### Tools
+
+Dartdoc allows you to filter parts of the documentation through an external tool
+and then include the output of that tool in place of the given input.
+
+First, you have to configure the tools that will be used in the `dartdoc_options.yaml` file:
+
+```yaml
+dartdoc:
+  tools:
+    drill:
+      command: ["bin/drill.dart"]
+      description: "Puts holes in things."
+    echo:
+      macos: ['/bin/sh', '-c', 'echo']
+      linux: ['/bin/sh', '-c', 'echo']
+      windows: ['C:\\Windows\\System32\\cmd.exe', '/c', 'echo']
+      description: 'Works on everything'
+```
+
+The `command` tag is used to describe the command executable, and any options
+that are common among all executions. If the first element of this list is a
+filename that ends in `.dart`, then the dart executable will automatically be
+used to invoke that script. The `command` defined will be run on all platforms.
+
+The `macos`, `linux`, and `windows` tags are used to describe the commands to
+be run on each of those platforms.
+
+The `description` is just a short description of the tool for use as help text. 
+
+Only tools which are configured in the `dartdoc_options.yaml` file are able to
+be invoked.
+
+To use the tools in comment documentation, use the `{@tool <name> [<options> ...] [$INPUT]}`
+directive to invoke the tool:
+
+```dart
+/// {@tool drill --flag --option="value" $INPUT}
+/// This is the text that will be sent to the tool as input.
+/// {@end-tool}
+```
+
+The `$INPUT` argument is a special token that will be replaced with the name of
+a temporary file that the tool needs to read from. It can appear anywhere in the
+options, and can appear multiple times.
+
+If the example `drill` tool with those options is a tool that turns the content
+of its input file into a code-font heading, then the directive above would be
+the equivalent of having the following comment in the code:
+
+```dart
+/// # `This is the text that will be sent to the tool as input.`
+```
+
 ### Auto including dependencies
 
 If `--auto-include-dependencies` flag is provided, dartdoc tries to automatically add

lib/src/dartdoc_options.dart

@@ -56,6 +56,9 @@ class DartdocFileMissing extends DartdocOptionError {
   DartdocFileMissing(String details) : super(details);
 }
 
+/// Defines the attributes of a category in the options file, corresponding to
+/// the 'categories' keyword in the options file, and populated by the
+/// [CategoryConfiguration] class.
 class CategoryDefinition {
   /// Internal name of the category.
   final String name;
@@ -73,6 +76,8 @@ class CategoryDefinition {
   String get displayName => _displayName ?? name;
 }
 
+/// A configuration class that can interpret category definitions from a YAML
+/// map.
 class CategoryConfiguration {
   /// A map of [CategoryDefinition.name] to [CategoryDefinition] objects.
   final Map<String, CategoryDefinition> categoryDefinitions;
@@ -110,6 +115,112 @@ class CategoryConfiguration {
   }
 }
 
+/// Defines the attributes of a tool in the options file, corresponding to
+/// the 'tools' keyword in the options file, and populated by the
+/// [ToolConfiguration] class.
+class ToolDefinition {
+  /// A list containing the command and options to be run for this tool. The
+  /// first argument in the command is the tool executable. Must not be an empty
+  /// list, or be null.
+  final List<String> command;
+
+  /// A description of the defined tool. Must not be null.
+  final String description;
+
+  ToolDefinition(this.command, this.description)
+      : assert(command != null),
+        assert(command.isNotEmpty),
+        assert(description != null);
+
+  @override
+  String toString() => '$runtimeType: "${command.join(' ')}" ($description)';
+}
+
+/// A configuration class that can interpret [ToolDefinition]s from a YAML map.
+class ToolConfiguration {
+  final Map<String, ToolDefinition> tools;
+
+  ToolConfiguration._(this.tools);
+
+  static ToolConfiguration get empty {
+    return new ToolConfiguration._({});
+  }
+
+  static ToolConfiguration fromYamlMap(
+      YamlMap yamlMap, pathLib.Context pathContext) {
+    var newToolDefinitions = <String, ToolDefinition>{};
+    for (var entry in yamlMap.entries) {
+      var name = entry.key.toString();
+      var toolMap = entry.value;
+      var description;
+      List<String> command;
+      if (toolMap is Map) {
+        description = toolMap['description']?.toString();
+        // If the command key is given, then it applies to all platforms.
+        var commandFrom = toolMap.containsKey('command')
+            ? 'command'
+            : Platform.operatingSystem;
+        if (toolMap.containsKey(commandFrom)) {
+          if (toolMap[commandFrom].value is String) {
+            command = [toolMap[commandFrom].toString()];
+            if (command[0].isEmpty) {
+              throw new DartdocOptionError(
+                  'Tool commands must not be empty. Tool $name command entry '
+                  '"$commandFrom" must contain at least one path.');
+            }
+          } else if (toolMap[commandFrom] is YamlList) {
+            command = (toolMap[commandFrom] as YamlList)
+                .map<String>((node) => node.toString())
+                .toList();
+            if (command.isEmpty) {
+              throw new DartdocOptionError(
+                  'Tool commands must not be empty. Tool $name command entry '
+                  '"$commandFrom" must contain at least one path.');
+            }
+          } else {
+            throw new DartdocOptionError(
+                'Tool commands must be a path to an executable, or a list of '
+                'strings that starts with a path to an executable. '
+                'The tool $name has a $commandFrom entry that is a '
+                '${toolMap[commandFrom].runtimeType}');
+          }
+        }
+      } else {
+        throw new DartdocOptionError(
+            'Tools must be defined as a map of tool names to definitions. Tool '
+            '$name is not a map.');
+      }
+      if (command == null) {
+        throw new DartdocOptionError(
+            'At least one of "command" or "${Platform.operatingSystem}" must '
+            'be defined for the tool $name.');
+      }
+      var executable = command.removeAt(0);
+      executable = pathContext.canonicalize(executable);
+      var executableFile = new File(executable);
+      var exeStat = executableFile.statSync();
+      if (exeStat.type == FileSystemEntityType.notFound) {
+        throw new DartdocOptionError('Command executables must exist. '
+            'The file "$executable" does not exist for tool $name.');
+      }
+      // Dart scripts don't need to be executable, because they'll be
+      // executed with the Dart binary.
+      bool isExecutable(int mode) {
+        return (0x1 & ((mode >> 6) | (mode >> 3) | mode)) != 0;
+      }
+
+      if (!executable.endsWith('.dart') && !isExecutable(exeStat.mode)) {
+        throw new DartdocOptionError('Non-Dart commands must be '
+            'executable. The file "$executable" for tool $name does not have '
+            'executable permission.');
+      }
+      newToolDefinitions[name] =
+          new ToolDefinition([executable] + command, description);
+    }
+    return new ToolConfiguration._(newToolDefinitions);
+  }
+}
+
 /// A container class to keep track of where our yaml data came from.
 class _YamlFileData {
   /// The map from the yaml file.
@@ -158,9 +269,10 @@ class _OptionValueWithContext<T> {
       return pathContext.canonicalize(resolveTildePath(value as String)) as T;
     } else if (value is Map<String, String>) {
       return (value as Map<String, String>)
-          .map((String mapKey, String mapValue) => new MapEntry<String, String>(
-              mapKey, pathContext.canonicalize(resolveTildePath(mapValue))))
-          .cast<String, String>() as T;
+          .map<String, String>((String key, String value) {
+        return new MapEntry(
+            key, pathContext.canonicalize(resolveTildePath(value)));
+      }) as T;
     } else {
       throw new UnsupportedError('Type $T is not supported for resolvedValue');
     }
@@ -252,8 +364,7 @@ abstract class DartdocOption<T> {
   void _onMissing(
       _OptionValueWithContext valueWithContext, String missingFilename);
 
-  /// Call [_onMissing] for every path that does not exist.  Returns true if
-  /// all paths exist or [mustExist] == false.
+  /// Call [_onMissing] for every path that does not exist.
   void _validatePaths(_OptionValueWithContext valueWithContext) {
     if (!mustExist) return;
     assert(isDir || isFile);
@@ -264,6 +375,9 @@ abstract class DartdocOption<T> {
       resolvedPaths = valueWithContext.resolvedValue.toList();
     } else if (valueWithContext.value is Map<String, String>) {
       resolvedPaths = valueWithContext.resolvedValue.values.toList();
+    } else {
+      assert(false, "Trying to ensure existence of unsupported type "
+          "${valueWithContext.value.runtimeType}");
     }
     for (String path in resolvedPaths) {
       FileSystemEntity f = isDir ? new Directory(path) : new File(path);
@@ -1024,6 +1138,7 @@ class DartdocOptionContext {
   List<String> get includeExternal =>
       optionSet['includeExternal'].valueAt(context);
   bool get includeSource => optionSet['includeSource'].valueAt(context);
+  ToolConfiguration get tools => optionSet['tools'].valueAt(context);
 
   /// _input is only used to construct synthetic options.
   // ignore: unused_element
@@ -1065,11 +1180,11 @@ Future<List<DartdocOption>> createDartdocOptions() async {
         negatable: true),
     new DartdocOptionArgFile<double>(
         'ambiguousReexportScorerMinConfidence', 0.1,
-        help:
-            'Minimum scorer confidence to suppress warning on ambiguous reexport.'),
+        help: 'Minimum scorer confidence to suppress warning on ambiguous '
+            'reexport.'),
     new DartdocOptionArgOnly<bool>('autoIncludeDependencies', false,
-        help:
-            'Include all the used libraries into the docs, even the ones not in the current package or "include-external"',
+        help: 'Include all the used libraries into the docs, even the ones not '
+            'in the current package or "include-external"',
         negatable: true),
     new DartdocOptionArgFile<List<String>>('categoryOrder', [],
         help:
@@ -1078,8 +1193,8 @@ Future<List<DartdocOption>> createDartdocOptions() async {
     new DartdocOptionFileOnly<CategoryConfiguration>(
         'categories', CategoryConfiguration.empty,
         convertYamlToType: CategoryConfiguration.fromYamlMap,
-        help:
-            "A list of all categories, their display names, and markdown documentation in the order they are to be displayed."),
+        help: 'A list of all categories, their display names, and markdown '
+            'documentation in the order they are to be displayed.'),
     new DartdocOptionSyntheticOnly<List<String>>('dropTextFrom',
         (DartdocSyntheticOption<List<String>> option, Directory dir) {
       if (option.parent['hideSdkText'].valueAt(dir)) {
@@ -1249,5 +1364,12 @@ Future<List<DartdocOption>> createDartdocOptions() async {
     new DartdocOptionArgOnly<bool>('verboseWarnings', true,
         help: 'Display extra debugging information and help with warnings.',
         negatable: true),
+    new DartdocOptionFileOnly<ToolConfiguration>(
+        'tools', ToolConfiguration.empty,
+        convertYamlToType: ToolConfiguration.fromYamlMap,
+        help: 'A map of tool names to executable paths. Each executable must '
+            'exist. Executables for different platforms are specified by '
+            'giving the platform name as a key, and a list of strings as the '
+            'command.'),
   ];
 }