Skip to content

External tool support #1756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Oct 2, 2018
Merged
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## 0.22.0
* Added the ability to run external tools on a section of documentation and
replace it with the output of the tool.

## 0.21.1
* Fix a problem where category ordering specified in categories option
was not obeyed. Reintroduce categoryOrder option to solve this problem.
Expand Down
12 changes: 10 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,16 @@ yet in the issue tracker, start by opening an issue. Thanks!
2. When a change is user-facing, please add a new entry to the [changelog](https://github.com/dart-lang/dartdoc/blob/master/CHANGELOG.md)
3. Please include a test for your change. `dartdoc` has both `package:test`-style unittests as well as integration tests. To run the unittests, use `dart test/all.dart`. Most changes can be tested via a unittest, but some require modifying the [test_package](https://github.com/dart-lang/dartdoc/tree/master/testing/test_package) and regenerating its docs via `grind update-test-package-docs`.
4. For major changes, run `grind compare-sdk-warnings` and `grind compare-flutter-warnings`, and include the summary results in your pull request.
5. Be sure to format your Dart code using `dartfmt -w`, otherwise travis will complain.
6. Post your change via a pull request for review and integration!
5. Be sure to format your Dart code using `dartfmt -w`, otherwise travis will complain.
6. Because there are generated versions of the dartdoc docs for stable and development versions of Dart,
you need to update the docs twice:
- Download and install the latest STABLE version of dart from [the Dart website](https://www.dartlang.org/tools/sdk).
(It's probably easiest to download a zip file and change your PATH to the extracted location's `bin` directory)
- Run `pub run grinder update-test-package-docs` to update the stable docs.
- Download and install the latest DEV version of dart from [the Dart website](https://www.dartlang.org/tools/sdk)
(It's probably easiest to download a zip file and change your PATH to the extracted location's `bin` directory)
- Run `pub run grinder update-test-package-docs` to update the dev docs.
7. Post your change via a pull request for review and integration!

## Testing

Expand Down
58 changes: 56 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,15 +221,15 @@ You can specify links to videos inline that will be handled with a simple HTML5
You can specify "macros", i.e. reusable pieces of documentation. For that, first specify a template
anywhere in the comments, like:

```
```dart
/// {@template template_name}
/// Some shared docs
/// {@endtemplate}
```

and then you can insert it via `{@macro template_name}`, like

```
```dart
/// Some comment
/// {@macro template_name}
/// More comments
Expand All @@ -240,6 +240,60 @@ dartdoc is currently documenting. This can lead to inconsistent behavior betwee
packages, especially if different command lines are used for dartdoc. It is recommended to use collision-resistant
naming for any macros by including the package name and/or library it is defined in within the name.

### Tools

Dartdoc allows you to filter parts of the documentation through an external tool
and then include the output of that tool in place of the given input.

First, you have to configure the tools that will be used in the `dartdoc_options.yaml` file:

```yaml
dartdoc:
tools:
drill:
command: ["bin/drill.dart"]
description: "Puts holes in things."
echo:
macos: ['/bin/sh', '-c', 'echo']
linux: ['/bin/sh', '-c', 'echo']
windows: ['C:\\Windows\\System32\\cmd.exe', '/c', 'echo']
description: 'Works on everything'
```

The `command` tag is used to describe the command executable, and any options
that are common among all executions. If the first element of this list is a
filename that ends in `.dart`, then the dart executable will automatically be
used to invoke that script. The `command` defined will be run on all platforms.

The `macos`, `linux`, and `windows` tags are used to describe the commands to
be run on each of those platforms.

The `description` is just a short description of the tool for use as help text.

Only tools which are configured in the `dartdoc_options.yaml` file are able to
be invoked.

To use the tools in comment documentation, use the `{@tool <name> [<options> ...] [$INPUT]}`
directive to invoke the tool:

```dart
/// {@tool drill --flag --option="value" $INPUT}
/// This is the text that will be sent to the tool as input.
/// {@end-tool}
```

The `$INPUT` argument is a special token that will be replaced with the name of
a temporary file that the tool needs to read from. It can appear anywhere in the
options, and can appear multiple times.

If the example `drill` tool with those options is a tool that turns the content
of its input file into a code-font heading, then the directive above would be
the equivalent of having the following comment in the code:

```dart
/// # `This is the text that will be sent to the tool as input.`
```

### Auto including dependencies

If `--auto-include-dependencies` flag is provided, dartdoc tries to automatically add
Expand Down
144 changes: 133 additions & 11 deletions lib/src/dartdoc_options.dart
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ class DartdocFileMissing extends DartdocOptionError {
DartdocFileMissing(String details) : super(details);
}

/// Defines the attributes of a category in the options file, corresponding to
/// the 'categories' keyword in the options file, and populated by the
/// [CategoryConfiguration] class.
class CategoryDefinition {
/// Internal name of the category.
final String name;
Expand All @@ -73,6 +76,8 @@ class CategoryDefinition {
String get displayName => _displayName ?? name;
}

/// A configuration class that can interpret category definitions from a YAML
/// map.
class CategoryConfiguration {
/// A map of [CategoryDefinition.name] to [CategoryDefinition] objects.
final Map<String, CategoryDefinition> categoryDefinitions;
Expand Down Expand Up @@ -110,6 +115,112 @@ class CategoryConfiguration {
}
}

/// Defines the attributes of a tool in the options file, corresponding to
/// the 'tools' keyword in the options file, and populated by the
/// [ToolConfiguration] class.
class ToolDefinition {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice if there were some brief docs for the new classes, at least

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

/// A list containing the command and options to be run for this tool. The
/// first argument in the command is the tool executable. Must not be an empty
/// list, or be null.
final List<String> command;

/// A description of the defined tool. Must not be null.
final String description;

ToolDefinition(this.command, this.description)
: assert(command != null),
assert(command.isNotEmpty),
assert(description != null);

@override
String toString() => '$runtimeType: "${command.join(' ')}" ($description)';
}

/// A configuration class that can interpret [ToolDefinition]s from a YAML map.
class ToolConfiguration {
final Map<String, ToolDefinition> tools;

ToolConfiguration._(this.tools);

static ToolConfiguration get empty {
return new ToolConfiguration._({});
}

static ToolConfiguration fromYamlMap(
YamlMap yamlMap, pathLib.Context pathContext) {
var newToolDefinitions = <String, ToolDefinition>{};
for (var entry in yamlMap.entries) {
var name = entry.key.toString();
var toolMap = entry.value;
var description;
List<String> command;
if (toolMap is Map) {
description = toolMap['description']?.toString();
// If the command key is given, then it applies to all platforms.
var commandFrom = toolMap.containsKey('command')
? 'command'
: Platform.operatingSystem;
if (toolMap.containsKey(commandFrom)) {
if (toolMap[commandFrom].value is String) {
command = [toolMap[commandFrom].toString()];
if (command[0].isEmpty) {
throw new DartdocOptionError(
'Tool commands must not be empty. Tool $name command entry '
'"$commandFrom" must contain at least one path.');
}
} else if (toolMap[commandFrom] is YamlList) {
command = (toolMap[commandFrom] as YamlList)
.map<String>((node) => node.toString())
.toList();
if (command.isEmpty) {
throw new DartdocOptionError(
'Tool commands must not be empty. Tool $name command entry '
'"$commandFrom" must contain at least one path.');
}
} else {
throw new DartdocOptionError(
'Tool commands must be a path to an executable, or a list of '
'strings that starts with a path to an executable. '
'The tool $name has a $commandFrom entry that is a '
'${toolMap[commandFrom].runtimeType}');
}
}
} else {
throw new DartdocOptionError(
'Tools must be defined as a map of tool names to definitions. Tool '
'$name is not a map.');
}
if (command == null) {
throw new DartdocOptionError(
'At least one of "command" or "${Platform.operatingSystem}" must '
'be defined for the tool $name.');
}
var executable = command.removeAt(0);
executable = pathContext.canonicalize(executable);
var executableFile = new File(executable);
var exeStat = executableFile.statSync();
if (exeStat.type == FileSystemEntityType.notFound) {
throw new DartdocOptionError('Command executables must exist. '
'The file "$executable" does not exist for tool $name.');
}
// Dart scripts don't need to be executable, because they'll be
// executed with the Dart binary.
bool isExecutable(int mode) {
return (0x1 & ((mode >> 6) | (mode >> 3) | mode)) != 0;
}

if (!executable.endsWith('.dart') && !isExecutable(exeStat.mode)) {
throw new DartdocOptionError('Non-Dart commands must be '
'executable. The file "$executable" for tool $name does not have '
'executable permission.');
}
newToolDefinitions[name] =
new ToolDefinition([executable] + command, description);
}
return new ToolConfiguration._(newToolDefinitions);
}
}

/// A container class to keep track of where our yaml data came from.
class _YamlFileData {
/// The map from the yaml file.
Expand Down Expand Up @@ -158,9 +269,10 @@ class _OptionValueWithContext<T> {
return pathContext.canonicalize(resolveTildePath(value as String)) as T;
} else if (value is Map<String, String>) {
return (value as Map<String, String>)
.map((String mapKey, String mapValue) => new MapEntry<String, String>(
mapKey, pathContext.canonicalize(resolveTildePath(mapValue))))
.cast<String, String>() as T;
.map<String, String>((String key, String value) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes, better

return new MapEntry(
key, pathContext.canonicalize(resolveTildePath(value)));
}) as T;
} else {
throw new UnsupportedError('Type $T is not supported for resolvedValue');
}
Expand Down Expand Up @@ -252,8 +364,7 @@ abstract class DartdocOption<T> {
void _onMissing(
_OptionValueWithContext valueWithContext, String missingFilename);

/// Call [_onMissing] for every path that does not exist. Returns true if
/// all paths exist or [mustExist] == false.
/// Call [_onMissing] for every path that does not exist.
void _validatePaths(_OptionValueWithContext valueWithContext) {
if (!mustExist) return;
assert(isDir || isFile);
Expand All @@ -264,6 +375,9 @@ abstract class DartdocOption<T> {
resolvedPaths = valueWithContext.resolvedValue.toList();
} else if (valueWithContext.value is Map<String, String>) {
resolvedPaths = valueWithContext.resolvedValue.values.toList();
} else {
assert(false, "Trying to ensure existence of unsupported type "
"${valueWithContext.value.runtimeType}");
}
for (String path in resolvedPaths) {
FileSystemEntity f = isDir ? new Directory(path) : new File(path);
Expand Down Expand Up @@ -1024,6 +1138,7 @@ class DartdocOptionContext {
List<String> get includeExternal =>
optionSet['includeExternal'].valueAt(context);
bool get includeSource => optionSet['includeSource'].valueAt(context);
ToolConfiguration get tools => optionSet['tools'].valueAt(context);

/// _input is only used to construct synthetic options.
// ignore: unused_element
Expand Down Expand Up @@ -1065,11 +1180,11 @@ Future<List<DartdocOption>> createDartdocOptions() async {
negatable: true),
new DartdocOptionArgFile<double>(
'ambiguousReexportScorerMinConfidence', 0.1,
help:
'Minimum scorer confidence to suppress warning on ambiguous reexport.'),
help: 'Minimum scorer confidence to suppress warning on ambiguous '
'reexport.'),
new DartdocOptionArgOnly<bool>('autoIncludeDependencies', false,
help:
'Include all the used libraries into the docs, even the ones not in the current package or "include-external"',
help: 'Include all the used libraries into the docs, even the ones not '
'in the current package or "include-external"',
negatable: true),
new DartdocOptionArgFile<List<String>>('categoryOrder', [],
help:
Expand All @@ -1078,8 +1193,8 @@ Future<List<DartdocOption>> createDartdocOptions() async {
new DartdocOptionFileOnly<CategoryConfiguration>(
'categories', CategoryConfiguration.empty,
convertYamlToType: CategoryConfiguration.fromYamlMap,
help:
"A list of all categories, their display names, and markdown documentation in the order they are to be displayed."),
help: 'A list of all categories, their display names, and markdown '
'documentation in the order they are to be displayed.'),
new DartdocOptionSyntheticOnly<List<String>>('dropTextFrom',
(DartdocSyntheticOption<List<String>> option, Directory dir) {
if (option.parent['hideSdkText'].valueAt(dir)) {
Expand Down Expand Up @@ -1249,5 +1364,12 @@ Future<List<DartdocOption>> createDartdocOptions() async {
new DartdocOptionArgOnly<bool>('verboseWarnings', true,
help: 'Display extra debugging information and help with warnings.',
negatable: true),
new DartdocOptionFileOnly<ToolConfiguration>(
'tools', ToolConfiguration.empty,
convertYamlToType: ToolConfiguration.fromYamlMap,
help: 'A map of tool names to executable paths. Each executable must '
'exist. Executables for different platforms are specified by '
'giving the platform name as a key, and a list of strings as the '
'command.'),
];
}
Loading