Skip to content

collect MDN API information #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 13, 2024
Merged

collect MDN API information #130

merged 7 commits into from
Jan 13, 2024

Conversation

devoncarew
Copy link
Member

  • contribute a script which scrapes https://developer.mozilla.org/en-US/docs/Web for interface and interface member api documentation
  • write that info into a json file in this repo

This PR collects the info that we'd later want to write into the API dart docs for package:web; some work towards #122.


  • I’ve reviewed the contributor guide and applied the relevant portions to this PR.
Contribution guidelines:

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

@devoncarew devoncarew requested a review from srujzs December 29, 2023 03:33
@devoncarew devoncarew changed the title Generate docs collect MDN API information Dec 29, 2023
@kevmoo
Copy link
Member

kevmoo commented Dec 29, 2023

Oooo!

@devoncarew
Copy link
Member Author

It looks like the json file's too large to display as part of the PR. Here's the first few lines of it:

{
  "__meta__": {
    "source": "[MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web)",
    "license": "[CC-BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5/)"
  },
  "ANGLE_instanced_arrays": {
    "docs": "The **`ANGLE_instanced_arrays`** extension is part of the WebGL API and allows to draw the same object, or groups of similar objects multiple times, if they share the same vertex data, primitive count and type.\n\nWebGL extensions are available using the `WebGLRenderingContext.getExtension()` method. For more information, see also Using Extensions in the WebGL tutorial.\n\n**Note:** This extension is only available to WebGL1 contexts. In WebGL2, the functionality of this extension is available on the WebGL2 context by default and the constants and methods are available without the \"`ANGLE`\" suffix.\nDespite the name \"ANGLE\", this extension works on any device if the hardware supports it and not just on Windows when using the ANGLE library. \"ANGLE\" just indicates that this extension has been written by the ANGLE library authors.\n\nSee also https://developer.mozilla.org/en-US/docs/Web/API/ANGLE_instanced_arrays."
  },
  "AesCbcParams": {
    "docs": "The **`AesCbcParams`** dictionary of the Web Crypto API represents the object that should be passed as the `algorithm` parameter into `SubtleCrypto.encrypt()`, `SubtleCrypto.decrypt()`, `SubtleCrypto.wrapKey()`, or `SubtleCrypto.unwrapKey()`, when using the AES-CBC algorithm.\n\nSee also https://developer.mozilla.org/en-US/docs/Web/API/AesCbcParams."
  },
  "Accelerometer": {
    "docs": "**Experimental:** **This is an experimental technology**\n\nCheck the Browser compatibility table carefully before using this in production.\n\nThe **`Accelerometer`** interface of the Sensor APIs provides on each reading the acceleration applied to the device along all three axes.\n\nTo use this sensor, the user must grant permission to the `'accelerometer'`, device sensor through the Permissions API.\n\nThis feature may be blocked by a Permissions Policy set on your server.\n\nSee also https://developer.mozilla.org/en-US/docs/Web/API/Accelerometer.",
    "properties": {
      "x": "Returns a double containing the acceleration of the device along the device's x axis.",
      "y": "Returns a double containing the acceleration of the device along the device's y axis.",
      "z": "Returns a double containing the acceleration of the device along the device's z axis."
    }
  },
  "AbortController": {
    "docs": "The **`AbortController`** interface represents a controller object that allows you to abort one or more Web requests as and when desired.\n\nYou can create a new `AbortController` object using the `AbortController()` constructor. Communicating with a DOM request is done using an `AbortSignal` object.\n\nSee also https://developer.mozilla.org/en-US/docs/Web/API/AbortController.",
    "properties": {
      "abort": "Aborts a DOM request before it has completed. This is able to abort fetch requests, consumption of any response bodies, and streams.",
      "signal": "Returns an `AbortSignal` object instance, which can be used to communicate with, or to abort, a DOM request."
    }
  },
...

Copy link
Contributor

@srujzs srujzs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks Devon!

const encoder = JsonEncoder.withIndent(' ');

final file = File('tool/mdn.json');
final json = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any benefit to writing out the json and then consuming it later versus just having the node information in memory? If this is to just visualize the contents for now, then ignore this comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collecting the info from MDN is ~slower (around 20s) and could fail depending on networking conditions. My idea was to separate collecting the info (something you might run ~monthly) from using that info to generate docs + generated code (something you do on every commit).

It's probably not critical to separate these things out, but is what I was going for w/ this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see, I didn't realize it was that slow. Then yes, let's avoid this slowing down iteration of the generation script if it's that significant.

I'm a little concerned on how to keep this updated (both in terms of the the script you added and the docs in the output files) if we aren't running the script to run the docs when we generate code regularly. This may not be a big issue after the original generation, but we're likely to have stale documentation. For this CL and the following ones, it's okay to not worry about this for now.

@devoncarew
Copy link
Member Author

It looks like the system-of-record for this information are markdown files (e.g. https://github.com/mdn/content/blob/main/files/en-us/web/api/htmlmediaelement/src/index.md?plain=1). We may want to look into reading from them directly instead of from the served html.

@kevmoo
Copy link
Member

kevmoo commented Jan 2, 2024

It looks like the system-of-record for this information are markdown files (e.g. https://github.com/mdn/content/blob/main/files/en-us/web/api/htmlmediaelement/src/index.md?plain=1). We may want to look into reading from them directly instead of from the served html.

Big fan of separating download from generate. Makes iterative hacking on the generator a LOT easier.

@devoncarew devoncarew merged commit 42e86ca into main Jan 13, 2024
@devoncarew devoncarew deleted the generate_docs branch January 13, 2024 19:44
srujzs added a commit to srujzs/web that referenced this pull request Jan 22, 2024
Generate extension type bindings

Generate extension types instead of classes

Refactors some code around overrides so we don't emit @js
unless they're needed.

Updates code_builder dep to 4.10.0 and dart_style to 2.3.4.

Clean up JS supertype generation

contribute a fix_data.yaml file to enable 'dart fix' renames (dart-lang#141)

contribute a fix_data.yaml file to enable 'dart fix' renames

collect MDN API information (dart-lang#130)

collect MDN info for use in API documentation

Redeprecate deprecated APIs in 0.4.1 (dart-lang#142)

Undeprecate some APIs and prepare for publish of 0.4.2 (dart-lang#140)

move bindings_generator/ to tool/generator/ (dart-lang#138)

rev to 0.4.1 in preparation for publishing (dart-lang#135)

Update generate copyright year to 2024 (dart-lang#136)

Change-Id: I0f90fc08fd7871f8331f42c158cb7ba34e6bf887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants