MDN annotations have ended up in scraped data

[foolip]

git grep -l ✔MDN in reffy-reports shows that a number of specs have MDN annotations in the scraped data. For example, ed/css/css-align.json looks like this:

{
  "spec": {
    "title": "CSS Box Alignment Module Level 3",
    "url": "https://drafts.csswg.org/css-align/"
  },
  "properties": {
    "✔MDN align-content In all current engines. Firefox28+Safari9+Chrome29+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android28+iOS Safari9+Chrome for Android29+Android WebView4.4+Samsung Internet2.0+Opera Mobile12.1+ align-content In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-content": {
      "name": "✔MDN align-content In all current engines. Firefox28+Safari9+Chrome29+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android28+iOS Safari9+Chrome for Android29+Android WebView4.4+Samsung Internet2.0+Opera Mobile12.1+ align-content In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-content",
      "value": "normal | <baseline-position> | <content-distribution> | <overflow-position>? <content-position>",
      "initial": "normal",
      "appliesTo": "block containers, multicol containers, flex containers, and grid containers",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN justify-content In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ justify-content In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ justify-content": {
      "name": "✔MDN justify-content In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ justify-content In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ justify-content",
      "value": "normal | <content-distribution> | <overflow-position>? [ <content-position> | left | right ]",
      "initial": "normal",
      "appliesTo": "multicol containers, flex containers, and grid containers",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN place-content In all current engines. Firefox45+Safari9+Chrome59+ Opera46+Edge79+ Edge (Legacy)NoneIENone Firefox for Android45+iOS Safari9+Chrome for Android59+Android WebView59+Samsung Internet7.0+Opera Mobile43+ place-content In all current engines. Firefox53+Safari11+Chrome59+ Opera46+Edge79+ Edge (Legacy)NoneIENone Firefox for Android53+iOS Safari11+Chrome for Android59+Android WebView59+Samsung Internet7.0+Opera Mobile43+ place-content": {
      "name": "✔MDN place-content In all current engines. Firefox45+Safari9+Chrome59+ Opera46+Edge79+ Edge (Legacy)NoneIENone Firefox for Android45+iOS Safari9+Chrome for Android59+Android WebView59+Samsung Internet7.0+Opera Mobile43+ place-content In all current engines. Firefox53+Safari11+Chrome59+ Opera46+Edge79+ Edge (Legacy)NoneIENone Firefox for Android53+iOS Safari11+Chrome for Android59+Android WebView59+Samsung Internet7.0+Opera Mobile43+ place-content",
      "value": "<'align-content'> <'justify-content'>?",
      "initial": "normal",
      "appliesTo": "block containers, flex containers, and grid containers",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "see individual properties",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN justify-self In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-self In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-self": {
      "name": "✔MDN justify-self In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-self In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-self",
      "value": "auto | normal | stretch | <baseline-position> | <overflow-position>? [ <self-position> | left | right ]",
      "initial": "auto",
      "appliesTo": "block-level boxes, absolutely-positioned boxes, and grid items",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN align-self In all current engines. Firefox20+Safari9+Chrome36+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android36+Android WebView37+Samsung Internet3.0+Opera Mobile12.1+ align-self In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-self": {
      "name": "✔MDN align-self In all current engines. Firefox20+Safari9+Chrome36+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android36+Android WebView37+Samsung Internet3.0+Opera Mobile12.1+ align-self In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-self",
      "value": "auto | normal | stretch | <baseline-position> | <overflow-position>? <self-position>",
      "initial": "auto",
      "appliesTo": "flex items, grid items, and absolutely-positioned boxes",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "place-self": {
      "name": "place-self",
      "value": "<'align-self'> <'justify-self'>?",
      "initial": "auto",
      "appliesTo": "block-level boxes, absolutely-positioned boxes, and grid items",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "see individual properties",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN justify-items In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ justify-items In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-items": {
      "name": "✔MDN justify-items In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ justify-items In all current engines. Firefox45+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android45+iOS Safari10.3+Chrome for Android57+Android WebView57+Samsung Internet6.0+Opera Mobile43+ justify-items",
      "value": "normal | stretch | <baseline-position> | <overflow-position>? [ <self-position> | left | right ] | legacy | legacy && [ left | right | center ]",
      "initial": "legacy",
      "appliesTo": "all elements",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s), except for legacy (see prose)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "✔MDN align-items In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ align-items In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-items": {
      "name": "✔MDN align-items In all current engines. Firefox20+Safari9+Chrome52+ Opera12.1+Edge79+ Edge (Legacy)12+IE11 Firefox for Android20+iOS Safari9+Chrome for Android52+Android WebView52+Samsung Internet6.0+Opera Mobile12.1+ align-items In all current engines. Firefox52+Safari10.1+Chrome57+ Opera44+Edge79+ Edge (Legacy)16+IENone Firefox for Android52+iOS Safari10.3+Chrome for Android52+Android WebView57+Samsung Internet6.2+Opera Mobile43+ align-items",
      "value": "normal | stretch | <baseline-position> | [ <overflow-position>? <self-position> ]",
      "initial": "normal",
      "appliesTo": "all elements",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "specified keyword(s)",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "place-items": {
      "name": "place-items",
      "value": "<'align-items'> <'justify-items'>?",
      "initial": "see individual properties",
      "appliesTo": "all elements",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "see individual properties",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "⚠MDN row-gap In only one current engine. Firefox63+SafariNoneChromeNone OperaNoneEdgeNone Edge (Legacy)NoneIENone Firefox for Android63+iOS SafariNoneChrome for AndroidNoneAndroid WebViewNoneSamsung InternetNoneOpera MobileNone row-gap Firefox61+SafariNoneChrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS SafariNoneChrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ row-gap": {
      "name": "⚠MDN row-gap In only one current engine. Firefox63+SafariNoneChromeNone OperaNoneEdgeNone Edge (Legacy)NoneIENone Firefox for Android63+iOS SafariNoneChrome for AndroidNoneAndroid WebViewNoneSamsung InternetNoneOpera MobileNone row-gap Firefox61+SafariNoneChrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS SafariNoneChrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ row-gap",
      "value": "normal | <length-percentage>",
      "initial": "normal",
      "appliesTo": "multi-column containers, flex containers, grid containers",
      "inherited": "no",
      "percentages": "see § 8.3 Percentages In gap Properties",
      "computedValue": "specified keyword, else a computed <length-percentage> value",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value type"
    },
    "column-gap": {
      "name": "column-gap",
      "value": "normal | <length-percentage>",
      "initial": "normal",
      "appliesTo": "multi-column containers, flex containers, grid containers",
      "inherited": "no",
      "percentages": "see § 8.3 Percentages In gap Properties",
      "computedValue": "specified keyword, else a computed <length-percentage> value",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value type"
    },
    "⚠MDN gap In only one current engine. Firefox63+SafariNoneChromeNone OperaNoneEdgeNone Edge (Legacy)NoneIENone Firefox for Android63+iOS SafariNoneChrome for AndroidNoneAndroid WebViewNoneSamsung InternetNoneOpera MobileNone gap In all current engines. Firefox61+Safari12+Chrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS Safari12+Chrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ gap Firefox61+SafariNoneChrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS SafariNoneChrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ gap": {
      "name": "⚠MDN gap In only one current engine. Firefox63+SafariNoneChromeNone OperaNoneEdgeNone Edge (Legacy)NoneIENone Firefox for Android63+iOS SafariNoneChrome for AndroidNoneAndroid WebViewNoneSamsung InternetNoneOpera MobileNone gap In all current engines. Firefox61+Safari12+Chrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS Safari12+Chrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ gap Firefox61+SafariNoneChrome66+ Opera53+Edge79+ Edge (Legacy)16+IENone Firefox for Android61+iOS SafariNoneChrome for Android66+Android WebView66+Samsung Internet9.0+Opera Mobile47+ gap",
      "value": "<'row-gap'> <'column-gap'>?",
      "initial": "see individual properties",
      "appliesTo": "multi-column containers, flex containers, grid containers",
      "inherited": "no",
      "percentages": "refer to corresponding dimension of the content area",
      "computedValue": "see individual properties",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value type"
    }
  },
  "descriptors": {},
  "valuespaces": {
    "<self-position>": {
      "value": "center | start | end | self-start | self-end | flex-start | flex-end"
    },
    "<content-position>": {
      "value": "center | start | end | flex-start | flex-end"
    },
    "<baseline-position>": {
      "value": "[ first | last ]? baseline"
    },
    "<content-distribution>": {
      "value": "space-between | space-around | space-evenly | stretch"
    },
    "<overflow-position>": {
      "value": "unsafe | safe"
    }
  }
}

It looks like the <aside class="mdn-anno wrapped"> in https://drafts.csswg.org/css-align/ will have to be removed to solve this.

@dontcallmedom @tidoust @tabatkins

[foolip]

Looks like this was a recent regression, the first of these came in w3c/webref@4cd3ad9 on Jun 26, and it wasn't because of a change in Reffy.

[pyoor]

This issue doesn't appear to have been resolved. Parsing https://drafts.csswg.org/css-transforms-2/ still include MDN annotations in the extracted CSS definitions.

This is due to the fact that the annotations are not contained in aside elements. This may have changed since #355 landed. The following line should likely be changed to match the mdn-anno class.

const annotations = cleanedLine.querySelectorAll(".mdn-anno");

https://github.com/tidoust/reffy/blob/master/src/browserlib/extract-cssdfn.js#L52-L61

[dontcallmedom]

@pyoor I can't reproduce the problem - I'm assuming you're using the CLI script in src/cli/extract-cssdfn.js on https://drafts.csswg.org/css-transforms-2/ (since the automatically extracted data on webref only deals with transforms-1 at the moment) - when I do that, I don't see any MDN annotations in the extracted definitions:

{
  "properties": {
    "translate": {
      "name": "translate",
      "value": "none | <length-percentage> [ <length-percentage> <length>? ]?",
      "initial": "none",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "relative to the width of the containing block (for the first value) or the height (for the second value)",
      "computedValue": "the keyword none or a pair of computed <length-percentage> values and optionally an absolute length",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value, adding a third 0 value if needed to match components, but see below for none"
    },
    "rotate": {
      "name": "rotate",
      "value": "none | <angle> | [ x | y | z | <number>{3} ] && <angle>",
      "initial": "none",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "the keyword none, or an <angle> with an optional axis consisting of a list of three <number>s",
      "canonicalOrder": "per grammar",
      "animationType": "as SLERP, but see below for none"
    },
    "scale": {
      "name": "scale",
      "value": "none | <number>{1,3}",
      "initial": "none",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "n/a",
      "computedValue": "the keyword none, or a list of 2 or 3 <number>s",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value, but see below for none"
    },
    "transform-style": {
      "name": "transform-style",
      "value": "auto | flat | preserve-3d",
      "initial": "auto",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "N/A",
      "computedValue": "specified keyword",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    },
    "perspective": {
      "name": "perspective",
      "value": "none | <length>",
      "initial": "none",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "N/A",
      "computedValue": "the keyword none or an absolute length",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value"
    },
    "perspective-origin": {
      "name": "perspective-origin",
      "value": "<position>",
      "initial": "50% 50%",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "refer to the size of the reference box",
      "computedValue": "see background-position",
      "canonicalOrder": "per grammar",
      "animationType": "by computed value"
    },
    "backface-visibility": {
      "name": "backface-visibility",
      "value": "visible | hidden",
      "initial": "visible",
      "appliesTo": "transformable elements",
      "inherited": "no",
      "percentages": "N/A",
      "computedValue": "specified keyword",
      "canonicalOrder": "per grammar",
      "animationType": "discrete"
    }
  },
  "descriptors": {},
  "valuespaces": {
    "<matrix3d()>": {
      "value": "matrix3d( <number> [, <number> ]{15,15} )"
    },
    "<translate3d()>": {
      "value": "translate3d( <length-percentage> , <length-percentage> , <length> )"
    },
    "<translateZ()>": {
      "value": "translateZ( <length> )"
    },
    "<scale3d()>": {
      "value": "scale3d( <number> , <number>, <number> )"
    },
    "<scaleZ()>": {
      "value": "scaleZ( <number> )"
    },
    "<rotate3d()>": {
      "value": "rotate3d( <number> , <number> , <number> , [ <angle> | <zero> ] )"
    },
    "<rotateX()>": {
      "value": "rotateX( [ <angle> | <zero> ] )"
    },
    "<rotateY()>": {
      "value": "rotateY( [ <angle> | <zero> ] )"
    },
    "<rotateZ()>": {
      "value": "rotateZ( [ <angle> | <zero> ] )"
    },
    "<perspective()>": {
      "value": "perspective( <length> )"
    }
  }
}

[tidoust]

Behavior may be transient in that these annotations are probably generated asynchronously, and thus may or may not be present when the extraction runs. I get the problem on one of the properties of CSS Transforms 2 when I run things locally. Also, it appears in other spesc as well, such as in the CSS extract for CSS Text Module Level 3

[foolip]

Reopening as git grep -l ✔MDN still finds the annotations in some files:
ed/crawl.json
ed/css/css-align.json
ed/css/css-flexbox.json
ed/css/css-fonts.json
ed/css/css-lists.json
ed/css/css-logical.json
ed/css/css-overflow.json
ed/css/css-position.json
ed/css/css-text.json
ed/css/mediaqueries.json