miniapp-tests

W3C Logo

Based on the EPUB testing methodology and code.

Writing tests for MiniApp

This repository contains tests to validate the implementations of the W3C’s MiniApp specifications, specifically MiniApp Packaging (the spec for the MiniApp format itself), MiniApp Manifest (the spec to define a MiniApp and its configuration), and MiniApp Lifecycle (the spec that defines the events and APIs to control the execution lifecycle of a MiniApp). Our objective is to test every normative statement (that is, every MUST or SHOULD or MAY, etc.).

Existing tests are described in the generated test reports.

This page explains how to write new tests.

Prerequisites

TODO: List the MiniApp user agents.

Step-by-step

  1. Find an untested normative statement in the MiniApp specifications (MiniApp Packaging, MiniApp Manifest, and MiniApp Lifecycle) specs to test — that is, a statement that does not have an expandable “tests” section. (Note that these links point at the working drafts of the spec on GitHub, not the published versions on w3.org; the published spec hides the “tests” sections. In the published versions, you can still see whether a statement is tested by checking whether its anchor element has a data-tests attribute.)

  2. Claim the normative statement by creating an issue in the w3c/miniapp-tests repo.

  3. If you are an owner of w3c/miniapp-tests, create a branch on that repo for your new test. Otherwise, fork the repo and create a branch on your fork. (It’s easier for reviewers to clone a PR to validate the test if it’s in the original repo.)

  4. Within the branch, copy the test template. Name your copy as explained in naming below.

  5. Modify the template as necessary to implement the test.

  6. Describe the test by adding the metadata documented below to the package document.

  7. Once your test is complete, compress the full directory into an MiniApp ZIP file.

  8. Open the MiniApp file in one or more MiniApp user agents to verify it behaves as expected. Fix as necessary.

  9. Create a pull request for your test change, including both the uncompressed folder and the compressed MiniApp file. Please ensure the PR’s description clearly indicates which statement is being tested. Await review.

  10. Once the pull request has been merged, fork the repo for the spec you are testing (i.e., MiniApp Packaging, MiniApp Manifest, or MiniApp Lifecycle).

  11. In the spec document, find the anchor element for the normative statement. If there is no anchor element, add one, using the same naming conventions as nearby anchors. Then add a data-tests attribute to the anchor element with the name(s) of your test(s) as comma-separated anchors:

    <li data-tests="#pkg-filesystem-root-pages,#pkg-filesystem-root-common">The file system of a package MUST have a file structure based on a root directory....</li>
    
  12. Create a pull request for your spec change and await review.

Naming

Test names should start with a three-letter abbreviation that corresponds to the value of the dc:coverage element below (for example, mnf for Manifest, pkg for Packaging, etc.), followed by a short hyphenated identifier that makes clear which requirement is under test.

If multiple tests are necessary for a single normative statement, differentiate the test cases by appending an underscore and a unique identifier.

Metadata

The root directory for each test must contain a test.jsonld file with the following metadata, which is used to generate test reports:

Example of test.jsonld document:

{
    "@context": {
      "dc": "http://purl.org/dc/elements/1.1/",
      "dcterms": "http://purl.org/dc/terms/",
      "foaf": "http://xmlns.com/foaf/0.1/",
      "earl": "http://www.w3.org/ns/earl#",
      "xsd": "http://www.w3.org/2001/XMLSchema#",
      "foaf:page": {
        "@type": "@id"
      }
    },
    "dcterms:rights": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
    "dcterms:rightsHolder": "https://www.w3.org",
    "@type": "earl:TestCase",
    "dc:coverage": "Manifest",
    "dc:creator": ["Jane Doe"],
    "dcterms:conformsTo": "must",
    "dc:date": "2022-07-08",
    "dc:title": "Fullscreen enabled in manifest",
    "dc:identifier": "mnf-window-fullscreen-true",
    "dc:description": "The window's fullscreen member is set to true in the manifest. The app must be shown in fullscreen.",
    "dcterms:isReferencedBy": [
      "https://www.w3.org/TR/miniapp-manifest/#dfn-process-the-window-s-fullscreen-member"
    ],
    "dcterms:modified": "2022-07-08T00:00:00Z"
}

(Note that, in this case, the dcterms:conformsTo property is not necessary, because that corresponds to the default value; it is only there as an example.)

Implementation reports

The reports directory contains implementation reports in form of JSON files, one per reading system. The structure of the JSON file is as follows:

Here is an example of a small test report:

{
    "name": "ACME Mini Programs",
    "ref": "https://example.org/",
    "variant" : "iOS",
    "tests": {
        "cnt-css-scoped-support": true,
        "mnf-window-fullscreen-default": true,
        "mnf-window-fullscreen-true": false,
        "mnf-window-orientation-default": true,
        "mnf-window-orientation-landscape": true,
        "mnf-window-orientation-portrait": false,
        "pkg-pages-same-filenames": true,
        "pkg-root-app-css-empty": true        
    }
}

The template file in reports/xx-template.json should list all available test identifiers, with all values set, initially, to null.

Generated test reports

When new tests are committed to the repo, a GitHub Actions workflow generates a report from the tests in the tests directory and implementation reports in the reports directory.

The report consists of two HTML pages, namely:

Debugging

If you want to create and debug your tests locally, you can do it using the local_tests/ directory instead of tests/. These local tests won´t be added to the repository (it´s in the .gitignore document). You can create the documentation based on these local tests, just running the debug script using the command npm run debug. The script will generate the documentation using the content of local_tests/.

Read more about Node.js and npm.

Generate .ma files

There is also another script that you can use to pack the unit tests you create. If you run npm run add_metadata, the script will add the dcterms:rightsHolder and dcterms:rights in every test description, if the metadata is not there. After that, the tests will be zipped into a .ma file with the same filename as the unit test directory you created.

For instance, the test under tests/cnt-css-global-support/ will generate the tests/cnt-css-global-support.ma MiniApp.

Acknowledgements

This testing methodology is based on the EPUB tests, generated and maintained by the EPUB 3 Working Group. So, a big thanks to Ivan Herman, Dan Lazin, and the rest of the group.