[Data Liberation] Expose experimental Markdown importer in the importWxr step

[adamziel]

🚧 Work in progress, don't merge 🚧

Enables importing markdown and epub files via the importWxr step (to be renamed) when the data-liberation importer is enabled.

CleanShot.2024-12-13.at.21.17.10.mp4

Here's the Blueprint you can use to import the "data basics" tutorial from the Gutenberg repo:

{
    "$schema": "https://playground.wordpress.net/blueprint-schema.json",
    "landingPage": "/adding-a-delete-button/",
    "features": {
        "networking": true
    },
    "steps": [
        {
            "step": "resetData"
        },
        {
            "step": "importWxr",
            "importer": "data-liberation",
            "phpImporterOptions": {
                "data_source": "markdown_directory",
                "source_site_url": "https://raw.githubusercontent.com/WordPress/gutenberg/HEAD/docs/how-to-guides/data-basics"
            },
            "importData": {
                "resource": "git:directory",
                "url": "https://github.com/WordPress/gutenberg.git",
                "ref": "HEAD",
                "path": "docs/how-to-guides/data-basics"
            }
        }
    ]
}

Requires WordPress/php-toolkit#121

Other code examples

Combining the new importers APIs is getting ridiculous. Here’s two entity readers:

• The first one sources posts, meta, etc. from XHTML files stored inside a remote .epub file
• The second one sources posts, meta, etc. from markdown files in a local .zip file

We can mix&match data sources (local filesystem, remote), formats (e.g. md, xhtml, wxr), and containes (plain, .zip, git in the future)

$reader = WP_Directory_Tree_Entity_Reader::create(
    new WP_Zip_Filesystem(
        WP_Remote_File_Ranged_Reader::create( 
            'https://github.com/IDPF/epub3-samples/releases/download/20230704/childrens-literature.epub'
        )
    ),
    array (
        'root_dir' => '/EPUB',
        'first_post_id' => 1,
        'allowed_extensions' => array( 'html', 'xhtml' ),
        'index_file_patterns' => array( '#^index\.x?html$#' ),
        'markup_converter_factory' => function( $content ) {
            return new WP_HTML_To_Blocks( $content );
        },
    )
);

$reader = WP_Directory_Tree_Entity_Reader::create(
    new WP_Zip_Filesystem(
        WP_File_Reader::create(__DIR__.'/../docs.zip')
    ),
    array (
        'root_dir' => '/',
        'first_post_id' => 1,
        'allowed_extensions' => array( 'md' ),
        'index_file_patterns' => array( '#^index\.md$#' ),
        'markup_converter_factory' => function( $content ) {
            return new WP_Markdown_To_Blocks( $content );
        },
    )
);

Remaining work

• Confirm the WXR import still works both for the regular importer and the data liberation one
• Add E2E coverage
• Rewrite relative markdown URLs
• Enable specifying additional URL mappings directly in the Blueprint
• Review the code and make any architectural adjustments necessary

[adamziel]

This PR needs to be split into smaller parts before merging. For sure the new vendor libraries will become a separate PR. Epub and HTML importers probably, too.

[adamziel]

I'm going to close this PR. I've reorganized it as a series of smaller ones that we can discuss granularly:

• [Data Liberation] Add Markdown parsing libraries #2092
• [Data Liberation] Move Markdown importer to a separate package #2093
• [Data Liberation] Build markdown importer as phar #2094
• [Data Liberation] Add HTML to Blocks converter #2095
• [Data Liberation] Refactor Entity Readers class diagram #2096
• [Data Liberation] Add EPub to Blocks converter #2097

After all the API changes, I'm no longer sure setting up the importer in blueprint.json in the way proposed in this PR will stand the test of time. Let's land all the plumbing from the above PRs and then discuss the public API in a dedicated discussion.