Declarative Transformations API #356

arumsey · 2024-05-16T19:32:51Z

Description

Introduce a higher-level Importer API that provides a declarative approach to transformations

offer a clearer API for developers to access that follows a consistent pattern
define a no-code configuration for most common transformation phases
allow for block-level transformation extensions
provide default metadata handling to avoid code duplication
opens the door for easier transformation automation
separates block transformations into their own modules

Related Issue

Fixes: #355

Motivation and Context

The AEM Importer import scripts are a powerful tool for manipulating a DOM into an Edge Delivery compatible format. These scripts however need to be written by developers with a high degree of knowledge of DOM APIs. The addition of a declarative transformation API on top of the existing low-level import script opens mor doors for less technical users to begin importing content by simply defining a collection of CSS selectors. Additionally, a no-code approach to defining import rules also allows them to be more easily created by automation and for them to be POSTED to a service to enable long running imports.

This feature does not introduce any breaking changes to the AEM Importer. The import script however can now be a JSON file or a JS object. This JSON structure (the ruleset) can then be passed to a createImporter factory that generates a valid import script. The Transformer class is able to consume a ruleset and runs through a series of phases to clean up the DOM and then generate any desired blocks through the use of CSS selector logic.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

I have signed the Adobe Open Source CLA.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

- added `Transformer` class - added import script that can consume transformation objects

- added support for import JSON object

- do not allow custom sections mapping script to bleed into other tabs

- remove test files

arumsey · 2024-05-16T19:33:56Z

js/shared/fields.js

@@ -18,14 +18,13 @@ const initOptionFields = (parent) => {
  const optionFields = getOptionFields(parent);
  optionFields.forEach((field) => {
    const value = localStorage.getItem(`option-field-${field.id}`);
-    if (value !== null) {
+    if (!field.classList.contains('locked') && value !== null) {


Fix to not have the custom sections mapping script bleed into other tabs.

arumsey · 2024-05-16T19:36:14Z

modules/transformer.js

+  }
+}
+
+export default class Transformer {


The Transformer could be added to helix-importer instead of being bolted on here. This class is where all the logic for processing a import ruleset lives. There are a number of phases that every import can participate in. The blocks phase is the most involved as there are a number of different parsing options that can be leveraged depending on the complexity of a site.

arumsey · 2024-05-16T19:36:53Z

js/shared/import.js

+    .replace(/[^a-z0-9/]/gm, '-');
+};
+
+const createImporter = (rules) => ({


A factory for creating compatible import scripts based on an import ruleset.

createImporter may be an overloaded term so I may try to come up with a more descriptive name.

arumsey · 2024-05-16T19:38:16Z

js/shared/pollimporter.js

+        const isImportScript = Object.keys(mod.default).some((key) => key === 'transformDOM' || key === 'transform');
+        if (isImportScript) {
+          $this.projectTransform = mod.default;
+        } else {
+          // declarative transformation
+          $this.projectTransform = createImportScript(mod.default);
+        }


Detect the type of import script and use the createImporter factory if needed.

arumsey · 2024-05-16T19:38:50Z

js/shared/pollimporter.js

+    const loadJson = (json) => {
+      try {
+        const importCfg = JSON.parse(json);
+        $this.projectTransform = createImportScript(importCfg);
+      } catch (err) {
+        console.error('Invalid transformation JSON');
+      }
+    };


An import script URL can now also be JSON for a no-code option.

arumsey · 2024-05-16T19:40:48Z

modules/parsers/block.js

@@ -0,0 +1,10 @@
+/* global WebImporter */
+
+export default function parse(element, { document, params: { cells } }) {


A generic block parser that is able to generate a Block table from a config object or two-dimensional array of cell rules.

arumsey · 2024-05-16T19:42:16Z

modules/transformer.js

+    // phase 3: block creation
+    blocks.forEach((blockCfg) => {
+      const { type, selectors = [], parse, target = 'replace', params = {} } = blockCfg;
+      const parserFn = parse || parsers[type] || parsers['block'];


Detect the type of parser that should be used.

arumsey · 2024-05-16T19:42:44Z

modules/transformer.js

+        // parse the element into block items
+        const items = parserFn.call(this, element, source);
+        // create the block
+        const block = WebImporter.Blocks.createBlock(document, {


A new function added to helix-importer. 👉 adobe/helix-importer#351

arumsey · 2024-05-16T19:43:50Z

modules/transformer.js

+   * @param element Root element to query from
+   * @param params Object of selector conditions
+   */
+  static buildBlockConfig(element, params) {


Function to map an object of name/value pairs with selector logic into a valid block config block.

arumsey · 2024-05-16T19:44:43Z

modules/transformer.js

+   * @param element
+   * @param cells
+   */
+  static buildBlockCells(element, cells) {


Function to map a two-dimensional array of selectors into actual elements that should be added to a block.

arumsey · 2024-05-16T19:45:53Z

Import Ruleset Example

{
  "root": "main",
  "cleanup": {
    "start": [
      ".cookie-status-message",
      ".breadcrumbs",
      ".messages",
      ".sidebar",
      "h1 + ul.instruments-menu",
      "h1"
    ]
  },
  "blocks": [
    {
      "type": "metadata",
      "target": "append",
      "params": {
        "metadata": {
          "keywords": "[name=\"keywords\"]",
          "Publication Date": "[property=\"og:article:published_time\"]",
          "category": [
            [":scope:has(.is-blog .post-list) .webinar-speaker-img", "Webinars"],
            [":scope:has(.is-blog .post-list)", ""]
          ],
          "series": [
            [":has(.is-blog .post-list) .webinar-speaker-img", ""]
          ],
          "eventDate": [
            [":has(.is-blog .post-list) .webinar-speaker-img", ""]
          ],
          "speakers": [
            [":has(.is-blog .post-list) .webinar-speaker-img", ".webinar-speaker-img + strong"]
          ]
        }
      }
    },
    {
      "type": "overview",
      "selectors": [
        ".entry div:has(div > p > img)",
        ".entry > div > div:first-of-type:has(div > img)"
      ],
      "params": {
        "cells": [
          ["div:has(p > img)", ":scope > div:last-child"]
        ]
      }
    },
    {
      "type": "columns",
      "selectors": [
        ".entry > .about-content",
        ".desc-img-wrapper:has(> :nth-child(2):last-child)"
      ]
    }
  ]
}

- add defaults to metadata parser

- refactor to `TransformFactory`

- add more comments

- add comments to `TransformFactory`

- add declarative transformation docs

- change `target` prop for blocks to `insertMode`

arumsey added 4 commits May 16, 2024 15:08

feat: Declarative Transformations API

857069d

- added `Transformer` class - added import script that can consume transformation objects

feat: Declarative Transformations API

120f9d1

- added support for import JSON object

feat: Declarative Transformations API

2443b4c

- do not allow custom sections mapping script to bleed into other tabs

feat: Declarative Transformations API

65853a1

- remove test files

arumsey marked this pull request as draft May 16, 2024 19:33

arumsey commented May 16, 2024

View reviewed changes

arumsey marked this pull request as ready for review May 21, 2024 14:06

arumsey added 6 commits May 21, 2024 10:09

feat: Declarative Transformations API

653ae61

- add defaults to metadata parser

feat: Declarative Transformations API

adfdae3

- refactor to `TransformFactory`

feat: [Importer] Transformer class

a726cfd

- add more comments

feat: Declarative Transformations API

9b42f2c

- add comments to `TransformFactory`

feat: Declarative Transformations API

83146e8

- add declarative transformation docs

feat: [Importer] Transformer class

a1139fc

- change `target` prop for blocks to `insertMode`

arumsey closed this May 23, 2024

arumsey deleted the transformation-object branch May 23, 2024 16:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Declarative Transformations API #356

Declarative Transformations API #356

arumsey commented May 16, 2024 •

edited

Loading

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey May 16, 2024

arumsey commented May 16, 2024

		@@ -0,0 +1,10 @@
		/* global WebImporter */

		export default function parse(element, { document, params: { cells } }) {

Declarative Transformations API #356

Declarative Transformations API #356

Conversation

arumsey commented May 16, 2024 • edited Loading

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arumsey commented May 16, 2024

arumsey commented May 16, 2024 •

edited

Loading