Build tools research

[danielfdsilva]

Over last week, whenever I had some time, I've been looking into what's out there in terms of build systems.
This was motivated by the work we've done on APT whose complexity leads to a pretty big bundle and dependency tree.

This could be mitigated by having tree shaking and code splitting which is not currently supported by project-seed.

Project-seed still works pretty well but for cases like this it has some drawbacks. It uses Gulp as a task orchestrator, which is incredibly simple to use and its ecosystem of plugins make it very easy to perform different file transformations, and Browserify as a bundler which is outdated tech and there's really no plugin replacement.

A more modern tool does all the building (task orchestration) and bundling (file processing) as one, so I was looking around.
Staring from scratch is always easier, but I wanted to see which tool stood up to the task of being included in a project already underway.

There are some things I'm looking for, which are not very negotiable:

• Asset import - Ability to import files as urls (like for images) and support for different file types (like json, md, etc)
• Structure configuration (I want the tool to fit my needs, not the other way around)
• Three shaking & code splitting
• Source maps and debug (Debug must be possible through VS Code for those who use it, but also through the browser debugger)
• Plugin system (To perform other file transformations - this can always be achieved with auxiliary scripts, but having it all together is better)
• Babel features support / code minification
• Dev server

I haven't had a lot of time to dedicate to this yet, but I've done some preliminary experiments, none successful so far.
I'll use this issue to log my progress and findings.

PS: @drewbo I'm not sure if this could count as labs time and whether could make it a labs project, but let me know.

Interesting resources:
https://bundlers.tooling.report/
https://css-tricks.com/comparing-the-new-generation-of-build-tools/

cc @developmentseed/team-impact

---

⚠️ For the time being these are just notes - the research is not complete but I wanted to let you know what I'm doing. I'll keep updating with more structured content.

Webpack

I don't like webpack. I've always found it too confusing and hard to use, and I'm not including it in this research. I agree with what this person says in this article but you can also find twice as many webpack supporters.

Parcel

Was easy to install and setup, but I hit problems with having the browser debugger run. I wasn't able to get the source maps working (Update 2021/05/24 This was a bug, fixed in a nightly build)

I also hit a problem with all the paths linked in the index.html being hashed. One of them was <link rel='author' type='text/plain' href='humans.txt' />. The name of this file should not be changed, but I was not able to find a way to exclude it from the hashing. (Update 2021/05/25: It is a bit annoying that something like this is not easy to achieve, but the humans.txt is not super relevant nowadays and can be removed. There could be other use cases for this feature, but none for now.)

Update 2021/05/26
A feature I was looking for (to replicate what project-seed does) is to have string replacement in the html file.
This is useful to dynamically print the application title and description for SEO purposes, and having them centralized in a config file.
This can be achieved with the posthtml-expressions plugin.

Three shaking is done out of the box, and code splitting in different files has to be user initiated (which is a normal practice).
This is an architectural decision and has to be taken into account when building the app, but a very easy way to get started is to do it route based with loadable components. This already reduces the overall bundle and only loads things when needed.

One thing I haven't found a solution for is how to handle the processing of other files.
The APT needs are the following:

• Documentation pages are written in markdown with one file per page.
• Each page gets converted to a json file with the html content
• An index listing all the existent files is created showing the page title and the json file to load.

We need this process so that the pages can be loaded as a fetch request only when needed. The index is used to render a list of available pages.

With project seed, this is setup as a task, which calls some custom code. This task can be added to the processing pipeline and also called when the relevant files change.
There doesn't seem to be a straightforward way to do this with parcel. Creating a plugin is very complex and doesn't seem to be the correct approach here.
Perhaps this is really the task of an orchestrator and not a bundler, and a separate script should indeed be used.

Vite

I had not heard of this one but it is pretty interesting and fast!
One thing I don't like is that any file with jsx must have the .jsx extension, otherwise it is not processed. Apparently it's not a bug, it's a feature. The behavior can be overridden, but it's annoying.
I also wasn't able to get the devseed ui library to work with it (seems to be a problem with styled components - need to investigate better)

Update 2021/05/21
This one seems bust. During the development process Vite uses ES modules (via esbuild) which are available in modern browsers and when bundling for production it creates a bundle using rollup. The rationale is here.

Now, there are some libraries which are not packaged as ESM, like @devseed-ui which uses UMD. To handle these, Vite converts them to ESM so that they can be used, and here's where the problems start. In this case there's something weird going on between @devseed-ui and styled-components which causes errors. Apparently the problem is known and happens with other popular libraries (like react-datetime-picker).
This tools seems promising, but its unreliability voids its usage, at least for the time being.

Rollup

Update 2021/05/31

Rollup is another interesting bundler which is very extensible thanks to its ecosystem of plugins and very easy to setup.

Unlike webpack or parcel, rollup is just a bundler. It doesn't do task orchestration, and although you have plugins to handle other assets types and even create a web server it is not its raison d'etre. Iterative development with it is a pain, since it recompiles the assets every time and it takes precious seconds making the development process very slow and frustrating.

The biggest issue with Rollup is that it only provides one feature, creating production bundles. article

The author of the article above, created Nollup which is a Rollup compatible bundler, designed to be used in development. It works well during development, however:

• We'd introduce a third party library which may lead to additional maintenance (at time of writing the plugins used in nollup's starter are already deprecated)
• We have to follow the restrictions introduced by this wrapper, which encapsulates the build process making it difficult to add tasks related to other files (similar to parcel, this maybe should be solved by using separate scripts.)

Because of its extensibility and how easy it is to setup to do just what is needed, I'd pick this for libraries, but when it comes to development environment I'm afraid is a no-go.

[drewbo]

@danielfdsilva I think this is both awesome and should be tracked as "team time" (rather than external labs) as outlined here: https://github.com/developmentseed/labs/issues/233. I'll be releasing updated guidance on this in the next few days

[danielfdsilva]

After some intense googling and reading I finished this research.
If I were to start a project now, or retrofit an existent one, I'd use a combination of Parcel and Gulp.

I'd keep using Gulp for task orchestration and to handle additional file pipeline, and then Parcel to create the bundle.
You could argue that Gulp could be removed and only use node (or even bash) scripts to handle other files, but this would require you to run multiple commands and you'd lose the ability to watch files. I wanted a more complete solution and this is the best one I found.

I created a repo with an example project for you to see how this would work.
Check it out and let me know your thoughts.

If there's interest I'll create a project-seed version of this approach.

cc @developmentseed/team-impact