In this post we’ll understand why at some point a component library needs to migrate from a single package to multi-package architecture. We’ll also see how we can transform our code to support this infrastructure.

Where do we begin?

This is part 1 of a 3 part series

• Part 1 - Where do we begin?
• Part 2 - Supporting the development environment
• Part 3 - Publish all of the components!

Preface - the problem with a single package

You have an awesome Component Library, it started small, few components here and there. At that point it was fine, it was manageable, what could go wrong with just wrapping a few native elements with your own UI language?

After some time, while you didn’t even notice, things got out of hand. You now have 100+ components, ranging from custom Header for all apps of your product to Rich Text Editors and virtualized tables.

You have one package to rule them all and it’s getting very very hard. You now want to change something in Button, it means the entire package needs a new major version, even if most of the stuff haven’t changed.

Your users are probably aren’t happy with that.

Let’s say you don’t have an automatic publish in CI (that’s OK if you don’t, but you should definitely consider it). You only publish at certain times when you feel it’s ready. This means that between two following versions there could be a changes to more than one component.

For instance, you added features in Button and DatePicker and decided to publish a new version. But oh no! you now realized that there’s a regression in DatePicker, unfortuantely you don’t have the resources to handle it right now so this bug will remain there for some time.

Now think about it like a user of this component library, you want the new Button feature but obviously you don’t want the DatePicker bug. Your only option is to wait for this bug to get fixed. So you see, that’s not a good experience.

Your users are probably aren’t happy with that either.

All of this can be solved. Instead of publishing all of your components in a single package, what if we publish each component in it’s own package? That way if one component changes it will only affect that component^*.

Your users can choose to install a specific version for each component, they can now both avoid the DatePicker bug AND get the Button feature!

^* - It will also affect all other components that depend (both directly & indirectly) on that component.

Lerna to the rescue

If you’ll recall, we said things got out of hand and we have 100+ components, how are we ever going to publish 100+ packages?? If you’ve had some research (or just looked at the title), you probably realized that the tool to help us with that is no other than - lerna.

lerna is a tool to help you manage multiple packages in a single repository. Sounds simple enough, right? Visit https://lernajs.io/ and update your repository to it and you’re done! that’s it!

You didn’t really think it was that simple, right? I do encourage you to go and visit the lerna docs, you won’t find a migration guide - how to turn an existing codebase from one huge package to many multiple packages. Isn’t it odd though? a tool that’s meant to help you with a certain type of infrastructure doesn’t have a migration guide towards that infrastructure?

My guess would be that it differs between each type of project, and that it’s just too broad of a subject to cover. Since you’re here, you probably care about a component-library type of project. This means that you’re in luck! I’m going to help you migrate your component library codebase so that lerna could take it from there and manage it for you.

This of course, won’t be an exhuastive guide (the fact that it’s 3 parts should’ve already hinted it), but I’m going to cover a lot of topics so buckle up!

Diving deep - defining our basic structure

First thing’s first, let’s define our current design & folder structure, that way we can understand how it needs to be changed when moving to lerna, your design & structure may differ but probably not by much.

This is the basic folder structure:

  components
    |
    |--- Button
    |      |
    |      |--- Button.js
    |      |--- Button.css
    |      |--- index.js
    |      |--- tests
    |             |
    |             |--- Button.spec.js
    |             |--- Button.visual.spec.js
    |
    |--- DatePicker
    |      |
    |      |--- DatePicker.js
    |      |--- DatePicker.css
    |      |--- index.js
    |      |--- tests
    |             |
    |             |--- DatePicker.spec.js
    |             |--- DatePicker.visual.spec.js
  ...

We have a folder for each component so everything related to a component is encapsulated. You may be using scss or some CSS-in-JS tool to style your components, you may be using jest snapshots, storybook or styleguidist - it doesn’t matter, as long as there’s a clear way of determining what files belong to which component. Here we’re using the folders/file names to do just that.

With this folder structure we have also defined the way we import components. For instance, in IconButton.js we could see this:

import Button from '../Button';
...

We now know the basics. We know how the components interact with each other and what are the boundaries of each of them. We can now start to think about the changes needed for the migration.

Diving deep - plan ahead

lerna has a concept of packages, which means that you can define some root folder(s) where all of your managed packages are. This conveniently is our components folder - not much to do here but just tell lerna that this folder will now contain all of our packages.

Sometimes, it won’t be this easy, you might have some general helpers located elsewhere or other graphic assets you also want to encapsulate in packages. In this case you have two options, either move everything to a new folder called packages, or if it makes sense add another glob to the packages array in lerna.json to capture the other packages.

That is probably the easiest part of the migration, where all you have to do is move a bunch of folders around and change some configuration. Now for the fun part, let’s talk Regexes and ASTs!

Since every component is going to be packaged individually, it won’t be able to relatively import other components anymore. So, to fix that we’ll need to update those relative imports, they’re now going to be package imports for the right pacakge within our monorepo.

Instead of going through each and every file in our repo that contains 100+ components (that would be a very tedious and error prone way to do it) we’ll write a script. That script will go over all of the files and updates those import paths. There are a few advantages of using a script:

• you can easily revert the changes and recreate them in case of a mistake.
• You’ll probably update most, if not all files in the repo, the chance of conflicts is much higher. With the help of the script you can revert the changes, pull the latest code and re-run the script - no conflicts!

The question remains though, how do we do all of these changes in a script? We have two options, Transforming them with ASTs or using Regexes.

In the project I migrated, I used Regexes because I initially thought (also, I didn’t really plan ahead):

just need to take from '../Button'; and turn that to from '@my-scope/my-button';, that’s fairly simple.

Boy was I wrong. Turns out I had a whole bunch of types of imports and it wasn’t that simple. Some types look the same but the context is different (import of a file in a test folder looks the same as an actual component import, etc.).

There is an upside to using Regexes though, it’s super fast - updating 100+ components (~500 files) was just a couple of seconds. Plus, you finally learn Regexes for real, you know how to read & write them which is a very useful skill in itself!

The other option is using an AST. A little primer on AST - AST is short for Abstract Syntax Tree, and it’s a way to represent a language through a data structure. ASTs are used within compilers to transform your code to another language, mostly to machine language.

Some of the most popular tools in web development use ASTs, webpack, babel, rollup, they all use some kind of AST to read your code and transform it somehow. We can take advantage of these tools to create an AST for each of our files and produce a new file with the imports transformed.

No matter what you choose, you’ll probably have to do at least a little matching. So let’s see how you get from import Button from '../Button'; to import Button from '@my-scope/my-button';.

Diving deep - Regex

The Regex we’re going to use is /'(\.\.\/([^\/]*)(.*))';$/. Now let’s break it down.

• '\.\.\/ - we’re just matching the relative path. Since all of our components are placed flat under the same directory this relative path “identifies” a component import. Make sure you apply your identification of a component here.
• ([^\/]*) - This is a capturing group, you’ll later see why we need to capture this part. here we match every character that’s not /, so in the case of '../Button'; this will be Button (we’re later matching ';).
• (.*) - This is another capturing group, this time we match every character. With that, this group doesn’t even have to exist because * matches zero or more times.
• ';$ - This just matches the end of the import line making sure there are no extra characters we’re matching here.

Note: notice there’s another capturing group surrounding the entire import path - it’s there so we can replace the entire thing with whatever we want.

Now, why do we need these capturing groups? That’s because we need to replace the import using these captured groups

• First capture group allows us to replace the entire import. We will use a lesser known signature of String.prototype.replace that takes a method.

const newImport = import.replace(componentMatcher, (entireMatch, importPath, componentName, restPath) => {
    return entireMatch.replace(importPath, ...);
});

It may look a bit confusing, but think of it this way “In the import statement, match it with component statements, now replace each component match by taking the entire match and replacing the import path with something else”.

• Second capture group we get the component name which we can easily turn to @my-scope/my-button, as simple as

const packageName = `@my-scope/my-${kebabCase(componentName)}`;

• Third capture group is for cases where you might be doing this import ButtonType from '../Button/Types';. We want to maintain these imports, so we need to make sure to keep it. But now, each package will reference a “production” version of other packages. You can’t just use the same restPath, you would also need to include the build folder. So it would turn to something like this

const additionalPath = restPath ? `/es${restPath}` : '';

All together now!

const newImport = import.replace(componentMatcher, (entireMatch, importPath, componentName, restPath) => {
    const packageName = `@my-scope/my-${kebabCase(componentName)}`;
    const additionalPath = restPath ? `/es${restPath}` : '';
    return entireMatch.replace(importPath, packageName + additionalPath);
});

There are probably other variants of imports in your project, for instance if you’re using scss you’ll need to look for @import statements in scss files as well, or if you’re within test files these kinds of relative imports aren’t “component imports”.

Some import types I saw doing my own migration:

• Normal import - import X from 'x';
• Depceptively normal import - import X from "x"; (notice the ")
• CJS import - const X = require('x');
• SCSS import - @import 'x';
• Jest mock - jest.mock('x', ...);
• Dynamic import - import('x'); (this could also include webpack magic comments )

You now have the basic tools to transform your imports to their new way.

What’s next?

Next up - collecting dependencies.

Diving deep - collecting dependencies

It’s nice that we updated all of our imports to their new variants, but does your package manager actually know about them? nope.

We’ll collect all of these dependencies so later we’ll be able to build a package.json for each component.

How do we collect them you ask? easy, right where we replace them. We actually build the new package name, just need to add it to the list of dependencies of the component we’re currently working on.

const newImport = imports.replace(componentMatcher, (entireMatch, importPath, componentName, restPath) => {
    const packageName = `@my-scope/my-${kebabCase(componentName)}`;
    const additionalPath = restPath ? `/es${restPath}` : '';
    const dependencies[packageName] = true;
    return entireMatch.replace(importPath, packageName + additionalPath);
});

Assuming dependencies gets cleared after each component, it’ll contain all the new dependencies. But what about old dependencies? like external packages you’re using in the component, like lodash or classnames? we need to add them to our dependencies as well. For that we’ll build another Regex!

/'([^\.][^\/]*).*';$/

• '[^\.] - This means that we want to match ' followed by any character that’s NOT .. This is because we don’t want to match relative imports, and relative imports always begin with either ../ or ./ so an easy way to discard all of them is by not matching anything that starts with a ..
• [^\/]* - Like before, we want to match anything that’s not / and that’s because it’s a directory separator, and if we just want external package name, we don’t want to get the whole path.
• .*';$ - Even though we only want the package name, we still need to match the entire import, so if you’re importing something from within a package like import bind from 'classnames/bind'; there’s a “gap” between the end of the statement (';) and the package name (classnames), the .* will take care of matching it,

Note: The capturing group will capture “The first character that’s not . and all the characters up to the first /” which will be the external dependency.

Now, this time, instead of replacing the import we just need to extract the package name from it, so it’s much simpler

const [_, dependency] = /'([^\.][^\/]*).*';$/.exec(import);
dependencies[dependency] = true;

dependency will contain just the package name, so we just add it to the map of dependencies of our package.

Now, we’re basically done!

Wait, what? really? sort of, now all of your code should do the right thing, but we still need to hook everything up. we still need package.json for each component. It’s time to put all the pieces together!

Let’s write the package.json file for each component. What fields do we want to have in each of the package.jsons?

• name - Obviously the package name, taken from the directory we’re in.
• private - false, we want to publish this package.
• main - This will be the index file in our build folder (something like es/index.js)
• repository - Copy this from the original package.json since this is a monorepo, this doesn’t change.
• keywords - You’ll probably want something like ['react', 'component', 'library', componentName]
• description - Create a sentence that fits your component library and can easily include the component name.
• version - You have two options here, either reset everything to 1.0.0 or continue from the current version of your library, your choice.
• dependencies - Remember the previous secion? where we collected the dependencies? that’s what it was for. The only thing is that we’ll need to copy the actual version from the main package.json. Also, remember to filter out react & react-dom
• peerDependencies - We’ll just add react & react-dom (or any other framework/library you’re working with) with the version specificed in the main package.json.
• scripts - These will be the scripts that can be run by each component, we’ll cover them in the next post.

Even though this should theoretically be enough I’d recommend you to also symlink your general configs to each of your packages. Stuff like .npmrc, .npmignore, .babelrc, etc.

The reason I believe you should do that is each package should still be able to act as an individual package, it shouldn’t rely on the fact that it’s managed by lerna. In the off chance that a package grows big enough to be extracted to its own repo you’d have a relatively easy time doing it - it’s basically copy-paste.

So why not just copy these files to each package then? because if one of them changes - you want all of them to change, that’s why symlink is the best choice here.

Backward compatibility

If you’ll take a closer look you’ll notice that we have now broken the API of our previous “main” package. Now there’s no one entry point for all of your packages, your consumers must install each package individually in order to use them, while previously they only had to install one package.

To fix that we can create a “main” package, this package won’t be any component but rather just export all of them, one by one. You probably already had that kind of file already, something like:

export { default as Button } from './Button';
export { default as DatePicker } from './DatePicker';
export { default as TextInput } from './TextInput';
...

Again, we have to deal with normalizing the imports, which means we also have to collect the dependencies and all the other stuff. This makes the “main” another package we need to incorporate within our transformation script. Make sure you replace the imports correctly - these imports might be a bit different (notice ./ instead of the regular ../).

You’re now supporting a “backward compatible” package to give your consumers some peace of mind when they do actually upgrade.

We’re now ready for our next challange - fixing our development environment.