How We Maintain Our Public Flutter Project Using a Private Monorepo

At PSPDFKit, we maintain and develop a wide range of products to cover many platforms. To make this process easier to manage, we do most of our development in a monorepo, which houses the majority of the code for our products. This allows us to consolidate our development — including issue tracking, project management, and code reviews — in one place.

This blog post details how we migrated development of our public Flutter repository to our private monorepo. It also covers how doing so has enabled us to speed up the development and release cycle of our Flutter project while maintaining the public repository and giving our Flutter SDK some much needed love.

If you’re interested in learning more about why we find using a monorepo beneficial, check out our The Many Benefits of Using a Monorepo blog post.

The Goal

In the past, our Flutter project often suffered from pull requests that were open for way too long and delayed maintenance work that fell off the radar. To address this, we decided to move project management and development to our monorepo: In addition to making the public repo passively update while maintaining its history, we also wanted to implement a quick and painless release process that updated both the GitHub repo and our shiny new pub.dev Flutter package in one fell swoop, allowing us to push updates out to customers quickly and reliably.

The Planning

To achieve our goal, we had to do some research to find which tool was best suited for the job, as there are a number of great open source projects available that related to our goal of managing a public repo from a private monorepo.

We evaluated the following projects for this task and found pros and cons to each, which are outlined here:

• FBShipIt — A library for copying commits from one repository to another

  • Maintained regularly by Facebook

  • Facebook uses this for projects like React, and it’s feature-rich, allowing for community contributions

  • Very well documented

  • Overly complex for our use case; akin to using a sledgehammer to crack a nut

• Lazo — A handy program that catches changes in parts of a repository (such as a monorepo) and takes over the boring work of copying commits to the other repositories

  • Autodetects pushes to GitHub with a webhook and updates the relevant repositories automatically

  • A great solution with little need for maintenance, but takes away the control we require over our release cycle

• git-subrepo — A tool used to clone an external Git repo into a subdirectory of another repo, allowing upstream changes to be pulled and local changes to be pushed.

  • This tool is relatively simple and closely resembles our use case, but it doesn’t solve our problem exactly

• sync-monorepo — From the homepage: “Sync a monolithic repository into multiple standalone subtree repositories”

  • On the surface, this looked like the perfect solution. However, we have some idiosyncrasies in our use case that made this option a good reference, but not ideal.

The above tools cover a wide range of use cases, but during the evaluation process, it became clear that all of them were either too complicated and feature-rich for us, or they solved a specific problem that didn’t give us the flexibility required for our task.

So, in the end, we decided to roll our own tool, as it’d give us the control and flexibility we needed over our release process.

The Solution

With the decision made, it was time to start implementing. The following section describes the process — from repository migration, to releasing and publishing to GitHub and pub.dev.

Repo inside a Repo

First off, we wanted to copy the files of our public repo, along with their entire git histories, to a subdirectory in the monorepo. To do this, we used git filter-repo, which is a tool that enables rewriting the history of files; it’s useful for moving files from one folder to another and retaining their histories.

This is a painless task with the power of git. The following commands were all we needed:

# Start with a fresh clone of the public repo.
git clone [email protected]:PSPDFKit/pspdfkit-flutter.git flutter-temp
cd flutter-temp
# Use `filter-repo` to move the contents into a subfolder that matches the structure we require for the monorepo.
# Inside the monorepo, we want our Flutter project to be in the `<monorepo_root>/flutter/pspdfkit-flutter` path.
git filter-repo --to-subdirectory-filter flutter/pspdfkit-flutter
# Now we can head into the monorepo, which is next to the fresh clone...
cd ../monorepo
# ...and add the public repo as a remote.
git remote add flutter ../flutter-temp/
git fetch flutter
# And merge the repos together, making sure to turn on the `--allow-unrelated-histories` flag. This allows the merge to go ahead even though the histories share a common ancestor.
git merge --allow-unrelated-histories flutter/master
# No need for the `flutter` remote now.
git remote remove flutter

Once we did the above, all that was left to do was to open a pull request on the monorepo and make sure the team was happy with the result.

The Release Process

The monorepo merge allows development to follow the same workflow as our other products, and it provides easy access to all our existing tools and scripts that have matured and evolved over time. In turn, development is easier, safer, and faster, which allows us to push releases out regularly and with better testing and control.

For our Flutter project, we have to release to two places: GitHub and pub.dev. Now that we’ve contained the Flutter project inside our internal project at a fixed location, it’s easy to add some scripts that can handle the release process and update both public release locations.

We don’t want a situation where we force push to a single commit in our public repo, because maintaining history is important, so recommitting all the files and force pushing to a single commit isn’t an option. Instead, we take a clone of the public repo and rsync the changes in our local development folder across to the fresh clone. From there, we can commit only the changes and tag the commit with a new release version. GitHub recently released a handy command-line tool that helps us automate the release of a new version via a shell script.

Preparation

Here are the release stages in detail written in a pseudo-bash script format:

# Set up some variables — in our codebase, these are passed to the script as arguments, so this is easily adaptable for other projects.
INTERNAL_REPO_DIR="<monorepo_root>/flutter/pspdfkit-flutter"
CLONE_DIR="./temp-clone-dir"
VERSION="1.2.3"
BRANCH="master"

# 1. Grab a fresh clone of the public repository.
git clone [email protected]:PSPDFKit/pspdfkit-flutter.git $CLONE_DIR

# 2. Sync the local changes into the fresh clone.
#   We can exclude anything `git`-related because we don't want to mess up the public repo.
#   Also by filtering out anything in the `.gitignore` file, it makes the process much faster, as we ignore any untracked build files.
rsync -ach --delete \
    --exclude ".git/" --exclude "*/.git/" \
    --filter=":- .gitignore" \
    "<monorepo_root>/flutter/pspdfkit-flutter" "$CLONE_DIR"

cd "$CLONE_DIR"

# 3. Stage the changes in the freshly cloned repo:
git add -Av

# 4. Commit all the changes in one:
git commit -m "Release v$VERSION"

# 5. Tag the commit and push the changes:
git tag "$VERSION"
git push origin "$BRANCH" --tags

Now that the public repo is up to date, we can publish the releases on both GitHub and pub.dev.

Release

Scripting a GitHub release can be achieved with a one-liner from the gh command-line tool:

# Create a GitHub release using the `gh` command-line tool.
# We can take the release notes from the `CHANGELOG.md` file contained in the repo.
local release_notes_file="<monorepo_root>/flutter/CHANGELOG.md"
gh release create "$VERSION" --target "$BRANCH" --title "$VERSION" --notes-file $release_notes_file

And pub.dev releases are even easier, thanks to the Flutter and Dart developers streamlining the release process. The publish command makes sure your package has all the right files in the right places and prevents you from publishing if your code doesn’t match the pub.dev requirements.

# Publishing to pub.dev is straightforward once you've set up the publishing access for the package.
flutter pub publish

The flutter command will prompt for a confirmation, so for this to run automatically on our build servers, we can use yes:

if [ -n "$CI" ]; then
    yes | flutter pub publish
else
    flutter pub publish
fi

Great! Now we have an easy way of releasing on two separate platforms quickly. 🎉

You can see how this has increased our release output, with seven releases published in the last four months since changing to this process, as opposed to only nine releases since the project began 16 months prior to this change!

Conclusion

In this blog post, we looked at the process taken to solve a conceptually simple task of migrating a public repository to a monorepo and maintaining it. We discussed possible tools that can be used for the job, covered how rolling our own solution gave us the control and simplicity required to improve our release and development process, and saw how the move to the monorepo ultimately enables us to push out regular releases to our users.

Watch this space for more updates to our Flutter package, and give our SDK a try for free with our new, low-resistance, quick start guides.