Patrol 1.0 - Powerful Flutter UI Testing Framework

6 months ago, in September 2022, we released Patrol to the public, and it was warmly welcomed by the Flutter community. Since then, we’ve received a lot of feedback, and we’ve been working hard on improving it. Internally at LeanCode, our Quality Assurance guild has also been making great use of Patrol in many projects across different business domains. 

Today, we feel ready to put the “1.0” sticker on it. This means all the API changes we'll make in the 1.x.y versions will be backward-compatible. The first stable version of Patrol – a powerful, open-source UI testing framework for Flutter apps is out there, waiting for you to use it. 

Before you give our Patrol 1.0 a try, read this article. We share more about Patrol itself – what it is, how it was born, and what our vision is for it, but also, shortly, what it is capable of now. Later we also recommend reading about its latest update: Patrol has reached v1.1 and has a Hot Restart feature.

What is the Patrol framework?

For those who haven't heard of the Patrol yet, we're in a hurry to explain because it will make it much easier for you to test your mobile app. Patrol is a Flutter-native UI testing framework overcoming the limitations of existing Flutter testing tools (automated testing). It is also reliable because made by LeanCode’s team, who built many Flutter apps from startup to enterprise projects and use it internally. 

The Flutter team provides testing libraries, such as flutter_test and integration_test, which assume running inside a Flutter app and don't know about the native platform beneath. It works very well in isolated environments like widget tests but can be troublesome in end-to-end UI testing - especially in complex apps running on Android or iOS.

Patrol doesn’t replace the flutter_test and integration_test packages but enhances them with two major features:

• Native automation - which enables Flutter integration tests to interact with a native side of their app during Flutter tests, such as permission request dialogs and WebViews.
• Custom finders - they enhance Flutter finders to shorten and simplify your test code radically.

We'll describe these features in greater detail in a minute, but first, we'd like to share with you what prompted us to start working on such a solution as Patrol, and it’s quite a story to tell.

Why did we create Patrol?

It all started with an internal experiment, which resulted in early proof-of-concept of Flutter Patrol in June 2022. It was born out of our frustration and disappointment with existing Flutter integration testing solutions. At that time, the mobile apps we worked on were getting more extensive and complex. We’d already experienced a few unpleasant situations on production, which could’ve been easily prevented if we had integration tests written.

Unfortunately, after researching the topic, we came to the conclusion that no solution existed that could satisfy our needs and requirements. Even though Flutter provides the integration_test package, it falls short of completing scenarios that are presented virtually in every app, like granting permissions, tapping on notifications, or signing in using WebView. We were left with a single choice – create a solution ourselves, which we actually like doing.

We knew we'd make our new framework open-source from the very beginning. We are firm believers in open source and wanted to contribute back to the great Flutter community with a unique tool that would solve the most significant problems related to testing Flutter apps.

The aim of our POC was to research how interaction with the native platform could be enabled in Flutter integration tests. After a few weeks of intense work and brainstorming, we accomplished the goal, and the results were very promising. That’s how the first main feature of Patrol was born – native automation.

If you want to find out what Patrol could do before we’ve made it even better, then read our earlier announcement.

What problems does Patrol solve?

Testers at LeanCode have been using Patrol since day one, which provided us with invaluable, first-hand feedback. Now, Patrol is quite an advanced framework, and we have confirmed its effectiveness on mobile application projects in Flutter built at LeanCode. Currently, Patrol is used in several projects in domains such as construction, well-being, loans, and crypto banking. We're happy that our clients see the value integration testing can bring them in the form of more stable, polished apps.

Patrol solves integration testing in Flutter.

It makes it possible to test production-grade Flutter apps using Dart, the language Flutter developers are familiar with. This allows devs to be more productive and makes the experience – more fun and seamless. They no longer need to use a different language only for UI tests, which they previously did if they used a popular alternative testing framework – Appium.

With Patrol, we solved the most upvoted issues related to integration testing in Flutter:

• flutter_driver can't interact with native elements
• flutter driver and android permissions

Native automation

The first problem that Patrol solves is a situation where a Flutter integration test is being blocked by some native obstacle, e.g., a permission request dialog or a browser-based sign-in.

The occurrence of a browser-based login, a permission request dialog, a WebView, or any other native UI element made it impossible for the integration_test package to proceed past them. That's because the Flutter test framework knows only about Flutter widgets but doesn't know about the UI of the platform it's running on.

We've solved this problem by leveraging native UI testing frameworks (UIAutomator on Android and XCUITest on iOS) and enabling interaction with them from Dart code. At first, we took an approach very similar to Appium – the native automation was running in a separate app as a server (the "automation server app" – separate for Android and iOS), and a Flutter app (the "app under test") was communicating with it over HTTP.

We've also developed a command-line tool – patrol_cli – to make running tests fully automated and seamless for developers. To run tests using Patrol's native automation feature, developers had to use the `patrol drive` command, which could do the following:

1. Downloaded the automation server app and installed it on the device.
2. Started the automation server app, exposing the native automation feature on the device's localhost.
3. Installed and ran the app under test with flutter_driver.
4. Uninstalled the automation server app after tests finished, to keep the device clean.

Here's a sample test that signs in to the app presented in the first screenshot above:

patrolTest('signs in using browser-based login', (PatrolTester $) async {
  await $.pumpWidgetAndSettle(AwesomeApp());
  await $.native.enterText(
    Selector(textContains: 'Email'),
    text: 'charlie@root.me',
  );
  await $.native.enterText(
    Selector(textContains: 'Password'),
    text: 'ny4ncat',
  );
  await $.native.tap(Selector(text: 'Continue'));
  expect($('Welcome, Charlie!').waitUntilVisible(), findsOneWidget);
});

And to act on the native permission request dialog shown on the second screenshot, you can do:

await $.native.grantPermissionWhenInUse();

In retrospect, depending on flutter_driver had been a painful mistake, and to solve it, we had to revamp the entire architecture of Patrol – but more on that later.

Custom finders

While developing early PoC of Patrol to solve the inability to interact with native UI, we realized that we were writing lots of boilerplate widget finding code in our tests. Using methods like find.ancestor() and find.descendant(), looking for the n-th widget and building complex finders, ended up in writing quite a bunch of code that was not readable and took away the light from the important lines that really express the test intent.

We brainstormed how to make the test code leaner (ha!) and more expressive. We thought about JQuery and how fluent it was to write after little training. The result of that was the  $ – expression of finding widgets. On the other hand, our team at LeanCode is pretty obsessed with strong typing and having everything safely checked during compilation. The tradeoff of those two things results in our custom finders that help developers write common code faster and cleaner to focus on the real testing logic.

Our custom finders let you focus on testing your app’s functionality, freeing you from verbose syntax and low-level Flutter mechanisms like pumpAndSettle():

patrolTest('signs up', (PatrolTester $) async {
  await $.pumpWidgetAndSettle(AwesomeApp());
  await $(#emailTextField).enterText('charlie@root.me');
  await $(#nameTextField).enterText('Charlie Root');
  await $(#passwordTextField).enterText('ny4ncat');
  await $(#termsCheckbox).tap();
  await $(#signUpButton).tap();
  expect($('Welcome, Charlie!'), findsOneWidget);
});

The same test, but without Patrol:

testWidgets('signs up', (WidgetTester tester) async {
  await tester.pumpWidget(AwesomeApp());
  await tester.pumpAndSettle();
  await tester.enterText(find.byKey(Key('emailTextField')), 'charlie@root.me');
  await tester.pumpAndSettle();
  await tester.enterText(find.byKey(Key('nameTextField')), 'Charlie Root');
  await tester.pumpAndSettle();
  await tester.enterText(find.byKey(Key('passwordTextField')), 'ny4ncat');
  await tester.pumpAndSettle();
  await tester.tap(find.byKey(Key('termsCheckbox')));
  await tester.pumpAndSettle();
  await tester.tap(find.byKey(Key('signUpButton')));
  await tester.pumpAndSettle();
  expect(find.text('Welcome, Charlie!'), findsOneWidget);
});

The first example reads much better and doesn’t have all the unnecessary noise. The difference is not purely syntactic, though - Patrol’s custom finders also help to reduce flakiness, thanks to built-in timeouts and additional safety checks before performing actions.

How Patrol has changed in the last 6 months?

Since the first public release in September 2022, Patrol has drawn the attention of Flutter developers, which made us very happy, proving that such a tool was actually needed. What’s unique about Patrol is that it’s not only an enhancer but also an enabler – with Patrol, as you could see above, you can do things in your Flutter tests that simply weren’t possible before.

We've seen many Flutter developers trying it out and giving us feedback. You were reporting bugs, sharing ideas for new features, and contributing to code and documentation – we'd like to say a huge thank you to everyone who did that.

Documentation

Except for native automation and custom finders, there's one more thing that we're proud of – Patrol's documentation and API reference. We put a lot of work into creating documentation that makes Patrol easy to pick up for everybody – whether they're a Flutter beginner or an expert – so that they can get up and running in no-time and quickly write tests.

Apart from how to use Patrol, we now also have a few more pages in the docs:

• Tips and Tricks is a collection of bits of knowledge that Patrol users often find helpful
• Effective Patrol explains how to develop maintainable tests and avoid common pitfalls

We know our work on docs is paying off when developers praise them – thanks!

Running on CI

We made the Patrol open-source without giving much thought to CI integration to get it out of the door as soon as possible so the community could try it. Up then, we decided to prioritize things to work on based on the community's feedback.

After a few weeks, unsurprisingly, one issue on GitHub clearly stood out – running Patrol on CI, including managed device lab platforms like Firebase Test Lab.

At first, we thought it was a task just like any other, following the standard approach: research the topic, find and implement the solution, and share it with the world. We created a simple GitHub Action demonstrating how to run Patrol tests in CI. The action would spin up an Android emulator and call patrol drive to run the tests.

flutter_driver flakiness

Then we realized we had stepped on a landmine  - flutter_driver turned out to be very flaky. It was plain impossible to get reliable test results when running tests on emulators on platforms such as Bitrise and GitHub Actions. Out of 10 test runs, about 7 would never start up due to a bug in flutter_driver. 

After looking at Flutter issues on GitHub, we discovered that others were experiencing the same problem. We inspected the issue with flutter_driver and contributed a fix that significantly improved its stability, but unfortunately, we weren't able to fix it completely. 

Another problem was that our PR was merged to Flutter's master channel, while most apps use the stable channel (with some of the more adventurous ones using beta). We could either tell our users to use the master channel - which was unacceptable – or wait for Flutter to have a new stable release in a few months – which we also decided was inappropriate.

Test reports

As if depending on flutter_driver didn't already cause enough problems, there was one more – and it was the inability to get test run results in a machine-readable format.

We attempted to solve this problem by migrating from the flutter_driver package to the more modern integration_test package (which, by the way, depends on flutter_driver as well and is also impacted by the flakiness issue). Unluckily, we couldn't do that because integration_test has a hardcoded behavior that makes it impossible to use custom bindings – a lower-level Flutter mechanism that Patrol uses.

Firebase Test Lab

We also realized that the very way in which flutter_driver works is incompatible with device lab platforms, and Patrol heavily depended on flutter_driver. It became obvious that if we wanted to make Patrol successful, we had to rethink our approach seriously. This led us to Patrol Next.

Patrol Next

We entered the emergency mode. On a snowy November day, the Patrol team met in our Warsaw HQ for a few hours-long workshops. We found a few alternative approaches, considered their pros and cons, and put the whiteboard to good use.

We chose the best solution and called it Patrol Next.

We decided to bring the native automation code inside the app under test. This meant ditching the separate "automation server app" – its functionality now lives alongside your app. As with every technical matter, this also came with its tradeoffs – the setup got more challenging and complicated (we did our best to document it), but the pros outweighed the cons - we solved all the issues mentioned above.

Below is a screenshot showing that Patrol works on Firebase Test Lab flawlessly:

What is next for Patrol?

Now that we have officially reached 1.0, we have space to get back to the drawing board and focus on making Patrol even better. It's clear to us that the Flutter community needed such a tool, and we're happy to provide it.

We know that we've got something good in our hands, but we don't want to stop here. We want to make it as good as possible.

Below you can learn what the next things that we're going to work on are:

1. Faster feedback loops

One thing that we still have to improve is the speed. Flutter is famous for its Hot Restart and Hot Reload, and we all love these features. Unfortunately, they're unavailable when developing integration test code, making it slower and much less fun than the development of the app's source code.

There are two slightly related yet different scenarios that we aim to speed up:

1. Developing tests
2. Running tests on CI

We plan to take advantage of Hot Restart to speed up test development. It'll let you quickly rerun tests after making changes to your app's and tests' code. Currently, rerunning tests with even a tiny code change requires building a new application binary, and while advanced caching features of tools like Gradle (on Android) help, it's still too slow.

Another problem we aim to solve is the unnecessary app rebuilds when running integration tests from more than 1 Dart file. In Flutter, integration tests essentially replace your app's standard entrypoint (lib/main.dart), so to run 3 different tests, the app has to be rebuilt 3 times, each time with a new entrypoint. It is very suboptimal, and we're going to fix it.

Some of the work might involve working upstream in Flutter itself, which we're excited for.

2. Improving stability

We love developing new features, but we know that developing and maintaining a successful open source project also involves a lot of necessary "grunt work".

We strive to triage issues and fix bugs as fast as possible. In fact, we prioritize it over developing new features – when a new bug is reported, we rush to fix it so you won't waste your time.

Wrapping up about Patrol 1.0

Our goal with developing Patrol further is to deliver an excellent testing experience for Flutter developers and testers. Patrol should be like your wise but fun work buddy who supports you and makes your day (and testing UI elements of the app) easier and more likable. We want to make running Flutter UI tests something QA teams and devs want to do instead of something you have to do - like a mundane chore.

That's why we're committed to developing and supporting it. Patrol welcomes all sorts of contributions. If open-source is your thing, don't hesitate to shoot us a PR and identify things/test cases that we can improve. We count on you, and you can rely on us.

Meanwhile, what're you waiting for? Try out Patrol!

The newest update: Patrol is stable and has reached version 2.0.