A few months ago, I was proud to announce during a team meeting that I had successfully deployed the deeplinking service for a Flutter app we were developing. However, just a few days later, I was jolted awake in the middle of the night by a flurry of Slack alerts. The deeplinking service was down, and customers couldn't pay using our payment links! Although I had thoroughly tested the service, I hadn't accounted for one thing: our recent switch to using shortened URLs through an Iterable SMS campaign.
In this article, I will share the solution that finally fixed the deeplinking service's shortcomings. We'll dive into how we developed a URL expander app using Dart's HttpClient class, making our service more robust and reliable. First, we'll outline the app's structure, explaining its purpose and how it works. Then, we'll provide some practical usage examples to give you a clearer understanding of its capabilities. We'll also identify some initial limitations that we needed to address to make the tool more effective.
Next, we'll go through the steps of unit testing the app, ensuring that it meets our requirements and behaves as expected. Unit tests are crucial for confirming that the code functions properly, especially when changes are made in the future. Through this journey, we aim to provide you with a comprehensive understanding of how to tackle issues like the one we faced with deeplinking, thereby improving your own development practices.
What is a URL expander?
We are likely very familiar with URL shortening services like bit.ly which allow for users to generate shorter (sometimes custom, meaningful) URLs from long and potentially unwieldy URLs. However the task of resolving the original links is trivialised by modern browsers, which often resolve to original urls automatically, thus little attention is often paid to URL expansion.
However, in non-browser environments, like a Flutter mobile app, automatic URL resolution isn't a given. In these contexts, URL expanders become essential. You can think of a URL expander as a tool or service that resolves a shortened URL back to its original, longer form. In this article, we'll walk you through how to implement such a URL expander.
The URL Expander
Our URL Expander is implemented as a Dart CLI app exposing a utility via which shortened URLs may be expanded to their original form.
Overview of URL Expander App
The structure of the App patterns in agreement with the template for dart CLI apps as highlighted below.
.
├── CHANGELOG.md
├── README.md
├── analysis_options.yaml
├── bin
│ └── url_expander.dart
├── build
│ ├── test_cache
│ │ └── build
│ │ └── c075001b96339384a97db4862b8ab8db.cache.dill.track.dill
│ └── unit_test_assets
│ ├── AssetManifest.bin
│ ├── AssetManifest.json
│ ├── FontManifest.json
│ ├── NOTICES.Z
│ └── shaders
│ └── ink_sparkle.frag
├── coverage
│ ├── coverage.json
│ └── lcov.info
├── lib
│ └── url_expander.dart
├── pubspec.lock
├── pubspec.yaml
├── test
│ └── url_expander_test.dart
└── tree.txt
9 directories, 17 files
Steps to Url Expansion
- Get input URL (shortened).
- Get input base URL for expected expanded URL.
- Validate input URL.
- IF input URL is invalid throw an exception; else
- Open a HTTP HEAD request to fetch headers associated with the URL
- Close request to receive resolved headers.
- Extract location header from the response
- IF location header is not found then throw an exception; else
- Validate the location against the expected base URL for the expansion of input URL
- IF validation fails for location but there is a redirect in the response; recursively goto 1; decrease times we can goto 1; else
- IF validation fails and there is no redirect throw an exception; else
- IF validation is successful for location return it as expanded URL
The UrlExpander
encapsulates these steps, the UrlExpander
class is described in greater detail in the following section.
The UrlExpander
class
The UrlExpander
class is the central functional unit of our app. The UrlExpander class exposes a public method getFullUrl
, a wrapper to the encapsulated _extractFullUrl
method. The UrlExpander
class also encapsulates the _client
variable (it's HttpClient instance). The _isValidUrl
method and the _isValidFullUrl
method, which serve as checks and guards for licit expansions.
import 'dart:developer';
import 'dart:io';
class UrlExpander {
final HttpClient _client = HttpClient();
Future<String> getFullUrl(
String url, {
required String expandedBaseUrl,
}) async {
try {
return await _extractFullUrl(url, expandedBaseUrl);
} catch (e, t) {
log('$e', error: e, stackTrace: t);
throw Exception(e);
}
}
_extractFullUrl(
String url,
String expandedBaseUrl, [
int iterationsAllowed = 5,
]) async {
try {
if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
if (!_isValidUrl(url)) throw "Invalid URL";
final uri = Uri.parse(url);
final request = await _client.headUrl(uri);
request.followRedirects = false;
final response = await request.close();
stdout.write(
"\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");
final fullUrl = response.headers.value(HttpHeaders.locationHeader);
if ((fullUrl ?? '').isEmpty) throw "URL not found";
final isValidUrl = _isValidFullUrl(
fullUrl,
expandedBaseUrl,
);
if (!isValidUrl && response.isRedirect) {
return await _extractFullUrl(
fullUrl!,
expandedBaseUrl,
--iterationsAllowed,
);
}
if (!isValidUrl) throw 'Cannot fetch expanded URL';
return fullUrl!;
} catch (_) {
rethrow;
}
}
bool _isValidUrl(String? url) {
if (url != null && Uri.tryParse(url) != null) {
final RegExp urlRegex = RegExp(
r"\b[(http(s)?):\/\/(www\.)?a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)\b",
caseSensitive: false,
);
return urlRegex.hasMatch(url);
}
return false;
}
bool _isValidFullUrl(String? url, String baseUrl) {
if (url == null) return false;
return url.startsWith(baseUrl);
}
}
The _extractFullUrl
method
_extractFullUrl(
String url,
String expandedBaseUrl, [
int iterationsAllowed = 5,
]) async {
try {
if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
if (!_isValidUrl(url)) throw "Invalid URL";
final uri = Uri.parse(url);
final request = await _client.headUrl(uri);
request.followRedirects = false;
final response = await request.close();
stdout.write(
"\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");
final fullUrl = response.headers.value(HttpHeaders.locationHeader);
if ((fullUrl ?? '').isEmpty) throw "URL not found";
final isValidUrl = _isValidFullUrl(
fullUrl,
expandedBaseUrl,
);
if (!isValidUrl && response.isRedirect) {
return await _extractFullUrl(
fullUrl!,
expandedBaseUrl,
--iterationsAllowed,
);
}
if (!isValidUrl) throw 'Cannot fetch expanded URL';
return fullUrl!;
} catch (_) {
rethrow;
}
}
The _extractFullUrl
method is the central functional unit of the UrlExpander
class. The method receives three arguments url
, expandedBaseUrl
and iterationsAllowed
.
The url
argument serves as the shortened URL input. The expandedBaseUrl
serves as input to the associated _isValidFullUrl
method (along with the url
input), ensuring that only full urls that match the host expected for the expanded shortened URL are marked as valid. The iterationsAllowed
is primarily given in recursive calls to prevent runaway/infinite recursions.
The stepwise operations of the _extractUrl
method are as follows;
- WHEN recursively called it check that the recursive iteration is permitted in a guard; if not an exception is thrown.
- It checks that the input URL is valid in a guard else it throws an exception; then.
- It generates
Uri
object instance from the input URL - It creates a
HttpClientRequest
instancerequest
from an HTTP HEAD request on the generatedUri
. - We disable automatic redirects on the request instance.
- We create a
HttpClientResponse
instance from the result of closing the request. - We print the response header in green.
- We extract the
fullUrl
string from the location header - IF
fullUrl
is null or empty we throw an exception; else - We validate
fullUrl
against its expected base URL denoted by theexpandedBaseUrl
argument; then - IF
fullUrl
is invalid but response has a redirect, we recursively call_extractFullUrl
, reducing the value of theiterationsAllowed
argument; else - IF
fullUrl
is invalid but there is no redirect in the response we throw an exception; else; - We return
fullUrl
as the output.
With these operations stated, we find it necessary to highlight the use of the CLI, this is shown below.
Current gaps of the _extractFullUrl
method
The current iteration of the _extractFullUrl
method while apparently well forms, has a shortcoming. It fails to parse full URL inputs correctly, as full URLs may not return the location header from a HEAD
request. This is highlighted below:
We can introduce a guard to force an early exit from the method when a valid full URL is input as shown below.
_extractFullUrl(
String url,
String expandedBaseUrl, [
int iterationsAllowed = 5,
]) async {
try {
if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
if (!_isValidUrl(url)) throw "Invalid URL";
if (_isValidFullUrl(url, expandedBaseUrl)) return url;
final uri = Uri.parse(url);
final request = await _client.headUrl(uri);
request.followRedirects = false;
final response = await request.close();
stdout.write(
"\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");
final fullUrl = response.headers.value(HttpHeaders.locationHeader);
if ((fullUrl ?? '').isEmpty) throw "URL not found";
final isValidUrl = _isValidFullUrl(
fullUrl,
expandedBaseUrl,
);
if (!isValidUrl && response.isRedirect) {
return await _extractFullUrl(
fullUrl!,
expandedBaseUrl,
--iterationsAllowed,
);
}
if (!isValidUrl) throw 'Cannot fetch expanded URL';
return fullUrl!;
} catch (_) {
rethrow;
}
}
Consequently the behaviour of the method changes, to return the valid full URL immediately as shown below.
Testing the url expander
Unit tests were written to ensure that the url expander functions as expected in a plethora of scenarios, a presentation of these tests in action is presented below.
The source code for these tests may be found here.
This article made a presentation on the implementation of a URL expander CLI app in Dart. I would like to thank Kenneth Ngedo and Samuel Abada for their reviews on earlier drafts of this article.
Top comments (0)