Server Driven UI: Prepared for the Unknown

You won’t know ‘till you get there

This blog post is the second half of a 2-part series on a new tool that REI’s Store Technology team has been working on for the last few months - the “Price Change Tool”. You can find Part 1 here.

For folks who haven’t read Part 1, this post comes from the Ascent team, a small-but-mighty product team that builds the iOS app used by REI store employees. Ascent includes features like product search, inventory lookup, and restocking task management. The recent addition of the Price Change Tool gives store employees a digital tool to help re-label items that change price.

At a high level, the Price Change Tool provides two main value-adds over the previous pen-and-paper solution:

1. An interface that makes properly executing a price change task simple and intuitive.
2. The ability to filter and sort price change tasks to match your unique work environment.

Thanks to a lot of prototyping work from our designer, Alyssa, we had a clear answer for #1 early into development, but a solution for #2 proved to be more illusive. This is because at REI, we have over 150 stores, each with a unique combination of square footage, staffing, warehouse to sales-floor ratio, traffic, age of the store, etc. This wide variety of cases made it daunting to design a solution that was supposed to work for all users across the enterprise. Even though we surveyed dozens of employees, we still weren’t confident that our initial release would cover all employee’s needs. We knew this was something we would need several iterations to get right, and we needed a way to do that quickly enough to not leave some employees without the support they needed.

How do you plan for the unknown?

Luckily, this tool was truly a greenfield project for us, so we were able to plan ahead for this kind of uncertainty and afford ourselves options ahead of time. The main way we did this was with our choice to leverage server-driven UI (SDUI). In short, SDUI is all about giving the backend as much of the control over how the UI looks as possible, so you can redesign the way the app looks without having to push an iOS app update (because who actually updates their mobile apps on a regular basis!?) While I wouldn’t say we fully embraced SDUI (like Airbnb has), we did choose to make UI components server-configurable wherever we were sufficiently unsure of our solution. As I mentioned before, filtering and sorting was really the place where we didn’t have the confidence to bake a single solution into our iOS app release.

Implementing SDUI in the Price Change Tool

In many ways, the API we created for the price change tool is just a really simple search interface. While it may not have a text search bar, we had to build a way for users to create custom queries that would search over the space of all price change activities.

Luckily, here at REI, we have two entire teams dedicated to search, so we asked for their wisdom on how to get started on such an endeavor.

Their advice: “you don’t want to set up an entire Solr instance for what you’re building.”

While there is a lot of features that come out-of-box with Solr (or Elasticsearch), they warned us that the amount of time and effort it takes to maintain that infrastructure wouldn’t be worth it for us. So now armed with what not to do, we set off to find a way to query our modestly sized database (~1M records) in a maintainable and performant manner.

Querydsl

Since our backend for the Price Change Tool is written in Java, and we connect to a PostgreSQL database through JPA, Querydsl stood out as a good option. Querydsl is an open source library that allows you to construct SQL queries in a simple Java object syntax. We had our concerns about performance when dynamically generating custom queries using Querydsl, but since launch, we’ve seen consistent <10ms P95 response times for our most common queries.

Overall, we’ve been very happy with the decision to use Querydsl. Most of the effort of setting up Querydsl for this use case came from designing a simple query language that we could translate filtering/sorting selections into Querydsl. Below are some high level notes on how the integration works, as well as a bonus section about making Querydsl query typing stronger (an area we found error-prone during development).

Example Implementation: Query Parameters -> SQL

Here’s a simple example of how we can take an arbitrary set of query params and convert them to SQL.

Sample URL

/rs/price-changes/list?filter=brand:Patagonia;department:Basics,Mens%20Outerwear

Converting Filter Values to a JPA query

Map<FacetKey, Collection<String>> filters = Map.of(BRAND, Set.of("Patagonia"), DEPARTMENT, Set.of("Basics", "Mens Outerwear"));

BooleanBuilder querydslPredicate = new BooleanBuilder();

for (Map.Entry<FacetKey, Collection<String>> entry : filters.entrySet()) {
    switch (entry.getKey()) {
        case BRAND -> querydslPredicate.and(QPriceChangeActivity.priceChangeActivity.brand.in(filters.get(entry.getKey())));
        case DEPARTMENT -> querydslPredicate.and(QPriceChangeActivity.priceChangeActivity.department.in(filters.get(entry.getKey())));
    }
}

return priceChangeActivityRepository.findAll(querydslPredicate);

The Resulting SQL Query

SELECT * FROM price_change_activity
WHERE brand IN ('Patagonia') AND department IN ('Basics', 'Mens Outerwear')

Of course, this example doesn’t show what it takes to parse the query string into the appropriate data structure, but that wiring is fairly trivial and only needs to be implemented once. Adding new filters simply becomes a matter of adding a new FacetKey case to the switch statement, and then exposing it as an option to the user.

Exposing Query Options with SDUI

Once we had the ability to convert query params to SQL, we just needed to tell the frontend client how to construct the query string. This is where the power of the server-driven UI pattern comes into play. We designed a flexible JSON schema that could allow us to create the filtering/sorting screens entirely from backend JSON responses. Using that same queryString from above as an example, here is the corresponding part of the API response which the client uses to generate the view:

{
    "filters": [
        {
            "name": "Brand",
            "selected": true,
            "resetQueryString": "?filter=department:Basics,Mens%20Outerwear",
            "options": [
                {
                    "label": "Patagonia",
                    "selected": true,
                    "queryString": "?filter=department:Basics,Mens%20Outerwear"
                },
                {
                    "label": "The North Face",
                    "selected": false,
                    "queryString": "?filter=brand:The%20North%20Face;department:Basics,Mens%20Outerwear"
                }
            ]
        }
    ]
}

While this example is much simpler than what we use in the Price Change Tool, it still shows how the backend can suggest a query that will give predictable results to the frontend, so it can display it in a nice way for the user to select. In addition to label and selected, we have grown this schema to include all kinds of things like doNotReorder, count, hasDefaultSelection, and showInQuickFilterRow. Each of these variables drive a different UI treatment that is hardcoded into the iOS client.

By following this pattern, we can make the iOS client forward-compatible with data the backend may not be ready to supply. We actually did this for our second phase of our pilot release, where I sent our frontend developer several handwritten JSON responses that mimicked what the backend response would someday look like. This way, we were able to support a wide variety of UI layouts without the need for disruptive frontend code changes.

So did it work?

Impact to our users

Since the time we released the tool in our first round of pilot stores, we have been able to rapidly iterate on filtering and sorting options in response to user feedback. We have made over 20 changes to filtering/sorting, each one going live without requiring an app update. In several cases, we have even been able to get a new filter into the hands of all users within a couple of days of receiving user feedback.

Here are several iterations of the app in the last month as we have added more and more filter functionality to the app.

Furthermore, our team has started to see requests to improve filtering and sorting in other areas of the app that don’t leverage this pattern. One of the greatest impacts I believe our team can have is giving our users hope that their technology frustrations can be solved, and they can be a part of that process! (This is the theme of Part 1 of this series)

Impact to our team

From our team’s perspective, this has significantly reduced the time required to deliver features. Here’s a comparison of the process for making a change to filtering in an older part of our app:

Overall, this design pattern frees our frontend dev and QA resources up to work on major UI changes and features, not plumbing minor data changes into an existing interface.

Bonus Section: Explicitly Typed Queries in Querydsl

Querydsl is great at producing queries that give you strongly typed parameters and responses, but unfortunately the queries themselves are not tied to a specific queryable type. In the Price Change Tool, we frequently need to map a queryString into queries for two different tables. Specifically, these queries may be for a table containing PriceChangeActivity entities, or StyleColorGroup entities. Since the queries aren’t explicitly typed, it’s easy to accidentally do things like this:

BooleanExpression be1 = QPriceChangeActivity.priceChangeActivity.storeId.eq("11"); 
BooleanExpression be2 = QStyleColorGroup.styleColorGroup.style.eq("123456");
BooleanExpression be3 = be1.and(be2); // compiler allows this
priceChangeActivityRepo.count(be3); // runtime error "Cannot find column 'style' in table 'price_change_activity'"

To make it harder to get caught up in this pitfall, I wrote a wrapper class called QueryDslExpression that attaches a JPA entity type to the query. Here’s how that looks in practice:

QueryDslExpression<PriceChangeActivity> qde1 = QueryDslExpression.of(QPriceChangeActivity.priceChangeActivity.storeId.eq("11"));
QueryDslExpression<StyleColorGroup> qde2 = QueryDslExpression.of(QStyleColorGroup.styleColorGroup.style.eq("123456"));
QueryDslExpression<PriceChangeActivity> qde3 = be1.and(be2); // compiler error

This class is trivial enough to show in its entirety here:

QueryDslExpression

public class QueryDslExpression<T extends QueryDslQueryableEntity> {

    private final BooleanBuilder expression;

    private QueryDslExpression(BooleanBuilder expression) {
        this.expression = expression;
    }

    public static <T extends QueryDslQueryableEntity> QueryDslExpression<T> empty() {
        return new QueryDslExpression<>(new BooleanBuilder());
    }

    public static <T extends QueryDslQueryableEntity> QueryDslExpression<T> of(BooleanExpression expression) {
        return new QueryDslExpression<>(new BooleanBuilder(expression));
    }

    public QueryDslExpression<T> and(QueryDslExpression<T> right) {
        if (right == null || right.getExpression() == null) {
            return this;
        }
        BooleanBuilder newBuilder = new BooleanBuilder(this.expression);
        newBuilder.and(right.getExpression());
        return new QueryDslExpression<>(newBuilder);
    }

    public QueryDslExpression<T> or(QueryDslExpression<T> right) {
        if (right == null || right.getExpression() == null) {
            return this;
        }
        BooleanBuilder newBuilder = new BooleanBuilder(this.expression);
        newBuilder.or(right.getExpression());
        return new QueryDslExpression<>(newBuilder);
    }

    public Predicate asPredicate() {
        return this.expression.getValue();
    }
}

QueryDslQueryableEntity

public interface QueryDslQueryableEntity {} // JPA @Entity that you want to query must implement this interface