Rakuten

Research background

In my previous position as a UX/UI Designer at Rakuten Payment, I was asked to review a product which was being developed by another team. It was a notification area inside a cashless payment app. The app is used widely in Japan, enabling people to make payments by QR codes.

It was an appropriation of a new fintech trend in Japan where banks and fintech apps send various campaigns to users in their “Inbox” in the form of notifications. As a user, I had found them very irrelevant and often ignored them. But as a UX investigator, I was eager to see the research behind it.

So, I took the opportunity to study the initial UX research. In my search and upon close inspection it was revealed that there was room for improvement since Information Architecture was not thoroughly explored.

I then proposed to conduct a research to validate the I.A. of the five categories in the notifications feature.

How to validate success?

The management wanted to add even more categories on the go. But the question was “is this the right way to go?”

If we put these messages in those categories or drawers, could most of the users find them intuitively without having to open and close each drawer? I believe this is the essence of I.A. which is rooted in library science.

Together with another UX designer, we investigated the ways we could validate and test the way information was organized. Of course, the question of usability about notifications was not if users could find the bell icon, tap on the categories and messages, and open the links successfully.

That is certainly important but not nearly enough to call a product “successful”. The real questions were what kind of information the users wanted to see, how they wanted it delivered, and if they could easily find it.

So we set out to investigate the validity of the current structure in an experiment by testing to see if users will sort given messages in the same way as they are currently organized. If they did, then we could be sure that our proposed model matches the users’ mental model. If not, we needed to iterate.

Research Design: Card-sorting

I gathered 20 sample notification messages prepared by our company and from the competitor company. I made a few trial card sorting versions online. Then after making several adjustments, we printed each message on small labels and put a magnet on each so that users could organize them easily on a board.

Then we invited interviewees to individually organize these messages into any categories that they see fit and give the categories labels. In this simple exercise, we understood how our users perceive these messages.

User results

As we expected, most users were not able to recreate the proposed groups. They had trouble identifying the nature of some messages and said they didn't want to be "notified" of such offers.

Users came up with all sorts of groupings based on their needs and prior experiences with notifications. Many just thought of them as spam and the rest wanted to pop all campaigns and deals from all sources into one pile.

We then asked users what categories were important to them and what kinds of messages they did not want to see under the notifications bell. I would hand them a large paper featuring a simplified scheme of the app and ask where they would look for such info. On the menu? Under the settings? Together with the other banners? In a separate box on the home.

The research proved that:

1. Most Users do not differentiate between the source of commercial campaigns as long as they were not tailored for them specifically. In other words, it did not matter to them if the offers came from the company itself or other merchants and they did not need those separated into different categories.
2. Most users did not care very much about these offers and would not go back more than a few days if they missed them.
3. Most Users did not wish to be notified of routine system messages. Examples included automatic system updates or new stores joining the company’s large network of merchants.
4. Most users thought of vital system or maintenance announcements as important. But almost all of them thought of them as time-based messages that were no longer relevant when expired.
5. Some users said they would look for system maintenance announcements under the main menu or in the settings sub-menu.
6. Some users said they expected to see campaigns and offers on the home page not in the notifications area.

Lessons learned

This exercise gave us very rich insights into how users think about these campaigns and offers disguised as “notifications”. We concluded that not only the proposed groups were inefficient for most users, but it was crucial to provide them with a degree of flexibility to customize their categories (like folders) and hide unwanted categories.

This research exercise initiated a good deal of healthy debate within the company. After all, what is the benefit of building a product that would be ignored by the users? In summary, the teams involved in this product development saw both the sensitivity of Information Architecture and also gained firsthand experience of how user research can unlock so much insights into the user's minds that were otherwise unknown to us.

The stakeholders were pleased with the initiative requesting our team to continue such research to validate usability and information organization in our products.