Implementing Social Media Activity Feed with Stream

In a recent project, we embarked on the exciting task of enhancing an existing app with dynamic social media elements. Choosing Stream as our ally, we navigated through challenges and came up with strategies that helped us overcome them, resulting in the successful delivery of a news feed feature. Join us as we reveal our journey, offering insights into our experiences and thoughts on the Stream service.

Introduction

Social media is prevalent in today's Internet, available as various apps, portals, and services. These services may employ very different formats, but they all use the same mechanism to keep you updated with everything others share. When you enter your social media app of choice, you'll see a list of things other users have published, mixed with suggestions of content you may like. This dynamic stream of actions that users perform on the given platform is called an activity feed, news feed, or simply a feed.

Feeds are the way we consume digital content these days. Given how widely spread they are nowadays, it's rather hard to believe that when Facebook introduced such a feature for the first time, it was fiercely criticised. Today, it feels like the most natural way to show what other people are doing.

Recently, we were asked by one of our clients to implement such a functionality in their app. We've decided to use an external service to simplify our work. We choose Stream, a cloud-based solution that provides ready-to-use elements and infrastructure for building feeds with real-time updates. We want to share the story about our journey with Stream to deliver the feature, highlight challenges we encountered, and delve into our thoughts about the service. 

Long story short, using Stream significantly simplified the development, but not without its fair share of hurdles to overcome. Read on to learn about our love-hate relationship with the service.

What is an activity feed?

Even without the previous paragraphs, you're most likely familiar with the concept of feeds, at least from the perspective of an Internet user. From now on, we will focus on the developer's perspective and dive into their inner workings. Activity feeds are a general concept, not limited to Stream's implementation, but we'll focus on their take on this. It may be worth noting that Stream's model of activity feeds is based on one of the attempts to standardize the protocol, the first version of Activity Streams.

Feed’s anatomy

Let's take a closer look at a typical example of a feed with an activity: Gandalf shares an article with his friends, and this action is visible on his timeline. This takes us to the first important thing about a feed: it's a concrete thing that can be identified with a combination of the feed's group and its id, unique within a group. We'll talk about groups later. For now, it's enough to note that some arbitrary string is part of the feed's identity. For the second part, id, we used a GUID, but it's not necessary. Ultimately, it's just some arbitrary string as well.

The representation of the activity looks sparse, given how it finally looks materialized for the end user. The idea is to contain only a minimal amount of data that can be used to identify and describe the activity. For identification purposes, it has an ID, which Stream assigns, plus foreign_id and time, assigned by the developer. The latter two are optional, but in practice, you need them to be able to modify or delete the activity later on.

actor, verb, and object fields are mandatory – together, they “tell a story”, describing what an activity consists of. The actor tells who has performed a given activity, the verb defines what action it was, and the object is the target of this action. Both actor and object should usually reference the actual actor and thing involved in the activity. The verb is an arbitrary string, which allows you to model any activity that your app may need: here in the example, we’ve got a “share”, but it may as well be a “post”, “tweet”, “poke”, etc. Optionally, you can also attach some custom data to the activity. It’s useful e.g., for keeping some metadata, if you need any.

Activities' supporting data

We can see that Aragorn and Frodo enthusiastically reacted to the content Gandalf shared. Reactions are modeled similarly to activities as generic entities containing a reference to the parent activity and the user expressing a given reaction. Optionally, they can also contain additional data, e.g., the comment’s content. Different reaction types are implemented by providing different arbitrary strings for the kind field. 

In the example, we have two such kinds: “comment” by Frodo and “like” by Aragorn, which probably are the most obvious choices, but again, you could use anything that suits your application’s case. Stream uses reactions’ kind to group and filter them, allowing to easily display comments or show a number of likes for a given activity.

User’s data is stored separately, in a storage space predictably called Users. Each user entity kept there has a unique ID and, optionally, some arbitrary data, like the user’s name, avatar, or other application-specific values. Similarly, a dedicated space to store any content attached to activities is called Collections. Any collection item is assigned to a specific collection, has some ID and data, and references a user that owns the item. The “specific collections” are identified by their name and allow segregating items by type, or at least this is how we would use them. There are also files and image spaces dedicated to storing such types of data.

Our problematic case I: content optionally private

Before we delve into one of our problems, let's recap the most basic setup: each member of the Fellowship owns a single feed to which the said member publishes their activities. In our example setting, the activities would probably be "battles" that members fought and "decisions" they made so that a feed would tell the member's story. In a more typical example, activities could be posts and shares. The feeds are public, meaning that everyone may freely view them, their activities, and those activities' content.

However, one of our client requirements was to have optionally private content. The case is as follows: users can include additional data in their activities, such as photos or a location where they created the activity. However, they can decide to keep this data private and only visible to them. Moreover, users can join some groups, and each group should own its feed to which each group member may publish. Those group admins may decide that activities shared within a group would have the sensitive content visible or hidden to other group members by default, or even if this default may or may not be overridden by users.

Exploring possibilities

We obviously needed some logic to figure out if a particular user could see the sensitive content attached to a particular activity. The problem is how do we incorporate this logic into the Enrichment process? Is there a Conditional Enrichment so we could ask Stream to enrich only some activities? Unfortunately, this process is not configurable on such a level; we can either request it or not. We needed to find some workaround.

One of the simplest solutions that comes to mind is "just don't show it" on the client side. We didn't consider this as a viable approach, though. Given that one of the sensitive elements is the user's location, it would be a serious privacy concern if we send it to each client that reads a feed when the user has requested to hide it. So that's out of scope. We need to filter sensitive data before it's sent to the reader. We have no choice but to proxy feed reads through our server and ensure proper activity privacy.

We could have sensitive content stored, along with other data, in Stream’s Collections and erase what shouldn’t be seen by the user requesting feed access. That would work fine, but migration to their servers seemed unnecessary as we were implementing Stream integration in a matured project, and we already had a substantial amount of data stored on our servers. We decided to fill activities retrieved from Stream with sensitive content when applicable. We uploaded only users’ data, that is, their IDs, nicknames, avatars, etc., to be later retrieved via Enrichment on Stream’s side.

The final solution

We decided to go with a mix of server-side and client-side integration. We knew that reactions to activities are meant to be simple in our client's app and that privacy rules do not apply to them. There was no need to channel their reads through our server. By uploading users' data to Stream's Users, we enabled the mobile app to fetch reactions directly from the service, with everything needed to display them. However, we needed our server to broker feed reads. 

This way, we were able to enrich activities on the feed with data stored in our database while applying the content privacy rules that our client's business required. We still used Enrichment on the server side to include some metadata about activities' reactions in the response, such as the number of reactions of each kind.

Following your friends' activity on feeds in public

Until now, we only explained things based on an example of a single feed belonging to a single user. While this may be sufficient for some apps to have only one isolated feed for each user, most would be more demanding. For example, when Gandalf shares something with his friends, he probably expects that it will also pop up on their feeds. Or, if we imagine a Wizards of the Middle-Earth feed, we could expect that Gandalf’s and other wizards’ activities could be seen there.

One way of achieving that is to add Gandalf’s activity to other wizards’ feeds explicitly. This strategy, called Targeting by Stream, comes in handy in some scenarios. Still, it may not always be trivial to establish a list of feeds that a given activity should target beforehand. In our example, we couldn’t know whether a character is a wizard. In such cases, another option is much more appropriate: feeds follow. When one feed follows another, all activities added to the followed feed will automatically appear in the following feed. 

It’s important to note that propagating activities to following feeds is carried out according to a mechanism called Fan-out, meaning that the following relation is unidirectional and only one layer deep (non-transitive). The principle is visualized in the scheme below: when Gandalf follows Tom Bombadil’s feed, he gets all activities that appeared on Tom’s feed, but those activities do not propagate further to Saruman’s feed, which follows Gandalf’s. Also, Tom doesn’t get activities from Gadalf’s feed.

From the perspective of the code, establishing this relation between feeds is very straightforward; we can do this within a single API call. All the management of those relationships and propagating activities is done by Stream, making this very convenient to use. The feature shines especially bright when used to model a highly dynamic environment like a social network.

We used it for the first time while implementing a separate feed that gathered the activities of users that belonged to some group. Users may join and unjoin those groups freely, so we just made the group feed follow and unfollow users’ feeds on those events, respectively. And that was it, a big feature done in a few lines of code.

Our problematic case II: system-generated posts are private

This time, by “private,” we mean that only the target user can see them. Activities generated by the system, such as “Welcome in the App!” or “Congratulations on your thousandth activity! 🎉” and alike, whether they have personalised content or are just generic contextual messages, should only be visible to the target user. Those activities should appear on the user’s personal feed as well as on the group feeds when the user sees them. This was one of the first problems we were modeling, and honestly, we’re not fully satisfied with the solution we came up with. It stayed with us mainly because remodeling it would be pretty costly.

Our approach

Let’s gather the requirements for clarity. We need two types of feeds visible to the end user: personal and group feeds. We also have two types of activities: created by the user and created by the system. The personal feed contains activities of both types that concern the feed owner. Group feeds gather all activities created by its members; plus, when a user reads this feed, they also see activities generated by the system for them, but not ones generated for other users. Activities created by a user on both types of feeds are the same activities. When a user creates an activity, we automatically add it to their personal feed and to all group feeds of groups that the user is a member of.

We decided to store activities generated for each user separately so we can easily access them when a given user views a group feed. We published those activities to dedicated “technical” feeds created for each user; let’s name them system-activities feeds. When a user views their personal feed (aka user-activities feed) or a group feed, we fetch their system-activities feed alongside and merge both to show them the final result as specified. This solution works in simple cases but is not perfect by any means.

Following the ways of Stream

We wouldn't fall into this trap if we'd asked Stream to do the heavy lifting for us. The problem could be easily solved with a proper model of feed follows. We wouldn't need to do anything else if we had a dedicated personal view feed that follows both system-activities feed and user-activities feeds. Similarly, we could have a dedicated group view feed for each user in a given group that follows the user-activities feeds of other group members and the system-activities feed of a given user. However, another problem potentially emerges here: the number of feed updates grows exponentially with the number of group members. 

With small groups, this may not be a problem; however, we knew that some groups in our client's product count hundreds and some even thousands of members. If we consider that each feed update costs us money, this could escalate into an overwhelming bill. We feared it would happen to us and decided to build a model contrary to how Stream intended it. In the end, it was probably a bit of premature optimization.

Feeds’ groups, types, and wonders

We mentioned the concept of feed groups earlier, saying that a group is part of the feed's identity. But that's just the tip of the iceberg, as there are quite a few features centered around feed groups, or at least managed on their level. You may choose what the type of feeds should be within a given group or turn on additional functionality, like sorting feed activities using a custom formula or requesting updates in real-time about feed content. Let's take a further look at groups and their features.

Groups Revisited

When creating a new, concrete feed, e.g. Gandalf's timeline feed, we need to give it its ID and assign this feed to a group, which would be a "timeline" group in the example, referenced just by its name. The group may be arbitrary, but it must be predefined using Stream's dashboard and cannot be created programmatically. Ultimately, groups gather feeds that relate to the same context. 

Following the Middle-earth example, we could have a "travels" feed group for publishing important events regarding the journeys of the Fellowship members and a "battles" feed group that would have feeds dedicated to fights that each of them took part in, and so on. We could use the "member" group to gather all activities from more specific feeds so anyone could see the given member's story. Back to our world, feed groups have some interesting properties and features that can be configured.

Feed Type

A substantial property of a feed's group is its type. There are three of them: flat, aggregated, and notification feeds. A feed type defines how activities are stored on feeds, how they can be consumed, and what features are available for them. The default and most commonly used type of feed is a flat feed. Flat feeds are your typical news feeds, and they're just a stream of individual activities. They're suitable for adding and consuming new activities in a "timeline" way. Also, only flat feeds can be followed or have the sorting of activities customized. 

Aggregated feeds are a bit more involved: whenever an activity is added to such a feed, it’s added to a group of activities within a feed rather than as a standalone item. How exactly those in-feed groups are defined is configured per feed group using a template-like syntax. An example could be {{ actor }}_{{ verb }}, meaning that activities are aggregated based on the user who created the activity and the activity’s action. Aggregated feeds simplify implementing some features, like tracking the number of specific activities or reactions, e.g., displaying “Aragorn and 10 others commented on the article you shared”. 

Finally, notification feeds are aggregated feeds supercharged with the functionality of marking what was seen and read. They are designed specifically for easy implementation of in-app notifications.

Updates in real-time

Another interesting feature, which can be turned on for selected feed groups, is the possibility of getting updates about new content added to a feed in real-time. This requires some setup in the application's code as well as in Stream's dashboard, starting with choosing one of the available transports: WebSockets, webhook, or Amazon SQS. Once configured, Stream will send back to your application activities recently added to the feed. You may then, for example, notify your users that there is new content for them to see.

Custom ranking

As mentioned a few times earlier, a custom ranking allows to influence the order of activities on a feed, incorporating factors other than the activity's time. Our client considered using the feature, so we looked into it quite a bit, but it didn't happen in the end. It requires defining a function that will be used to calculate a score for each activity on the feed. Stream provides some predefined components to deal with the time-dependent part of the score, add a random factor to it, or use a high school level math, but generally, it's just a mathematical equation to be evaluated.

One downside of the feature is that it’s not easy to score different types of activities in another way, as no conditionals are available. We came up with some workarounds with clever math, but they were rather complex and seemed hackier than we liked. On the upside, though, is that reaction counts or analytics metrics can be easily incorporated into score evaluation. Stream also promises to actively work with their clients to establish the best scoring method for their business.

Our problematic case III: notify who reacted

This sounds reasonable to expect from an activity feeds service: notifications in the form similar to "Frodo and 7 others liked your article". Our client wanted to have such a feature in their app, but it turned out that it's not available out of the box in Stream. We needed to implement it ourselves, and, as you might anticipate, it didn't go smoothly. It was surprisingly burdensome, actually.

Technical difficulties

At the time, we were in between mobile and backend integration: activities and feed management were done via the backend, but reactions were added through the mobile client. We needed to catch them on the backend to send a push notification to the activity’s owner. We decided to set up a dedicated aggregated feed with real-time updates enabled to collect new reactions. 

Creating a new feed group is a breeze. You just hit the add button in the dashboard, type in a name for a new group, select its type, and you’re done. As mentioned, setting up an aggregated feed also requires specifying the aggregation format. Although it looks easy enough to do at first glance, the lack of documentation made us dubious. We wondered which variables we should use to get what we want, as they are described sparsely.

But the real trouble was setting up real-time updates. We're not using Amazon SQS and didn't want to add another service provider to our stack. We've chosen to work with the webhook, as WebSockets are only available in the javascript front-end client. We proceeded to prepare the data-receiving part and learned there is no support for it in the SDK. They're just sending a POST to the webhook address with data as a JSON. Its format differs from the regular feed activities and is almost undocumented, save a single, short example. Well, it's something, but still, we had a hard time figuring out what we should or shouldn't expect about the payload's structure and how to deserialize it safely.

Hacking it together

When we finally sorted out all the technical issues, we realized that the core of the feature in preparation wouldn't be easily done either. We may request a number of reactions of each kind to be included in the response, but we need to know how many individual people left a comment under the given activity. There is no option to query for that. We had to read all reactions, which are paginated with a relatively small maximum page size, and count unique reacting users on our side. Is it a terrible solution? Maybe, maybe not. We didn't like it, as it seemed hacky and unnecessarily resource-hungry to us.

An alternative would be to route adding reactions through our backend and store reactions-related activities' metadata on our side. We could actually implement the notification feature entirely without calling Stream's API this way, but we wanted to avoid bearing the cost of remodeling our integration then. Another argument for the chosen solution was that we could also use the real-time-enabled feed to implement in-app notifications, but this stayed in the backlog in the end.

Getting your hands dirty

Enough about "the what". Let’s talk about "the how"! Stream provides a few official SDKs for easy integration with their API, including a .NET one written in C#. Since this is our go-to backend technology, we gladly used it. We wanted to keep code examples sparse in this article because SDK’s basic usage is accessibly presented in Stream’s documentation, but those few snippets present are written in C#. Please bear in mind that this won’t be a tutorial; we just want to show what communication with Stream looks like.

Creating the client

First things first, to access Stream, we need to instantiate the client. Server-side authentication uses an API key and secret, as it’s typically done. However, front-end SDKs also allow you to authenticate using a token issued for a particular user by your server. Unfortunately, given that we’re just starting, we now arrive at the first awkward thing that we didn’t like in their .NET client: the SDK is located just in the Stream namespace, which is quite intrusive, taking into account how commonly is the term “stream” used in the industry. We’re not the first ones unhappy about this, as their GitHub page shows: #29, #45, and #108.

using Stream;
using Stream.Models;
var client = new StreamClient("YOUR_API_KEY", "API_KEY_SECRET");

var gandalfId = "ed128745-2309-4bf6-..."// normally stored in your DB ofc
var timelineGandalf = client.Feed("timeline", gandalfId);
var shareArticle = new Activity(
    client.Users.Ref(gandalfId),
    "share",
    client.Collections.Ref("article", "cs-union-types"));

await timelineGandalf.AddActivityAsync(shareArticle);

Reading activities

As you can see, adding a new activity is pretty straightforward, but let’s walk through it anyway. Firstly, we instantiate Gandalf’s timeline feed using a dedicated factory method on the client, along with the activity we will add to it. Activity’s constructor expects an actor, verb, and object, the three mandatory values mentioned earlier. Both actor and object are references to concrete entities. We used dedicated Ref() methods to construct those so Stream could understand them, but they’re also just some strings. This is another thing we didn’t like about the SDK – we missed some more type safety. After ranting about this, we finally published the activity to the feed.

Once you have a feed object, reading activities from it is even simpler, we just do: 

var response = await timelineGandalf.GetFlatActivitiesAsync() 

and get the most recent activities from this feed. You can also provide some options to the method call to specify how many items you want to fetch, specify offset, or filter. You may wonder: why does it say “Flat” in this method’s name…? Flat feed is one of three feed types available in Stream; we will discuss them later in more detail. Now let’s see what we’re getting back:

new GenericGetResponse<Activity>()
{
    List<Activity> Response = new()
    {
        new()
        {
            string Actor = "SU:ed128745-2309-4bf6-...",
            string Verb = "share",
            string Object = "SO:article:cs-union-types",
            string ForeignId = null,
            DateTime? Time = null,
        },
    }
}

There are two main problems with this. First, we only get the references back and still need to get user and object data somehow. Second, they didn't update their code to use Nullable Reference Types. As much as we hate the latter, we can't do anything about it besides occasionally shooting ourselves in the foot, so let's focus on the former. Of course, we could request each of those missing entities using a relevant endpoint on the Stream's, API, namely via client.Users.GetAsync() and client.Collections.GetAsync() and then merge the results. 

However, this is such a common process that Stream provides it out of the box, branded as Enrichment. To request it, you just need to use the GetEnrichedFlatActivitiesAsync() method instead of the one we used previously, and both actor and object references will be replaced with the relevant objects.

Enrichment

This works magically with a single API call if you already have all the data stored in Stream’s Users and Collections, of course. You can easily upload a single new element to Stream’s servers in a way very similar to adding new activities, except you’d pack all the data you wish to store in a Dictionary<string, object>.

Below is how we’d upload Gandalf’s article to Stram’s Collections in a day-to-day scenario.

However, if you’d like to start using Stream in an existing project, you probably would need to upload more data, and doing it item by item would be impractical, at least. The cause is not lost, as there are also batch methods and data imports available that allow uploading larger amounts of data at once.

var data = new Dictionary<string, object>()
{
      { "link", new Uri("r.mtdv.me/cs-union-types") },
      { "title", "The most awaited..." },
      { "desc", "Union types finally in C#" },
};
await client.Collections.AddAsync("article", data, "cs-union-types");

Reactions

Finally, there are reactions, which are a way for users to interact with activities on a feed. They are tied to their parent activity: they cannot be created without one and are usually retrieved along with it or using its ID. Adding a new reaction is as easy as calling a client.Reactions.AddAsync() with its kind, parent activity ID, reacting user’s reference, and optionally custom data as parameters. Reactions can also be added to other reactions using  client.Reactions.AddChildAsync() method, but nesting is limited to 3 levels.

Reading reactions is more involved, as it may be done in a few different ways and with a number of options. We’ll cover the basics here: how to read an activity along with its reactions as a part of the Enrichment process. Having an instance of the feed we’d like to read activities from, we can pass options to the feed.Get[...]Async method that, among others, controls if and which reactions should be attached to retrieved activities.

This way, we can request to include the most recent reactions, reactions added by a particular user, reactions of a particular kind, a total count of reactions of each kind, and so on. In reality, only relying on Enrichment for reading reactions is not enough, as it will only allow you to get a limited number of items. However, methods provided by the client.Reactions will also let you get further reactions and filter them. 

var activitiesWithReactions = await timelineGandalf
    .GetEnrichedFlatActivitiesAsync(
        GetOptions.Default.WithReaction(
            ReactionOption.With().Recent().Counts()));

How much Stream costs?

Stream provides their service with typical subscription-based pricing with three subscription tiers plus a custom plan, as seen in the screenshot below. The main factor in pricing is the number of feed updates available within the plan per month, but other quantities, like a monthly number of reactions or API calls, are capped differently within each available option.

Our experience summed up

We're sure that using Stream allowed us to deliver social media-related features much faster than if we'd decided to implement social feeds ourselves. Of course, this is not a revelation because why reinvent the wheel? As proponents of Domain Driven Design principles, we could say that the social aspect of the app under development was not the Core of our client's Domain but rather a Generic Domain, so buying it was a viable thing to do. 

It saved us from building and maintaining the necessary infrastructure, and we see a tremendous value in this. However, applying the bought solution was not a bliss. Some lessons were learned, and the most universal one would be that integrating a service the way the provider envisioned it is easy, but adapting it to the client's needs may be painful.

Even though we were not fully happy while working with Stream, it's generally a decent service. It seems a good pick for small or medium-sized businesses that focus on something other than being a social media platform but want to add such a feature to their product. Stream provides almost everything such a product may need, and its basic integration is very easy. Features that are not available, which we missed the most, are moderation and search tools. The rant on the former may be shortly outdated, as Stream recently announced an Auto-Moderation service in early access.

Rant on the Development Kits

The officially provided .NET SDK is okay-ish but has some annoying shortcomings. What bothered us the most was that it doesn’t use Nullable Reference Types, and we got an unexpected null value a few times here and there. It generally has a relatively poor type safety, as available models are quite basic.

Almost all kinds of IDs and references used in the SDK are just strings, so it’s easy to mismatch them. Custom data added to activities and reactions is only available as  Dictionary<string, object>, where we’d rather have generic types and built-in deserialization. There are also some minor inconsistencies, like no standard way of instantiating options objects. ne has a With() factory method, another Default property, etc.

We used the official Flutter SDK on the client side. Our mobile team had some more serious problems with it than the inconveniences we had on the backend side. They found some bugs that made a certain feature virtually unusable. Of course, we reached out to Stream only to learn that they suggest using server-first integration instead, and they don't intend to fix the reported bugs. This wasn't very clear to us, as all of their documentation praised client-side integration with multiple examples focused on this matter. This was the most frustrating moment of our work with Stream, as we were primarily interested in mobile-first integration, and everything suggested that that's a well-supported approach. Currently, the said SDK is at least marked as no longer actively maintained; a note about it was added half a year after we got the disappointing news.

Documentation and Support

Despite the unsatisfying answer that we got then, we must admit that Stream’s support generally made a good impression on us. They were responding quickly to our requests, their responses were not generic, and they seemed to really try to help with the problems we encountered. For clients that buy one of the higher subscription plans, they also offer consulting services regarding the architecture of feeds in your app, setup of the custom ranking, or configuration of their feeds' personalization feature.

However, the first source of information about the service is usually its documentation, so it’s worth reviewing it as well. It is freely available on Stream’s website, and we find it quite nice and informative,  save the missing notice about the preferred integration style. There’s a short, interactive tutorial that walks you through the very basics, and the docs themselves are more than enough to guide you through the key features and their simple usage. Docs on more advanced topics are a bit lacking, however.

Final thoughts

Why did we choose to use Stream in particular? To be honest, we didn’t really research other options before we started the development. Our client came across Stream and showed it to us. We thought that it was a cool service and decided to use it. That’s it, really. Would we use Stream again? Knowing about client-side SDK deprecation, not necessarily, as we wanted to go with mobile-first integration. But setting aside this preference, we think that it was a good choice. Would we recommend using Stream? Yes, it’s a decent service with good support. Just don’t work against it.

If you have any questions regarding this solution in the context of developing your business mobile or web app project, you can order a free 30-minute consultation call with our experts.