Setup WebSockets on a Django project

We'll be using Django Channels and Django Signals to listen to changes in a model reactively and then send some data to a client using WebSockets.

This post will help you set up Websockets on a Django project and use it on a frontend client (we’re using Vue, but you can use plain JS). We’ll talk about both the local setup and the production setup.

For a quick refresher on what Websockets really is, this video is highly recommended.

The problem WebSockets helped up solve.

I’ll quickly lay out the problem that we were trying to solve and how WebSockets was the tool we used to solve the problem.

Use case 1—

While starting out, our platform was in closed alpha and the users were waitlisted by default. Upon reviewing and talking to them, their status was changed from "waitlist" to "approved" upon which they get access to the platform.

What we wanted to implement was, if a user was logged in to the platform and an administrator changed the status of a user from "waitlist" to "approved" (or the other way round), it would get reflected on the user’s browser and the user should be able to use the platform.

This required communication to begin from the server’s end. Traditional REST API and HTTP requests are initiated from the client’s end and the server responds. WebSockets allow the server to communicate with the client on its own accord and send and receive data asynchronously.

Use case 2—

By default, every user is assigned a personal workspace on our platform. Apart from that, organizations can request a team workspace where all the members of a team can work on assets created by other team members. Users who are a part of the team workspace see a workspace switcher dropdown where they can select and switch to a specific workspace (much like notion does!)

What we wanted to implement here was, if a user is logged in to the platform and an administrator added/removed a user from an organization’s workspace, it should reflect on the workspace switcher dropdown instantly.

This required the backend to listen to this change, and send a message to the client, updating the list of workspaces that the user is a part of.

The objective of this article

Disclaimer — We will not attempt to explain what WebSockets are and we assume you've watched the video linked above or done your own research! This is a long technical article. You might need to go through it more than once, so take your time!

We’ll walk through how to set up WebSockets on a Django — Vue application to solve for a very specific use case. One can follow this example and apply it anywhere else. This is what we will do —

1. Connect the client to the server using a WebSocket connection so the information can flow in both directions.

2. Assuming we're working with a User model in our Django application, we'll make Django detect a change in that model's attributes (which can be triggered by an API call).

3. Have Django send the User's updated attributes to the client using the WebSocket connection made in the first step.

4. The client reactively listens to the change and updates the user interface.

Setup Django Channels

Installation and boilerplate setup

Django Channels is the package we’re going to use to integrate WebSockets into our Django web server. This tutorial is written for Channels 3.0, which supports Python 3.6+ and Django 2.2+.

Install the package using pip (or add it to your requirements.txt, depending on your use case)

python -m pip install -U channels

Then add it to your app settings.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    ...
    'channels',
)

Before moving forward, you have to understand (on a higher level) the difference between WSGI and ASGI.

What is WSGI and ASGI?

These are middle layers that handle requests/responses between the client and a python webserver. WSGI stands for Web Server Gateway Interface and ASGI stands for Asynchronous Server Gateway Interface.

WSGI has workers which can only handle requests sequentially, one at a time. ASGI is a superior successor to WSGI because it can handle requests/responses asynchronously. A simple and short explainer can be seen here and here.

To use WebSockets, we need a middle layer that can handle requests in an asynchronous way, which ASGI helps us to do.

Make your application ASGI compatible

Back to the setup, after adding the package to your settings.py, you have to configure the ASGI layer. By default, a Django app uses a WSGI setup. You can confirm this by going to your settings.py file and searching for a line that might look like this —

WSGI_APPLICATION = "myproject.wsgi.application"

Create a new setting (or edit the existing one if it’s already defined in the settings.py) in your settings.py file which will tell the channels package where to find the root object to configure the ASGI application.

ASGI_APPLICATION = "myproject.asgi.application"

Now channels know where to find the root object to initialize the application. But we haven’t written any logic there yet. So, go ahead and create a file asgi.py in your app’s root folder. Eg. if your app is named myproject, the file will live at myproject/asgi.py

This is what your asgi.py file might look like. (you can copy the code from here)

Setting up a Protocol router and Auth middleware

In the code above, we defined handlers for different protocols inside the ProtocolTypeRouter object. It routes the incoming requests to different handlers depending on the type of request.

To handle HTTP/HTTPS requests, Django provides a handler method called get_asgi_application() which basically says “let the HTTP/HTTPS requests be handled by Django views like they were handled before”.

The roles of ProtocolTypeRouter, AuthMiddlewareStack, and URLRouter as defined by Channels docs —

This root routing configuration specifies that when a connection is made to the Channels development server, the ProtocolTypeRouter will first inspect the type of connection. If it is a WebSocket connection (ws:// or wss://), the connection will be given to the AuthMiddlewareStack.

The AuthMiddlewareStack will populate the connection’s scope with a reference to the currently authenticated user, similar to how Django’s AuthenticationMiddleware populates the request object of a view function with the currently authenticated user. Then the connection will be given to the URLRouter.

The URLRouter will examine the HTTP path of the connection to route it to a particular consumer, based on the provided URL patterns.

In the code above, you can see that the URL pattern path that is passed to the URLRouter is myproject.urls.websocket_urlpatterns. We will define these URL patterns in a while.

Setting up this asgi.py file gives channels a pointer to the root application, which can then integrate itself in Django and take control of the runserver command.

From the docs —

When Django accepts an HTTP request, it consults the root URLconf to lookup a view function and then calls the view function to handle the request. Similarly, when Channels accepts a WebSocket connection, it consults the root routing configuration to lookup a consumer and then calls various functions on the consumer to handle events from the connection.

So we need to do two things.

1. We need to define some URL patterns for our WebSocket connections. Channels will consult these URL patterns and then decide what to do with the request.

2. We need to write the logic for what needs to be done with the request after URL patterns are resolved. For normal HTTP requests, the logic is usually written in views. For WebSocket requests, the logic is written in Consumers.

Setting up URL Patterns

Now it’s time to define a set of URLs that we want Django to match, and call a particular consumer to handle the connection. In the code snippet above, you can see that the websocket protocol is responsible for routing those requests to a URLRouter named myproject.urls.websocket_urlpatterns. Go ahead and create a variable named websocket_urlpatterns inside myproject/urls.py.

Now, what should we put inside that URL pattern list? All the URL paths using which the client will try to make WebSocket connections.

For our specific use case, we want a WebSocket connection for each unique user. This will allow us to transfer data from the server to a particular client’s browser, and similarly, transfer data to multiple clients at the same time.

For our use case, the URL patterns list will look something like this. (copy the code from here)

As you can see in the above code snippet, the urls.py file can contain both standard URL patterns (which your WSGI app was using), and also the WebSocket-specific URL patterns.

Our API’s root URL is api/v1/ so I’m just using that and adding /users/USER_ID will let us access the endpoint for a particular user.

Setting up Consumers

Now comes the (somewhat) confusing part, Consumers (which you might’ve noticed in the snippet above). This is a snippet from the official documentation -

Channels provide you with Consumers, a rich abstraction that allows you to make ASGI applications easily.

Consumers do a couple of things in particular —

• Structures your code as a series of functions to be called whenever an event happens, rather than making you write an event loop.

• Allow you to write synchronous or async code and deal with handoffs and threading for you.

If you’re familiar with Django class-based views (which you should be if you’ve stuck around here till now!), you can assume consumers to be like class-based views in a sense. We’ll write our core logic inside a Consumer class. That’ll include things like accepting/rejecting connections, sending/receiving data.

To explain on a very high level, we’ll write a UserConsumer class which will extend the synchronous WebsocketConsumer class. This class will be responsible for accepting/rejecting incoming connections. This class will also handle receiving and sending a user’s data to that user’s client machine through the established WebSocket connection.

Add a file named consumers.py in your project’s app.

Generally, four methods are defined in a consumer class —

• connect -

  • Accept the incoming connection. This is done using self.accept().

  • It is recommended that accept() be called as the last action in the connect() method if you choose to accept the connection.

  • We might want to reject a connection if a user is, let’s say, not authorized.

• disconnect - Close the ongoing connection. Done using self.close()

• send - Send message(s) back to the client.

• receive - Logic to handle incoming messages from the client.

Given the information above, now let’s write our extended consumer class. (copy the below code from here)

What all is going on in the above class?

• If an incoming connection is noticed, the connect method is called, which accepts the connection. We can disconnect the connection by calling disconnect on the UserConsumer instance

• In our given use case, as we’re not receiving any data from the client, our receive method doesn’t contain any logic. We’ll only be sending data to the client.

• In our send method, we’re just sending some test data back to the client. You’ll notice that the name of our send method is send_user. You can name it however you like. You’ll see later on that we can specify which send method we want to invoke when.

Let’s visit our use case again. We want the client to know automatically whenever a user’s status has been updated.

Listen to changes in a Django Model

To detect when a user’s status is updated, we need to detect changes to our Django models. This can be done using Django Signals.

From the official docs —

Django includes a “signal dispatcher” which helps decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. They’re especially useful when many pieces of code may be interested in the same events.

So in a nutshell, this is the whole process —

Whenever someone updates a user’s status through an API call, a change to the respective Django model is made. This change is picked up by Django Signals, and depending on the logic configured, a call to UserConsumer is made, which sends the updated user’s data to the client through a WebSocket.

Add a signal to a model

Now let’s write the code which will listen to the changes on a Django model and run some logic if a change is detected.

The below code will be triggered whenever any change to a User’s model is made.

We add a receiver to the User model. The type of receiver is pre_save, i.e., it’ll be triggered before a change to the model is saved. There are many different types of receivers. More on them here. Where to keep these signals files? More on that here.

Now for the next step, we need to trigger our UserConsumer's send method which will actually send the new updated User value to the client. We need to trigger this from our Signals receiver that is shown in the image above.

Your instinct would tell you that we might need to do something like

userConsumer = new UserConsumer()
userConsumer.send_user(user_data)

But unfortunately, this will not work. A consumer instance cannot talk to other parts of Django directly. To use a consumer’s code outside of the class, we need something called Channel Layers.

Help Consumers communicate with Django Signals

The following explanation from the official docs explains things really well —

A channel layer is a kind of communication system. It allows multiple consumer instances to talk with each other, and with other parts of Django.

A channel layer provides the following abstractions —

• A channel is a mailbox where messages can be sent to. Each channel has a name. Anyone who has the name of a channel can send a message to the channel.

• A group is a group of related channels. A group has a name. Anyone who has the name of a group can add/remove a channel to the group by name and send a message to all channels in the group. It is not possible to enumerate what channels are in a particular group.

Every consumer instance has an automatically generated unique channel name, and so can be communicated with via a channel layer.

In the graphic above, we can see that if a piece of code has the name of a channel, it can send messages to that particular channel.

If a piece of code has the name of a group, it can send messages to all the channels at once. It can also remove or add channels to that group.

Now, for our specific use case, we’ll have to use a workaround.

Setup channel layers

We need some way of sending a message to a consumer from the Signals receiver code. As described above, the way Django Channels work, we can only communicate with a consumer from the other parts of Django using a channel layer. So we need a channel layer. Let’s first configure that.

To use the channel layers, Django channels need to store the information somewhere. To avoid shuttling everything through a database, it’s recommended to use a NoSQL store like Redis. You can also use your in-memory cache for development purposes.

Let’s use a Redis store as it’ll be easy to replicate the process in a production setup. You can either spin up a separate docker container that is holding a Redis store or add it as a build step in your existing docker-compose file.

To start a fresh container running Redis, use this snippet (copy the code here).

To use the Redis store as a part of your existing docker-compose, you can use this snippet (copy the code here).

We also need to install channels_redis so that Channels knows how to interface with Redis. Run the following command —

$ python3 -m pip install channels_redis

After getting the Redis service up and installing channels_redis, we need to tell Django that it should use Redis as a backing store for the channel layers feature and also tell it how and where to connect to the service.

Add the below code to your settings.py. You can specify REDIS_HOSTNAME and REDIS_PORT in your .env file. Better to always avoid hardcoding! You can also use a cloud Redis service (like Elasticache by AWS) and configure its endpoints as the hostname and port. (Copy the code from here)

Let’s make sure that the channel layer can communicate with Redis. Open a Django shell and run the following commands — (copy the code from here)

Now that the channel layers have been set up, we’ll set up how we can communicate with the UserConsumer class from the signals receiver.

Connect consumers with channel layers

Whenever a UserConsumer is instantiated, a channel is automatically created with a unique name. Now to be able to communicate with that channel and send messages from outside the class, we need to wrap it into a group. Once we have a group for each instance of the UserConsumer channel, we can send messages to that group from outside the class.

On a high level, this is how the architecture might look like.

Let’s modify the code in our consumers.py such that whenever a connection request is received, we create a new group with a unique name. We then add the newly created channel to this group. (copy the code from here)

Let’s go over the code above in some detail —

• Whenever a connection request is received, we also get the id of the user who is requesting to establish a connection. We can use this id to create a unique group.

• Whenever an instance of UserConsumer is created, a unique channel is created. We’ll then add this channel to a group. We can access the name of the current channel using self.channel_name. The adding to the group is done using channel_layer.group_add coroutine. From the docs —

• What about the method async_to_sync? This is needed because our UserConsumer class is a synchronous class and the group_add method is an asynchronous method. In fact, all channel layer methods are asynchronous in nature.

• Once we add the current channel to a group, we accept the connection. In the disconnect method, we’ll use the group_discard coroutine provided by channel_layer which will remove the current channel from the group.

• In the send_user method, we’re receiving a User instance which we parse as a JSON and send that data to the client over the WebSocket.

Now that we have edited our UserConsumer class to handle adding a channel to a group, we’ll access this group in the Django Signals receiver and send a message when needed.

Connect Django signals with the channel layer

Edit your @reciever code (wherever you’ve written it, either in the views.py or a separate signals.py file) with the code below (copy the code from here)

Going through the above code, if a user’s status has been updated, we’ll pass the instance through a UserSerializer. We can get the channel layer object using get_channel_layer(). After that, we’ll construct the group name that we want to connect to. To send a message to that group, we’ll use channel_layer.group_send. It takes two arguments. The first is the group name that we want to connect to. The second argument is an object which takes what data we need to send and the method that we want to invoke (which in our case is send_user).

Client-side implementation

Now to implement WebSockets on the client, and to request a ws:// or wss:// connection to the server, there are tons of methods you can use. Browsers natively support the WebSocket API so you can use that itself. A lot of libraries are also available (eg. socket.io). I’ll show how we used the plain API provided by the browsers.

We needed to integrate our WebSocket code with our Vuex store. Why? Because we want to listen to changes to the store. If a user logs out or another user logs in, we update our store, and accordingly, we need to close the existing WebSocket connection/create a new one.

Before that, add an environment variable that will store the root URL of the WebSocket connection.

VUE_APP_BACKEND_WEBSOCKET = ws://0.0.0.0:8001/api/v1

The way we integrated it with our store is to make a Vuex plugin. Create a file usersWebSocketPlugin.js under your src/store/plugins. (copy the code from here)

Let’s go over the above plugin’s code in some detail

• We export a module called as createUserWebSocket. This will be imported into Vuex’s index file. We’ll get to that soon.

• The exported module contains a function connect(). This function takes a reference to the store itself.

• The connect() function instantiates an object from the WebSocket class and attempts to make a connection.

• Different event listeners can be added to that object. More details here.

Now let’s add this plugin to our Vuex store. We are using some modules as well. I’ve added that in the screenshot for a comparison of what should go where.

Summary

So there you have it! I know that was a lot. So let’s summarise whatever we discussed.

• Websockets allow us to have free-flowing conversations between the server and the client(s). The client does not need to request a resource. The server can send it on its own.

• Websockets is implemented in Django using a package called Django Channels.

• Django Channels has a boilerplate setup that is explained well in their Tutorial section.

• We turn our app into an ASGI app (basically an asynchronous app), define the URL routes which will invoke our WebSocket code, and create Consumers to handle the respective connections.

• We detect changes in our Django models using Django Signals. If we detect some change that we need to propagate to the client, we use the channel layers to send that information to a group.

• A channel layer is like a communication system. It allows different consumers to talk to each other and also allows the rest of Django to communicate with those consumers. Channel layers provide two features, groups, and channels.

• A group is a collection of channels. A channel is like a mailbox, it contains messages that it receives and can send it out. Every time Django sees a URL with ws:// or wss:// connection type, it spins up an instance of a consumer class that you must’ve configured.

• With every instance, a unique channel is created for you. You can add that channel to a group or remove that channel from a group.

• On the client end, we can use the native WebSocket API offered by the browsers or can use any library.

• On the client’s end, we’ll make a connection request to the server. If the server accepts the connection, the server will be able to send across the user’s updated data to the client. The client will then set it in its store accordingly.

Production specific details

Things to note about setting this up on production

• Don’t use python’s default HTTP server in a production setting. You should ideally use something like NGINX on a deployment version of this. This is because there will be hundreds or thousands of concurrent ongoing connections. It depends on the usage of your backend honestly.

• Every time a WebSocket connection is established, a new file is opened on the server. Sometimes, if the number of concurrent WebSocket connections increases a lot, you might run into errors and the connections would start dropping. This happens because of ulimit. This is a limit put by operating systems on the number of concurrent files you can open at once. To resolve this, you'll have to check and increase the ulimit. More on this here.

• For detailed steps of how to deploy this, you can check out Plio’s backend documentation here.