Ranch: what's under the hood?

A deep dive into the home of cowboys.


Introduction

Hi there! I am Nicolas - developer.

This document is a summary of my analysis of Ranch 2.0.0-rc.1 source code, performed in August 2019. It has been done purely in an old-school way, cloning the project from github and going through it manually.

By helping understand the intricacy of a well-established but somewhat small Erlang library, the reader can:

  • improve its Erlang proficiency

  • be able to help maintaining Ranch

  • be able to reuse Ranch codebase or get inspired by it for other projects

This analysis has been performed while I just started learning Erlang, right after going through Learn You Some Erlang For Great Good. If you find any errors, a pull-request or a new GitHub issue is welcome! ;)

1. License and disclaimer

This work is distributed under the same license as Ranch itself, ISC License.

Ranch was originally built by Loïc Hoguin, which kindly encouraged me to publish this code analysis.
I am not related to Nine Nines.
Donate if you wish to support Ranch.

2. TL;DR

Please, refer to the Conclusion.

3. Prerequisites

Having had basic experience in any development project with any programming language is highly recommended to grasp the following content.

It is assumed that the reader have basic knowledge of Erlang/OTP as well as hands-on experience in managing TCP/IP sockets - or at least enough theoretical knowledge to understand their implementation (in any language).

4. What is Ranch about?

Ranch is a "Socket acceptor pool for TCP protocols". It is at the foundation of the Erlang Web Server Cowboy, that is itself at the foundation of the Elixir Web Framework Phoenix. Both Web Servers are widely used in the Erlang/Elixir community.

There is an excellent manual maintained by the developers of the project.

I advice you to read this official documentation before continuing:

However, it will not harm to summarize here the purpose of this library and describe the main concepts: Listener, Transport, Protocol, Acceptor, Pool, Connection…​etc. I found it hard personally to understand how it all works together, so I’ll explain it with my words here.

Understanding these keywords and how they shape together is primordial to fathom the source code. Why? Because you’ll see a lot of those here and there in the code. And it will be way more fuzzy in the codebase that it is in the documentation. The developers of the codebase expect you to know about the terms already.

That’s why, you would rather learn them before. When you see a variable named "Protocol", you’ll then directly know what is its purpose, without having to finish the code analysis to do so.

(Actually, that’s a general truth in software development. You cannot write/read code efficiently if you don’t master what it is about on the first place.)

4.1. Description

I will simply quote the official documentation:

Ranch is a socket acceptor pool for TCP protocols.

Ranch manages listeners which are a set of processes that accept and manage connections. The connection’s transport and protocol modules are configured per listener. Listeners can be inspected and reconfigured without interruptions in service.

It is worth emphasizing that Ranch manage a pool of listeners.

4.2. Definitions

4.2.1. Generic terms

Let’s remind us about some fundamentals here.

Process

It’s the Erlang OTP process.

Quoting the official Erlang documentation:

Erlang is designed for massive concurrency. Erlang processes are lightweight (grow and shrink dynamically) with small memory footprint, fast to create and terminate, and the scheduling overhead is low.

Pool

A pool, is a general programming concept frequently used. Quoting Wikipedia:

A pool is a collection of resources that are kept ready to use, rather than acquired on use and released afterwards.

In the context of Ranch, and even in general in Erlang/OTP, we will mainly talk about pools of processes.

4.2.2. Ranch-specific concepts

Even though the following abstractions are seen in most networking librairies, their precise meaning in the context of their project differs from one to another.

If you weren’t already familiar with these notions, understanding what they are about in the context of Ranch will definitely help you identifying them in other libraries, or re-using them in new projects.

Listener (process)

Quoting the Ranch documentation:

A listener is a set of processes that accept and manage connections.

A listener is a set of processes whose role is to listen on a port for new connections. It manages a pool of acceptor processes, each of them indefinitely accepting connections. When it does, it starts a new process executing the protocol handler code. All the socket programming is abstracted through the use of transport handlers.

It is worth emphasizing that each listener manage a pool of acceptor processes.

Acceptor (process)
According to the quote above, we can say that
  • the role of an acceptor is to indefinitely accept connections

  • for each new connection being accepted, it starts and forwards the logic to a new process, named "connection process"

It is interesting to notice that while it’s the acceptor that starts the connection processes, it is the supervisor of the acceptor process, the listener, that supervises them:

The listener takes care of supervising all the acceptor and connection processes, allowing developers to focus on building their application.

Connection (process)

Quoting from the listener definition above:

[The connection process] executes the protocol handler code. All the socket programming is abstracted through the use of transport handlers.

As seen above, it is started at each accepted connection by the acceptor process, and is supervised by the listener process.

Its role is to actually implement the communication logic (what to do when receiving that message, or sending that other one, etc..).

Protocol (handler)

Quoting the Ranch documentation:

A protocol handler starts a connection process and defines the protocol logic executed in this process.

Actually this might be a little confusing at this point. Isn’t it the acceptor process that’s supposed to start the connection process???

I found it hard to understand the distinction between a "connection process" and a "Protocol handler" in the documentation. Well, to better comprehend this, I’ve had to cheat and already dive a little bit into the code. I will give here the result of my research, to help you out!

Basically, when they say in the quote above that "a protocol handler STARTS a connection process", you must somewhat understand "a protocol handler IS a connection process" instead. While it is not exactly true (the original sentence is correct), thinking this way personally helped me understand things better on the first place.

The distinction is subtle. What happens, in fact, is that everytime a connection is accepted, the acceptor process will use a method from its corresponding "connection supervisor" to "start the protocol". This function will start the Protocol initially passed as parameter, by the end user of the library. This Protocol (which is NOT a gen_server), when started, will in turn spawn a linked process (like a gen_server does). This process is what’s called the "connection process". This process shall embed all the protocol logic implemented by the user (like sending and receiving data). There is no requirement on what it does, it is left entirely to the user.

It means that a Protocol is a sort of interface that the user has to implement and pass as a parameter when starting the pool of listeners.

We can already expect such "interface" to be implemented as a custom behaviour. It is the idiomatic way of doing so in Erlang.

[question] Hey! Can you tell us more about how you found that out in the code? I am curious! [question]

Okay…​ If you insist :).

See ranch_acceptor.erl line 41: ranch_conns_sup:start_protocol/3 is being used in its ranch_acceptor:loop/5 main method on success of Transport:accept/2.

ranch_conns_sup:start_protocol/3 is sending to himself a "start_protocol" type of message using the bang operator (!). Then, in its main ranch_conns_sup:loop/4 method, this message is pattern matched in a receive block.

On reception of the message, it is running Protocol:start_link(Ref, Transport, Opts). (See ranch_conns_sup.erl line 122.) Protocol, being a parameter that was being pushed through from the beginning by the end user of the library using ranch:start_listener/5.

A quick glance to ranch_protocol.erl and in the examples of the official documentation confirms that Protocol is the one process that’s supposed to actually implement the user’s high-level message sending and receiving logic.

And Protocol:start_link/3, in the examples, like for gen_server, is spawning a new linked OTP process.

Okay, let’s wrap it up!

The distinction between a Protocol handler and a connection process is therefore that a Protocol handler is the interface, and ultimately the parameter (a simple variable) being pushed through by the user. Being a parameter is the key that explain why it is called a "handler". In other words, it is a delegate (generic programming notion).

On the other side, the connection process is the actual OTP process being spawned by the Protocol when the Protocol is started. That’s what they meant in the official documentation by saying that "the Protocol handler starts a connection process".

Transport (handler)

Quoting the official documentation:

A transport defines the interface to interact with a socket.

Transports can be used for connecting, listening and accepting connections, but also for receiving and sending data. Both passive and active mode are supported, although all sockets are initialized as passive.

In short, everywhere in the code when an actual action is needed towards the sockets (accepting the tcp/ip connection, sending, receiving messages…​etc), such action is ultimately delegated to the Transport interface. When you read interface, like for Protocol, so you probably guess now that it is implemented as a custom behaviour.

But contrary to Protocol, the user doesn’t have to implement this behaviour in every situation.

The library already only exposes two sorts of transports that have implemented the custom Transport interface:
  • the TCP Transport (ranch_tcp)

  • the SSL Transport (ranch_ssl).

Each of them are different modules that implemented the same behaviour. They are thin wrappers on top of the gen_tcp and ssl modules, which are the default erlang modules to do TCP/IP Socket programming and SSL encryption.

However, users can write, if they wants, a custom transport handler. For that, they need to implement the ranch_transport behaviour and follow the the guide in the documentation.

The internal documentation

Ranch does not publish all its documentation to the internet. The file ranch/doc/src/guide/internals.asciidoc contains useful information for us!

I’ll just rewrite part of it with my own words here and double check them in the code. Feel free to skip that part if you prefer just reading it instead (it is quite short).

1. The Ranch Server

  • When Ranch library starts, it starts the ranch_server module

  • Ranch has a top supervisor that is supervising ranch_server and all the listeners started

"Ranch library starts" == Ranch is loaded as a dependency module by the end-user application

2. The ETS Table

There is an ETS table named ranch_server started on application startup by the module carrying the same name, and shared by all the application

It is own by the top application supervisor to avoid loss of information in case of restart.

If you guys don’t remember what’s an ETS table, check LYSEFGG or the official documentation.

3. On managing connections

There is a custom supervisor for managing connections. A glimpse at the code shows that it is ranch_conns_sup_sup.

It keeps track of the number of connections so it can handle connection limits.

4. About listeners

There are a few relevant information concerning listeners:
  • listeners have their own supervisor, ranch_listener_sup

  • ranch_listener_sup starts three kind of processes:

    • the listener gen_server (cannot see track of it in the code???)

    • the acceptor process supervisor ranch_acceptors_sup, which in turn start the acceptor process

    • the connection process supervisor supervisor ranch_conns_sup_sup, which in turn starts the connection supervisor ranch_conns_sup, which in turn starts the connection process through the Protocol.

  • ranch_server monitors some of these processes with various amount of information

That part is quite cool and would have saved us from a lot of troubles earlier ;) :

Protocol handlers start a new process, which receives socket ownership, with no requirements on how the code should be written inside that new process.

Take note of the "socket ownership" part, it’s rather important and interesting. If you have no clue what it means, read the end of this chapter of LYSEFGG. It seems logical that it is the process that’s ultimately taking care of the connection that become the owner of the socket.

The info on Transport is nothing new for us at this point.

The Examples

To understand the code, better starting off knowing how to use the library!

We’ll therefore go on and look at the two example project shipped with the source code:
  • tcp_echo

  • tcp_reverse

The official documentation also extensively use them to explain the concepts.

1. TCP Echo Example

Start a TCP acceptor pool that will echo anything sent to its underlying sockets.

1.1. ebin/tcp_echo.app

It is worth noticing here that the ranch dependency is part of the applications field, as expected. A quick check proves that the ranch binaries are available in the deps/ directory.

tcp_echo_app is the entry point of the application, as it is the value of the mod parameter.

1.2. src/tcp_echo_app.erl

That’s the entry point of the application. The module implements the "application" API with the methods start/2 and stop/1 - as expected.

In start/2, ranch:start_listener/5 is called. That’s the core of Ranch. It starts the pool of TCP/IP listener. Take note that the module echo_protocol is being passed as parameter.

The next line tcp_echo_sup:start_link() basically starts a custom supervisor.

Let’s see the supervisor code.

1.3. src/tcp_echo_sup.erl

It’s written as a dynamic supervisor, but…​ No child is ever added using supervisor:start_child/2 in the rest of the files. So as is, this supervisor is useless…​

1.4. src/echo_protocol.erl

This module was passed as parameter in ranch:start_listener/5. It implements the ranch protocol behaviour (-behaviour(ranch_protocol)). The entry point being start_link/3 that basically spawn a linked process that will ultimately run init/3. Contrary to a traditional gen_server that’s not init/1 that’s being called.

init/3 runs ranch:handshake(Ref) which most probably initiates the Socket acceptors…​? Then it runs a recursive loop function, that simply pattern match the data being received. Transport:recv(Socket, 0, 5000) being probably a blocking operation, it just waits until data is received.

Then, it reads the first data and if anything is received, echoes back the same message using Transport:send(Socket, Data) and start the same function loop again. Ultimately, when no data is received, it closes the socket using Transport:close(Socket). Then it goes out of the loop and the process terminates.

1.5. What did we learn?

Now, we know that:
  • ranch:start_listener/5 must be used by the user, and is the starting point of the TCP/IP listener pool

  • as expected, ranch_protocol is a behaviour that people have to implement and then pass to ranch:start_listener/5 as an argument.

  • ranch:handshake/1 must be run when initializing the Protocol, before starting the connection process

2. TCP Reverse example

Start a TCP acceptor pool that will echo the reversed version of anything sent to its underlying sockets.

2.1. ebin/tcp_reverse.app

Nothing much different from before here, just a basic .app file which tells us that tcp_reverse_app is the starting module of the application.

2.2. src/tcp_reverse_app.erl

Starting the ranch_listener, like before. Still using ranch_tcp transport, and this time the custom reverse_protocol.

2.3. src/tcp_reverse_sup.erl

Same as before. Useless supervisor here.

2.4. src/reverse_protocol.erl

This example is useful to show how to integrate a gen_statem custom protocol handler with Ranch. It would also work the same way for a gen_server.

Read LYSEFGG and the official Erlang documentation to know more about Finite State Machines and gen_statem.

3. Summary

We can now already make a schema summarizing how the different notions integrate together, based on the examples and the documentation.

Ranch Summary Architecture

A few things to notice here:
  • the "supervising" and "monitoring" part here are to be understood in a very high-level way. We are not yet talking about how it actually works under the hood in terms of supervision tree between the modules. Still, it helps creating a mental image of the Ranch architecture.

  • the steps related to the Acceptor are also simplified and will be discussed further later

Diving into Ranch Source code

We’ll follow these two distinct paths to study the code:
  • what happens when the library is loaded?

  • what happens when the user call ranch:start_listener/5?

1. What happens when the library is loaded?

1.1. ebin/ranch.app

This is the main file defining the project. Let’s decrypt it together and remind ourselves what is the meaning of the different values.

  • {application, 'ranch'}

    • Thats the name of the OTP application being created. 'ranch' is an atom.

  • {description, "Socket acceptor pool for TCP protocols."}

    • The description of the application. A string.

  • {vsn, "2.0.0-rc.1"}

    • The version of the application. A string.

  • {modules, ['ranch','ranch_acceptor','ranch_acceptors_sup','ranch_app','ranch_conns_sup','ranch_conns_sup_sup','ranch_crc32c','ranch_embedded_sup','ranch_listener_sup','ranch_protocol','ranch_proxy_header','ranch_server','ranch_server_proxy','ranch_ssl','ranch_sup','ranch_tcp','ranch_transport']}

    • "All modules introduced by this application. systools uses this list when generating start scripts and tar files. A module can only be defined in one application."

    • It does correspond to all the .erl files of the src/ folder.

  • {registered, [ranch_sup,ranch_server]}

    • The list of processes that will be accessible by their name instead of only by PID (== registered processes, see official Erlang documentation).

  • {applications, [kernel,stdlib,ssl]}

    • "All applications that must be started before this application is allowed to be started. systools uses this list to generate correct start scripts. Defaults to the empty list, but notice that all applications have dependencies to (at least) Kernel and STDLIB."

      • Here we can note that Ranch needs the ssl external module to be loaded beforehand. Obviously, Ranch does use this security protocol so it makes sense.

  • {mod, {ranch_app, []}}

    • "Specifies the application callback module and a start argument, see application(3). Key mod is necessary for an application implemented as a supervision tree, otherwise the application controller does not know how to start it. mod can be omitted for applications without processes, typically code libraries, for example, STDLIB."

      • That’s interesting! The starting point of the application being always ranch_app. We’ll keep that in mind and start by this file for our source code anaylisis in the next chapter.

  • {env, []}

    • "Configuration parameters used by the application. The value of a configuration parameter is retrieved by calling application:get_env/1,2. The values in the application resource file can be overridden by values in a configuration file (see config(4)) or by command-line flags (see erts:erl(1))."

      • No default configuration parameter being set here.

]}.

1.2. ranch_app.erl

Yep, it has -behaviour(application), which makes sense because it is part of the mod parameter above. The file is quite small and easy to understand.

Two things to notice here:
  • that’s where profiling logic is in

  • ranch_sup:start_link/0 is run on start of the application.

That’s the supervisor for the whole Ranch app. We can now start our supervisor tree:

Ranch Source 1

See LYSEFGG for more info on the Application Master. Let’s dive in ranch_sup to know what’s behind these question marks.

1.3. ranch_sup.erl

As expected, it implements the supervisor behaviour.

The following code:

start_link() → supervisor:start_link({local, ?MODULE}, ?MODULE, []).

is standard. No argument will be passed to init/1.

init/1 with an empty argument list will be run in the spawned supervisor process. We will now study its content.

1.3.1. The restart intensity configuration parameters

application:get_env(ranch_sup_intensity) is used to access the ranch_sup_intensity configuration parameter. Similarly the ranch_sup_period configuration parameter is being accessed.

Then it is returned by the init/1 function as follow: {ok, {#{intensity ⇒ Intensity, period ⇒ Period}, Procs}}.

What is the meaning of these parameters? Quote from the official documentation:
  • "To prevent a supervisor from getting into an infinite loop of child process terminations and restarts, a maximum restart intensity is defined using two integer values specified with keys intensity and period in the above map. Assuming the values MaxR for intensity and MaxT for period, then, if more than MaxR restarts occur within MaxT seconds, the supervisor terminates all child processes and then itself. The termination reason for the supervisor itself in that case will be shutdown. intensity defaults to 1 and period defaults to 5."

  • See official Erlang documentation

Interestingly, these configuration parameters are not documented in the official Ranch Documentation. In migration_from_1.5.asciidoc, I quote:

"It is now possible to configure the restart intensity for ranch_sup using the OTP application environment. This feature will remain undocumented unless there is popular demand for it."

How would you use these parameters? A quick search shows these examples from a unit test in test/ranch_ct_hook.erl:
  • application:set_env(ranch, ranch_sup_intensity, 10)

  • application:set_env(ranch, ranch_sup_period, 1)

1.3.2. The ChildSpec

The last parameter returned by init/1 is the variable Procs that corresponds to what’s called ChildSpec in the official Erlang documentation.

We have the following:
  • Procs = [#{id ⇒ ranch_server, start ⇒ {ranch_server, start_link, []}}

which is pretty standard.

ranch_sup is therefore supervising ranch_server, which entry point is start_link/0.

We can update our supervision tree:

Ranch Source 2

It is also worth noticing that an ETS table is being initialized using ets:new/2. The ETS table is named after ranch_server module so it is expected that this module will be interacting with it.

From now on, we have two paths to study the code

We will start by ranch_server.

1.4. ranch_server.erl

The first thing to notice is that it implements gen_server, as expected by its name.

Which state is that gen_server keeping? As usual, you need to look for it at the beginning of the file, after the list of export:
  • A macro TAB is being defined as ?MODULE but not used throughout the whole file - which is a bit awkard. Sometimes we see the usage of ?MODULE, sometimes ?TAB. I don’t understand the point of this macro. Maybe there’s a good reason for it that I am missing out.

  • The interesting part is the record state being initialized with an empty list. state record contains a value of type monitors() which is itself defined as follows:

start_link/0 without surprises is running gen_server:start_link/4, with standard arguments.

We know that the init/1 function is the starting point of this process. We therefore scroll the file down to see its implementation.

As expected, init/1 returns {ok, State} with State being here of type state, record defined previously (see above).

The list of all connections and listener supervisors is fetched from the ETS table previously created. ranch_server starts monitoring them, and store the returned reference in the state, together with its corresponding Pid, the Reference of the supervisor process itself, and its Id (for connection supervisor).

From that point on, it would be interesting to look for the location where the connection and listener supervisors are added to the ETS table.

To do so, we can search for "conns_sup" or "listener_sup" or "ets:insert" in the project.

The following appears in ranch_server:handle_call/3:
  • State = set_monitored_process({conns_sup, Ref, Id}, Pid, State0) called by ranch_server:set_connections_sup/3, which is itself called by ranch_conns_sup:init/7

  • State = set_monitored_process({listener_sup, Ref}, Pid, State0) called by ranch_server:set_listener_sup/2, which is itself called by ranch_listener_sup:init/1

ranch_server:set_monitored_process/3 is simply making ranch_server monitor the connection/listener supervisor, update the ETS Table accordingly and return the new updated state.

It is called on creation of the supervisors.

There are a few things I don’t understand here:
  • Why is erlang:monitor(process, Pid) being called in init/1? It will stack with another monitoring reference, because the supervisors fetched from the ETS Table are supposed to be already monitored (see ranch_server:set_monitored_process/3)…​ Why isn’t the ETS Table also storing the MonitorRef? It would avoid having to do that…​ Or is that the intended behaviour?

  • In any case, most likely, the ETS Table will be empty at this point, because ranch_server:init/1 has to have finished running before the ranch_server gen_server becomes actually available for users, and before they can eventually call ranch_server:handle_call/3! Therefore, the returned initial state will be empty on start of ranch_server and the code that fetch the ETS Table is irrelevant…​

Oh! I just found out!

What happens when ranch_server dies and is restarted by ranch_sup? In this case, there will still be data in the ETS Table, data that will be monitored by…​ the ranch_server process that just died! So, basically not monitored at all! That’s it! That’s the reason why the use of erlang:monitor(process, Pid) in ranch_server:init/1 does not stack with another monitoring. Its only (and important!) point is in case the process that was previously monitoring died!

A quick search also shows that ranch_server:get_listener_sup/1 is used in:
  • ranch:get_status/1.

  • ranch:suspend_listener/1

  • ranch:resume_listener/1

  • ranch:remove_connection/1

  • ranch:info/0 and ranch:info/1

  • ranch:procs/1

Similarly, ranch_server:get_connections_sup/1 is used in:
  • ranch:get_connections/2

  • ranch:apply_transport_options/2

These functions are the API that’s accessible by the user of the library. They are features. See the official documentation to know more about their usage.

Finally, ranch_server:get_connections_sup/2 is used by ranch_acceptor:start_link/5. If you remember well the previous chapter, the listener was the one that was creating the connection supervisor supervisor and the acceptor supervisor. Which means they were basically creating them. However, it is obviously the acceptor which needs to trigger the connection supervisor to start the connection process. So the acceptor has to know about a process that’s neither his parent nor his child. That’s why he needs to get it from an ETS table, that’s handled by ranch_server!

We can now improve our schema:

Ranch Source 3

So, as expected after the study of the documentation, everytime a connection supervisor or a listener supervisor is started, it is registered within the ranch_server.

Now we are wondering: when are they started?

2. What happens when the user call ranch:start_listener/5?

2.1. ranch.erl

That’s the where all the API functions are stored. The most important one here being ranch:start_listener/5. Let’s dive in its implementation.

Open the source code, and look at it with me!

It starts by "normalizing" the TransportOpts0 parameter using ranch:normalize_opts/1.

It is simply the method that’s responsible for handling the two different data structure containing the transport options you can pass to ranch:start_listener/5:
  • a list of socket options OR

  • a map

The list of socket options being the old way, it is still handled by modern Ranch for backward compatibility purpose.

Interestingly, code:ensure_loaded(Transport) is being used. If you are no idea what module loading, interactive and embedded mode are, then check the official documentation. The commit message from Andrew Majorov corresponding to the usage of this method is the following:

Ensure transport module is loaded before checking exports

Tests were constantly failing without this patch. It seems ct starts erlang code server in interactive mode, so application module loading is defered.

It should be self-explanatory, but I found the whole usage of the code module interesting enough to point it out here!

Then, once the Transport module loaded, the Transport options are being validated using ranch:validate_transport_opts/1.

If it passes, then the method kindly asks the ranch_sup module to start the ranch_listener_sup supervisor using: supervisor:start_child(ranch_sup, ChildSpec).

It was possible to use ranch_sup atom (= the reference for the process named ranch_sup) instead of the process Pid only because the process is locally registered as ranch_sup. That’s because ranch_sup:start_link/1 calls supervisor:start_link({local, ?MODULE}, ?MODULE, []). See the official Erlang documentation.

The maybe_started/1 method is just there for error reporting enhancement purpose.

Hey! We can improve our little schema again!

Ranch Source 4

2.2. ranch_listener_sup.erl

It implements the supervisor behaviour, as expected.

This file is very simple. It basically starts two children, both being supervisors: ranch_conns_sup_sup and ranch_acceptors_sup. It was expected from the chapter dealing about the documentation.

The restart strategy is rest_for_one. It means that if ranch_conns_sup_sup dies, then ranch_acceptors_sup will be terminated and then both will be restarted sequentially. However, if ranch_acceptors_sup dies, it will be restarted but ranch_conns_sup_sup will not be affected.

If a supervisor starts A, then B, and B depends on A, the rest_for_one strategy is usually selected. Here, it is indeed used because ranch_acceptor relies on ranch_conns_sup. We will see later how. I anticipate a little bit here, as we wouldn’t know just now.

We can update our schema!

Ranch Source 5

2.3. ranch_conns_sup_sup.erl

This supervisor is starting the ranch_conns_sup supervisors. It starts by cleaning up all the ranch_conns_sup supervisors monitored by ranch_server. That’s because if ranch_conns_sup_sup is restarted, all its children would have been been killed and restarted as well.

There are two Transport parameters being used here:
  • num_acceptors - used only to get the default value of num_conns_sups

  • num_conns_sups

num_acceptors being defaulted at 10 while num_conns_sups is default to the current value of num_acceptors.

As indicated by its name, num_conns_sup is the number of connection supervisor ranch_conns_sup being started by ranch_conns_sup_sup for each listener.

I quote the official documentation:

num_acceptors (10)

Number of processes that accept connections.

num_conns_sups - see below

Number of processes that supervise connection processes. If not specified, defaults to be equal to num_acceptors.

By default Ranch will use one connection supervisor for each acceptor process (but not vice versa). Their task is to supervise the connection processes started by an acceptor. The number of connection supervisors can be tweaked.

Note that the association between the individual acceptors and connection supervisors is fixed, meaning that acceptors will always use the same connection supervisor to start connection processes.

Specifying a custom number of connection supervisors
{ok, _} = ranch:start_listener(tcp_echo,
	ranch_tcp, #{socket_opts => [{port, 5555}], num_conns_sups => 42}],
	echo_protocol, []
).

Also the following commit message from the 5th of August 2019 by "juhlig":

Add the num_conns_sups option

This new option allows configuring the number of connection supervisors. The old behavior can be obtained by setting this value to 1. A value larger than num_acceptors will result in some connection supervisors not being used as the acceptors currently only use one connection supervisor.

I am not certain I fully understand the purpose of num_conns_sups at the moment - you will see why later.

Note that each ranch_conns_sup module being spawned is attributed an Id - passed as a parameter to ranch_conns_sup:start_link/6 function. This Id is a digit ranging from 1 to the value of the maximum number of connection supervisors (num_conns_sups).

2.4. ranch_acceptors_sup.erl

It implements the supervisor behaviour.

This module’s responsibility is to start the listenning socket(s) and then delegating the socket(s) responsibility to each of the corresponding the ranch_acceptor being spawned.

The file uses two Transport options:
  • num_listen_sockets: Ranch will simply create as many sockets as indicated in this option - and then start listening to each of them. See the documentation:

The experimental num_listen_sockets option has been added. It allows opening more than one listening socket per listener. It can only be used alongside the Linux SO_REUSEPORT socket option or equivalent. It allows working around a bottleneck in the kernel and maximizes resource usage, leading to increased rates for accepting new connections.

  • num_acceptors: total number of ranch_acceptor module being spawned. One of the socket reference parameter previously created is passed to this module. Note that each ranch_acceptor module being spawned is attributed an AcceptorId - passed as a parameter to ranch_acceptor:start_link/5 function. This AcceptorId is a digit ranging from 1 to the value of the maximum number of acceptors (num_acceptors).

How do Ranch determines which socket reference to pass to the acceptor? The following code answers this question:

LSocketId = (AcceptorId rem NumListenSockets) + 1

As you probably know, the remainder r of an euclidian division a/b is such as

0 <= r < b

therefore we have :

1 <= LSocketId <= NumListenSockets

which is what we were looking for.

How is the distribution of LSocketId given AcceptorId is variable and NumListenSockets is constant?

When it could seem obvious that this distribution is even, you’ll read here that it isn’t. This specific post was in case AcceptorId would be constant instead of NumListenSockets. I haven’t been able to answer that question. If anyone has a glimpse, please start an issue on the GitHub repo!

2.5. ranch_conns_sup.erl

This file is rather unusual compared to what we faced until now. It does not implement the traditional supervisor behaviour. It’s a so-called special process. This will be interesting! If you don’t know anything about it, I recommend going through this article. I found it quite insightful.

As seen earlier, ranch_conns_sup:start_protocol/3 is called by ranch_acceptor. It ultimately calls Protocol:start_link/3 - which starts the connection process. It is not started the usual way like a supervisor would do because we’re using the delegate parameter. That’s the reason why a special process is used here instead of a regular supervisor.

This file is by far the most complex - and deserve all your attention.

Note that ranch_conns_sup registers itself to ranch_server’s ETS Table on its initialization using ranch_server:set_connections_sup/3.

We notice that this module maintains a state record that in particular hold the Parent Pid of this supervisor - which is ranch_conns_sup_sup Pid. It also holds the Transport and Protocol options as well as the max_conns parameter and the logger.

2.6. ranch_acceptor.erl

Right at its initialization, this module calls ranch_server:get_connections_sup(Ref, AcceptorId).

I’ll copy-paste the full implementation of the function here:

get_connections_sup(Ref, Id) ->
    ConnsSups = get_connections_sups(Ref),
    NConnsSups = length(ConnsSups),
    {_, Pid} = lists:keyfind((Id rem NConnsSups) + 1, 1, ConnsSups),
    Pid.

As a matter of readability, I’ll extend what get_connections_sups(Ref) is doing, and rename the parameter Id to what is really is, i.e AcceptorId:

get_connections_sup(Ref, AcceptorId) ->
    ConnsSups = [{Id, Pid} || [Id, Pid] <- ets:match(?TAB, {{conns_sup, Ref, '$1'}, '$2'})].
    NConnsSups = length(ConnsSups),
    {_, Pid} = lists:keyfind((AcceptorId rem NConnsSups) + 1, 1, ConnsSups),
    Pid.
Let’s summarize. The following steps will happen in this sequential order:
  1. First, ranch_listener_sup module will spawn ranch_conns_sup_sup

    1. ranch_conns_sup_sup will itself spawn num_conns_sups amount of connection supervisors ranch_conns_sup.

    2. During their initialization, each connection supervisor will be assigned an unique Id ranging from 1 to num_conns_sups and will register its Id to the ranch_server conns_sup ETS Table.

  2. Only after that, ranch_listener_sup will spawn ranch_acceptors_sup

    1. ranch_acceptors_sup will itself spawn num_acceptors amount of acceptors ranch_acceptor.

    2. During their initialization, each acceptor will be assigned an unique AcceptorId ranging from 1 to num_acceptors. and will call the above code with its own attributed AcceptorId. This code will simply pick a connection supervisor randomly (using the euclidian division remainder trick - like before)

See the ranch_listener_sup booting order exposed earlier to understand why these two steps happen sequentially.

Now, how does ranch_acceptor use the connection supervisor? It’s simple - let’s just look at the ranch_acceptor:loop/5 function.
  1. It calls Transport:accept(LSocket, infinity) which is a blocking operation.

  2. Then, on a new accepted connection, it calls Transport:controlling_process(CSocket, ConnsSup) to give control of the socket CSocket to ConnsSup, the connection supervisor previously picked.

  3. Finally it calls ranch_conns_sup:start_protocol(ConnsSup, MonitorRef,CSocket) - that will basically start the connection process.

3. Summary

We can now finalize our schema. It will include all the parameters and logic we’ve explicited in the two previous chapters. As a matter of simplicity, we’ll consider that the user only uses ranch:start_listener/5 once.

Ranch Source 6

In the next chapter, we’ll focus on the files we haven’t studied before.

4. The other files

The other files are all related to the other ranch modules API. They are implementing the features other than simply calling ranch:start_listener/5. In the first version of this document, the analysis of those other functions is not included.

Feel free to go on and contribute to this guide. You are more than welcome to add an issue our the GitHub repository and send a pull-request!

Conclusion

1. What is Ranch?

Ranch is a "Socket acceptor pool for TCP protocols".

4. To be continued

The analysis of what happens when calling other API functions than ranch:start_listener/5 has not been done yet. You’re welcome to open issues and send pull-request to Ranch: what’s under the hood @ GitHub .