[question] Hey! Can you tell us more about how you found that out in the code? I am curious! [question]
Introduction
Hi there! I am Nicolas - developer.
This document is a summary of my analysis of Ranch 2.0.0-rc.1 source code, performed in August 2019. It has been done purely in an old-school way, cloning the project from github and going through it manually.
By helping understand the intricacy of a well-established but somewhat small Erlang library, the reader can:
-
improve its Erlang proficiency
-
be able to help maintaining Ranch
-
be able to reuse Ranch codebase or get inspired by it for other projects
⚠
|
This analysis has been performed while I just started learning Erlang, right after going through Learn You Some Erlang For Great Good. If you find any errors, a pull-request or a new GitHub issue is welcome! ;) |
1. License and disclaimer
This work is distributed under the same license as Ranch itself, ISC License.
Ranch was originally built by Loïc Hoguin,
which kindly encouraged me to publish this code analysis.
I am not related to Nine Nines.
Donate if you wish to support Ranch.
2. TL;DR
Please, refer to the Conclusion.
3. Prerequisites
Having had basic experience in any development project with any programming language is highly recommended to grasp the following content.
It is assumed that the reader have basic knowledge of Erlang/OTP as well as hands-on experience in managing TCP/IP sockets - or at least enough theoretical knowledge to understand their implementation (in any language).
4. What is Ranch about?
Ranch is a "Socket acceptor pool for TCP protocols". It is at the foundation of the Erlang Web Server Cowboy, that is itself at the foundation of the Elixir Web Framework Phoenix. Both Web Servers are widely used in the Erlang/Elixir community.
There is an excellent manual maintained by the developers of the project.
However, it will not harm to summarize here the purpose of this library and describe the main concepts: Listener, Transport, Protocol, Acceptor, Pool, Connection…etc. I found it hard personally to understand how it all works together, so I’ll explain it with my words here.
Understanding these keywords and how they shape together is primordial to fathom the source code. Why? Because you’ll see a lot of those here and there in the code. And it will be way more fuzzy in the codebase that it is in the documentation. The developers of the codebase expect you to know about the terms already.
That’s why, you would rather learn them before. When you see a variable named "Protocol", you’ll then directly know what is its purpose, without having to finish the code analysis to do so.
(Actually, that’s a general truth in software development. You cannot write/read code efficiently if you don’t master what it is about on the first place.)
4.1. Description
I will simply quote the official documentation:
Ranch is a socket acceptor pool for TCP protocols.
Ranch manages listeners which are a set of processes that accept and manage connections. The connection’s transport and protocol modules are configured per listener. Listeners can be inspected and reconfigured without interruptions in service.
It is worth emphasizing that Ranch manage a pool of listeners.
4.2. Definitions
4.2.1. Generic terms
Let’s remind us about some fundamentals here.
Process
It’s the Erlang OTP process.
Quoting the official Erlang documentation:
Erlang is designed for massive concurrency. Erlang processes are lightweight (grow and shrink dynamically) with small memory footprint, fast to create and terminate, and the scheduling overhead is low.
Pool
A pool, is a general programming concept frequently used. Quoting Wikipedia:
A pool is a collection of resources that are kept ready to use, rather than acquired on use and released afterwards.
In the context of Ranch, and even in general in Erlang/OTP, we will mainly talk about pools of processes.
This might help: https://learnyousomeerlang.com/building-applications-with-otp
4.2.2. Ranch-specific concepts
Even though the following abstractions are seen in most networking librairies, their precise meaning in the context of their project differs from one to another.
If you weren’t already familiar with these notions, understanding what they are about in the context of Ranch will definitely help you identifying them in other libraries, or re-using them in new projects.
Listener (process)
Quoting the Ranch documentation:
A listener is a set of processes that accept and manage connections.
A listener is a set of processes whose role is to listen on a port for new connections. It manages a pool of acceptor processes, each of them indefinitely accepting connections. When it does, it starts a new process executing the protocol handler code. All the socket programming is abstracted through the use of transport handlers.
It is worth emphasizing that each listener manage a pool of acceptor processes.
Acceptor (process)
-
the role of an acceptor is to indefinitely accept connections
-
for each new connection being accepted, it starts and forwards the logic to a new process, named "connection process"
It is interesting to notice that while it’s the acceptor that starts the connection processes, it is the supervisor of the acceptor process, the listener, that supervises them:
The listener takes care of supervising all the acceptor and connection processes, allowing developers to focus on building their application.
Connection (process)
Quoting from the listener definition above:
[The connection process] executes the protocol handler code. All the socket programming is abstracted through the use of transport handlers.
As seen above, it is started at each accepted connection by the acceptor process, and is supervised by the listener process.
Its role is to actually implement the communication logic (what to do when receiving that message, or sending that other one, etc..).
Protocol (handler)
Quoting the Ranch documentation:
A protocol handler starts a connection process and defines the protocol logic executed in this process.
Actually this might be a little confusing at this point. Isn’t it the acceptor process that’s supposed to start the connection process???
I found it hard to understand the distinction between a "connection process" and a "Protocol handler" in the documentation. Well, to better comprehend this, I’ve had to cheat and already dive a little bit into the code. I will give here the result of my research, to help you out!
Basically, when they say in the quote above that "a protocol handler STARTS a connection process", you must somewhat understand "a protocol handler IS a connection process" instead. While it is not exactly true (the original sentence is correct), thinking this way personally helped me understand things better on the first place.
The distinction is subtle. What happens, in fact, is that everytime a connection is accepted, the acceptor process will use a method from its corresponding "connection supervisor" to "start the protocol". This function will start the Protocol initially passed as parameter, by the end user of the library. This Protocol (which is NOT a gen_server), when started, will in turn spawn a linked process (like a gen_server does). This process is what’s called the "connection process". This process shall embed all the protocol logic implemented by the user (like sending and receiving data). There is no requirement on what it does, it is left entirely to the user.
It means that a Protocol is a sort of interface that the user has to implement and pass as a parameter when starting the pool of listeners.
We can already expect such "interface" to be implemented as a custom behaviour
.
It is the idiomatic way of doing so in Erlang.
Okay… If you insist :).
See ranch_acceptor.erl line 41: ranch_conns_sup:start_protocol/3
is being used
in its ranch_acceptor:loop/5
main method on success of Transport:accept/2
.
ranch_conns_sup:start_protocol/3
is sending to himself a "start_protocol"
type of message using the bang operator (!
).
Then, in its main ranch_conns_sup:loop/4
method,
this message is pattern matched in a receive
block.
On reception of the message, it is running Protocol:start_link(Ref, Transport, Opts)
.
(See ranch_conns_sup.erl line 122.) Protocol, being a parameter that was being pushed through from the beginning
by the end user of the library using ranch:start_listener/5
.
A quick glance to ranch_protocol.erl and in the examples of the official documentation confirms that Protocol is the one process that’s supposed to actually implement the user’s high-level message sending and receiving logic.
And Protocol:start_link/3
, in the examples, like for gen_server,
is spawning a new linked OTP process.
Okay, let’s wrap it up!
The distinction between a Protocol handler and a connection process is therefore that a Protocol handler is the interface, and ultimately the parameter (a simple variable) being pushed through by the user. Being a parameter is the key that explain why it is called a "handler". In other words, it is a delegate (generic programming notion).
On the other side, the connection process is the actual OTP process being spawned by the Protocol when the Protocol is started. That’s what they meant in the official documentation by saying that "the Protocol handler starts a connection process".
Transport (handler)
Quoting the official documentation:
A transport defines the interface to interact with a socket.
Transports can be used for connecting, listening and accepting connections, but also for receiving and sending data. Both passive and active mode are supported, although all sockets are initialized as passive.
In short, everywhere in the code when an actual action is needed towards the sockets (accepting the tcp/ip connection, sending, receiving messages…etc), such action is ultimately delegated to the Transport interface. When you read interface, like for Protocol, so you probably guess now that it is implemented as a custom behaviour.
But contrary to Protocol, the user doesn’t have to implement this behaviour in every situation.
-
the TCP Transport (
ranch_tcp
) -
the SSL Transport (
ranch_ssl
).
Each of them are different modules that implemented the same behaviour.
They are thin wrappers on top of the gen_tcp
and ssl
modules, which are
the default erlang modules to do TCP/IP Socket programming and SSL encryption.
However, users can write, if they wants, a custom transport handler.
For that, they need to implement the ranch_transport
behaviour and follow the
the guide in the
documentation.
The internal documentation
Ranch does not publish all its documentation to the internet. The file ranch/doc/src/guide/internals.asciidoc contains useful information for us!
I’ll just rewrite part of it with my own words here and double check them in the code. Feel free to skip that part if you prefer just reading it instead (it is quite short).
1. The Ranch Server
-
When Ranch library starts, it starts the
ranch_server
module -
Ranch has a top supervisor that is supervising
ranch_server
and all the listeners started
"Ranch library starts" == Ranch is loaded as a dependency module by the end-user application
2. The ETS Table
There is an ETS table named ranch_server
started on application startup by the module
carrying the same name, and shared by all the application
It is own by the top application supervisor to avoid loss of information in case of restart.
If you guys don’t remember what’s an ETS table, check LYSEFGG or the official documentation.
3. On managing connections
There is a custom supervisor for managing connections. A glimpse at the code shows that it is ranch_conns_sup_sup
.
It keeps track of the number of connections so it can handle connection limits.
4. About listeners
-
listeners have their own supervisor,
ranch_listener_sup
-
ranch_listener_sup
starts three kind of processes:-
the listener gen_server (cannot see track of it in the code???)
-
the acceptor process supervisor
ranch_acceptors_sup
, which in turn start the acceptor process -
the connection process supervisor supervisor
ranch_conns_sup_sup
, which in turn starts the connection supervisorranch_conns_sup
, which in turn starts the connection process through the Protocol.
-
-
ranch_server
monitors some of these processes with various amount of information
That part is quite cool and would have saved us from a lot of troubles earlier ;) :
Protocol handlers start a new process, which receives socket ownership, with no requirements on how the code should be written inside that new process.
Take note of the "socket ownership" part, it’s rather important and interesting. If you have no clue what it means, read the end of this chapter of LYSEFGG. It seems logical that it is the process that’s ultimately taking care of the connection that become the owner of the socket.
The info on Transport is nothing new for us at this point.
The Examples
To understand the code, better starting off knowing how to use the library!
-
tcp_echo
-
tcp_reverse
The official documentation also extensively use them to explain the concepts.
1. TCP Echo Example
Start a TCP acceptor pool that will echo anything sent to its underlying sockets.
1.1. ebin/tcp_echo.app
It is worth noticing here that the ranch
dependency is part of the applications
field, as expected.
A quick check proves that the ranch binaries are available in the deps/
directory.
tcp_echo_app
is the entry point of the application, as it is the
value of the mod
parameter.
1.2. src/tcp_echo_app.erl
That’s the entry point of the application.
The module implements the "application" API with the methods start/2
and stop/1
- as expected.
In start/2
, ranch:start_listener/5
is called. That’s the core of Ranch. It starts the pool of TCP/IP listener.
Take note that the module echo_protocol
is being passed as parameter.
The next line tcp_echo_sup:start_link()
basically starts a custom supervisor.
Let’s see the supervisor code.
1.3. src/tcp_echo_sup.erl
It’s written as a dynamic supervisor, but…
No child is ever added using supervisor:start_child/2
in the rest of the files.
So as is, this supervisor is useless…
1.4. src/echo_protocol.erl
This module was passed as parameter in ranch:start_listener/5
.
It implements the ranch protocol behaviour (-behaviour(ranch_protocol)
).
The entry point being start_link/3
that basically spawn a linked process that will
ultimately run init/3
.
Contrary to a traditional gen_server
that’s not init/1
that’s being called.
init/3
runs ranch:handshake(Ref)
which most probably initiates the Socket acceptors…?
Then it runs a recursive loop
function, that simply pattern match the data being received.
Transport:recv(Socket, 0, 5000)
being probably a blocking operation, it just waits until data is received.
Then, it reads the first data and if anything is received, echoes back the same message using Transport:send(Socket, Data)
and start the same function loop
again.
Ultimately, when no data is received, it closes the socket using Transport:close(Socket)
.
Then it goes out of the loop and the process terminates.
1.5. What did we learn?
-
ranch:start_listener/5
must be used by the user, and is the starting point of the TCP/IP listener pool -
as expected,
ranch_protocol
is a behaviour that people have to implement and then pass toranch:start_listener/5
as an argument. -
ranch:handshake/1
must be run when initializing the Protocol, before starting the connection process
2. TCP Reverse example
Start a TCP acceptor pool that will echo the reversed version of anything sent to its underlying sockets.
2.1. ebin/tcp_reverse.app
Nothing much different from before here, just a basic .app
file which tells us
that tcp_reverse_app
is the starting module of the application.
2.2. src/tcp_reverse_app.erl
Starting the ranch_listener, like before.
Still using ranch_tcp
transport, and this time the custom reverse_protocol
.
2.3. src/tcp_reverse_sup.erl
Same as before. Useless supervisor here.
2.4. src/reverse_protocol.erl
This example is useful to show how to integrate a gen_statem
custom protocol handler
with Ranch.
It would also work the same way for a gen_server
.
The official Ranch documentation is enough.
Read LYSEFGG and the official Erlang documentation to know more about Finite State Machines and gen_statem.
3. Summary
We can now already make a schema summarizing how the different notions integrate together, based on the examples and the documentation.
-
the "supervising" and "monitoring" part here are to be understood in a very high-level way. We are not yet talking about how it actually works under the hood in terms of supervision tree between the modules. Still, it helps creating a mental image of the Ranch architecture.
-
the steps related to the Acceptor are also simplified and will be discussed further later
Diving into Ranch Source code
-
what happens when the library is loaded?
-
what happens when the user call
ranch:start_listener/5
?
1. What happens when the library is loaded?
1.1. ebin/ranch.app
This is the main file defining the project. Let’s decrypt it together and remind ourselves what is the meaning of the different values.
-
{application, 'ranch'}
-
Thats the name of the OTP application being created.
'ranch'
is an atom.
-
-
{description, "Socket acceptor pool for TCP protocols."}
-
The description of the application. A string.
-
-
{vsn, "2.0.0-rc.1"}
-
The version of the application. A string.
-
-
{modules, ['ranch','ranch_acceptor','ranch_acceptors_sup','ranch_app','ranch_conns_sup','ranch_conns_sup_sup','ranch_crc32c','ranch_embedded_sup','ranch_listener_sup','ranch_protocol','ranch_proxy_header','ranch_server','ranch_server_proxy','ranch_ssl','ranch_sup','ranch_tcp','ranch_transport']}
-
"All modules introduced by this application. systools uses this list when generating start scripts and tar files. A module can only be defined in one application."
-
It does correspond to all the
.erl
files of thesrc/
folder.
-
-
{registered, [ranch_sup,ranch_server]}
-
The list of processes that will be accessible by their name instead of only by PID (== registered processes, see official Erlang documentation).
-
-
{applications, [kernel,stdlib,ssl]}
-
"All applications that must be started before this application is allowed to be started. systools uses this list to generate correct start scripts. Defaults to the empty list, but notice that all applications have dependencies to (at least) Kernel and STDLIB."
-
Here we can note that Ranch needs the ssl external module to be loaded beforehand. Obviously, Ranch does use this security protocol so it makes sense.
-
-
-
{mod, {ranch_app, []}}
-
"Specifies the application callback module and a start argument, see application(3). Key mod is necessary for an application implemented as a supervision tree, otherwise the application controller does not know how to start it. mod can be omitted for applications without processes, typically code libraries, for example, STDLIB."
-
That’s interesting! The starting point of the application being always
ranch_app
. We’ll keep that in mind and start by this file for our source code anaylisis in the next chapter.
-
-
-
{env, []}
-
"Configuration parameters used by the application. The value of a configuration parameter is retrieved by calling application:get_env/1,2. The values in the application resource file can be overridden by values in a configuration file (see config(4)) or by command-line flags (see erts:erl(1))."
-
No default configuration parameter being set here.
-
-
]}.
Source: Official Erlang Documentation
1.2. ranch_app.erl
Yep, it has -behaviour(application)
, which makes sense because it is part of the mod
parameter above.
The file is quite small and easy to understand.
-
that’s where profiling logic is in
-
ranch_sup:start_link/0
is run on start of the application.
That’s the supervisor for the whole Ranch app. We can now start our supervisor tree:
See LYSEFGG for more info on the Application Master. Let’s dive in ranch_sup to know what’s behind these question marks.
1.3. ranch_sup.erl
As expected, it implements the supervisor
behaviour.
The following code:
start_link() →
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
is standard. No argument will be passed to init/1
.
init/1
with an empty argument list will be run in the spawned supervisor process.
We will now study its content.
1.3.1. The restart intensity configuration parameters
application:get_env(ranch_sup_intensity)
is used to access the ranch_sup_intensity
configuration parameter.
Similarly the ranch_sup_period
configuration parameter is being accessed.
Then it is returned by the init/1
function as follow:
{ok, {#{intensity ⇒ Intensity, period ⇒ Period}, Procs}}
.
-
"To prevent a supervisor from getting into an infinite loop of child process terminations and restarts, a maximum restart intensity is defined using two integer values specified with keys intensity and period in the above map. Assuming the values MaxR for intensity and MaxT for period, then, if more than MaxR restarts occur within MaxT seconds, the supervisor terminates all child processes and then itself. The termination reason for the supervisor itself in that case will be shutdown. intensity defaults to 1 and period defaults to 5."
Interestingly, these configuration parameters are not documented in the official Ranch Documentation. In migration_from_1.5.asciidoc
, I quote:
"It is now possible to configure the restart intensity for
ranch_sup
using the OTP application environment. This
feature will remain undocumented unless there is popular
demand for it."
test/ranch_ct_hook.erl
:-
application:set_env(ranch, ranch_sup_intensity, 10)
-
application:set_env(ranch, ranch_sup_period, 1)
1.3.2. The ChildSpec
The last parameter returned by init/1
is the variable Procs
that corresponds to
what’s called ChildSpec
in the official Erlang documentation.
-
Procs = [#{id ⇒ ranch_server, start ⇒ {ranch_server, start_link, []}}
which is pretty standard.
ranch_sup
is therefore supervising ranch_server
, which entry point is start_link/0
.
We can update our supervision tree:
It is also worth noticing that an ETS table is being initialized using ets:new/2
.
The ETS table is named after ranch_server
module so it is expected that this module will be interacting with it.
We will start by ranch_server
.
1.4. ranch_server.erl
The first thing to notice is that it implements gen_server
, as expected by its name.
gen_server
keeping? As usual, you need to look for it at the beginning of the file, after the list of export
:-
A macro
TAB
is being defined as?MODULE
but not used throughout the whole file - which is a bit awkard. Sometimes we see the usage of?MODULE
, sometimes?TAB
. I don’t understand the point of this macro. Maybe there’s a good reason for it that I am missing out. -
The interesting part is the record
state
being initialized with an empty list.state
record contains a value of typemonitors()
which is itself defined as follows:-
[{{reference(), pid()}, any()}]
see official Erlang documentation for explanation of the types
-
start_link/0
without surprises is running gen_server:start_link/4
, with standard arguments.
We know that the init/1
function is the starting point of this process. We therefore scroll the file down to see its implementation.
As expected, init/1
returns {ok, State}
with State
being here
of type state
, record defined previously (see above).
The list of all connections and listener supervisors is fetched from the ETS table previously created.
ranch_server
starts monitoring them,
and store the returned reference in the state,
together with its corresponding Pid,
the Reference of the supervisor process itself,
and its Id (for connection supervisor).
From that point on, it would be interesting to look for the location where the connection and listener supervisors are added to the ETS table.
To do so, we can search for "conns_sup" or "listener_sup" or "ets:insert" in the project.
ranch_server:handle_call/3
:-
State = set_monitored_process({conns_sup, Ref, Id}, Pid, State0)
called byranch_server:set_connections_sup/3
, which is itself called byranch_conns_sup:init/7
-
State = set_monitored_process({listener_sup, Ref}, Pid, State0)
called byranch_server:set_listener_sup/2
, which is itself called byranch_listener_sup:init/1
ranch_server:set_monitored_process/3
is simply making ranch_server
monitor the connection/listener supervisor,
update the ETS Table accordingly and return the new updated state.
It is called on creation of the supervisors.
-
Why is
erlang:monitor(process, Pid)
being called ininit/1
? It will stack with another monitoring reference, because the supervisors fetched from the ETS Table are supposed to be already monitored (seeranch_server:set_monitored_process/3
)… Why isn’t the ETS Table also storing the MonitorRef? It would avoid having to do that… Or is that the intended behaviour? -
In any case, most likely, the ETS Table will be empty at this point, because
ranch_server:init/1
has to have finished running before theranch_server
gen_server becomes actually available for users, and before they can eventually callranch_server:handle_call/3
! Therefore, the returned initial state will be empty on start ofranch_server
and the code that fetch the ETS Table is irrelevant…
What happens when ranch_server
dies and is restarted by ranch_sup
?
In this case, there will still be data in the ETS Table, data that will be monitored by…
the ranch_server
process that just died! So, basically not monitored at all!
That’s it! That’s the reason why the use of erlang:monitor(process, Pid)
in ranch_server:init/1
does not stack with another monitoring.
Its only (and important!) point is in case the process that was previously monitoring died!
ranch_server:get_listener_sup/1
is used in:-
ranch:get_status/1
. -
ranch:suspend_listener/1
-
ranch:resume_listener/1
-
ranch:remove_connection/1
-
ranch:info/0
andranch:info/1
-
ranch:procs/1
ranch_server:get_connections_sup/1
is used in:-
ranch:get_connections/2
-
ranch:apply_transport_options/2
These functions are the API that’s accessible by the user of the library. They are features. See the official documentation to know more about their usage.
Finally, ranch_server:get_connections_sup/2
is used by ranch_acceptor:start_link/5
.
If you remember well the previous chapter, the listener was the one that was creating
the connection supervisor supervisor and the acceptor supervisor. Which means they were basically creating them.
However, it is obviously the acceptor which needs to trigger the connection supervisor to start the connection process.
So the acceptor has to know about a process that’s neither his parent nor his child.
That’s why he needs to get it from an ETS table, that’s handled by ranch_server
!
We can now improve our schema:
So, as expected after the study of the documentation,
everytime a connection supervisor or a listener supervisor is started,
it is registered within the ranch_server
.
Now we are wondering: when are they started?
2. What happens when the user call ranch:start_listener/5
?
2.1. ranch.erl
That’s the where all the API functions are stored.
The most important one here being ranch:start_listener/5
. Let’s dive in its implementation.
Open the source code, and look at it with me!
It starts by "normalizing" the TransportOpts0
parameter using ranch:normalize_opts/1
.
ranch:start_listener/5
:-
a list of socket options OR
-
a map
The list of socket options being the old way, it is still handled by modern Ranch for backward compatibility purpose.
Interestingly, code:ensure_loaded(Transport)
is being used.
If you are no idea what module loading, interactive and embedded mode are, then
check the official documentation.
The commit message from Andrew Majorov corresponding to the usage of this method is the following:
Ensure transport module is loaded before checking exports
Tests were constantly failing without this patch. It seems ct starts erlang code server in interactive mode, so application module loading is defered.
It should be self-explanatory,
but I found the whole usage of the code
module interesting enough to point it out here!
Then, once the Transport module loaded, the Transport options are being validated
using ranch:validate_transport_opts/1
.
If it passes, then the method kindly asks the ranch_sup
module
to start the ranch_listener_sup
supervisor using: supervisor:start_child(ranch_sup, ChildSpec)
.
⚠
|
It was possible to use
ranch_sup atom (= the reference for the process named ranch_sup ) instead of the process Pid only
because the process is locally registered as ranch_sup .
That’s because ranch_sup:start_link/1 calls supervisor:start_link({local, ?MODULE}, ?MODULE, []) .
See the official Erlang documentation.
|
The maybe_started/1
method is just there for error reporting enhancement purpose.
Hey! We can improve our little schema again!
2.2. ranch_listener_sup.erl
It implements the supervisor
behaviour, as expected.
This file is very simple. It basically starts two children, both being supervisors:
ranch_conns_sup_sup
and ranch_acceptors_sup
.
It was expected from the chapter dealing about the documentation.
The restart strategy is rest_for_one
.
It means that if ranch_conns_sup_sup
dies, then ranch_acceptors_sup
will be terminated and then both will be restarted sequentially.
However, if ranch_acceptors_sup
dies, it will be restarted but ranch_conns_sup_sup
will not be affected.
If a supervisor starts A, then B, and B depends on A, the rest_for_one strategy is usually selected.
Here, it is indeed used because ranch_acceptor
relies on ranch_conns_sup
.
We will see later how.
I anticipate a little bit here, as we wouldn’t know just now.
We can update our schema!
2.3. ranch_conns_sup_sup.erl
This supervisor is starting the ranch_conns_sup supervisors. It starts by cleaning up all the ranch_conns_sup supervisors monitored by ranch_server. That’s because if ranch_conns_sup_sup is restarted, all its children would have been been killed and restarted as well.
-
num_acceptors - used only to get the default value of num_conns_sups
-
num_conns_sups
num_acceptors
being defaulted at 10
while num_conns_sups
is default to the current value of
num_acceptors
.
As indicated by its name, num_conns_sup
is the number of connection supervisor ranch_conns_sup
being started by ranch_conns_sup_sup
for each listener.
I quote the official documentation:
- num_acceptors (10)
Number of processes that accept connections.
- num_conns_sups - see below
Number of processes that supervise connection processes. If not specified, defaults to be equal to
num_acceptors
.
By default Ranch will use one connection supervisor for each acceptor process (but not vice versa). Their task is to supervise the connection processes started by an acceptor. The number of connection supervisors can be tweaked.
Note that the association between the individual acceptors and connection supervisors is fixed, meaning that acceptors will always use the same connection supervisor to start connection processes.
Specifying a custom number of connection supervisors{ok, _} = ranch:start_listener(tcp_echo, ranch_tcp, #{socket_opts => [{port, 5555}], num_conns_sups => 42}], echo_protocol, [] ).
Also the following commit message from the 5th of August 2019 by "juhlig":
Add the num_conns_sups option
This new option allows configuring the number of connection supervisors. The old behavior can be obtained by setting this value to 1. A value larger than num_acceptors will result in some connection supervisors not being used as the acceptors currently only use one connection supervisor.
I am not certain I fully understand the purpose of num_conns_sups
at the moment -
you will see why later.
Note that each ranch_conns_sup
module being spawned is attributed an Id
- passed as a parameter
to ranch_conns_sup:start_link/6
function.
This Id
is a digit ranging from 1 to the value of the
maximum number of connection supervisors (num_conns_sups
).
2.4. ranch_acceptors_sup.erl
It implements the supervisor behaviour.
This module’s responsibility is to start the listenning socket(s) and then delegating
the socket(s) responsibility to each of the corresponding the ranch_acceptor
being spawned.
-
num_listen_sockets
: Ranch will simply create as many sockets as indicated in this option - and then start listening to each of them. See the documentation:
The experimental
num_listen_sockets
option has been added. It allows opening more than one listening socket per listener. It can only be used alongside the LinuxSO_REUSEPORT
socket option or equivalent. It allows working around a bottleneck in the kernel and maximizes resource usage, leading to increased rates for accepting new connections.
-
num_acceptors
: total number ofranch_acceptor
module being spawned. One of the socket reference parameter previously created is passed to this module. Note that eachranch_acceptor
module being spawned is attributed anAcceptorId
- passed as a parameter toranch_acceptor:start_link/5
function. ThisAcceptorId
is a digit ranging from 1 to the value of the maximum number of acceptors (num_acceptors
).
How do Ranch determines which socket reference to pass to the acceptor? The following code answers this question:
LSocketId = (AcceptorId rem NumListenSockets) + 1
As you probably know, the remainder r of an euclidian division a/b is such as
0 <= r < b
therefore we have :
1 <= LSocketId <= NumListenSockets
which is what we were looking for.
How is the distribution of LSocketId given AcceptorId is variable and NumListenSockets is constant?
When it could seem obvious that this distribution is even, you’ll read here that it isn’t. This specific post was in case AcceptorId would be constant instead of NumListenSockets. I haven’t been able to answer that question. If anyone has a glimpse, please start an issue on the GitHub repo!
2.5. ranch_conns_sup.erl
This file is rather unusual compared to what we faced until now.
It does not implement the traditional supervisor behaviour.
It’s a so-called special process
. This will be interesting!
If you don’t know anything about it, I recommend going through
this article.
I found it quite insightful.
As seen earlier, ranch_conns_sup:start_protocol/3
is called by ranch_acceptor
.
It ultimately calls Protocol:start_link/3
- which starts the connection process.
It is not started the usual way like a supervisor would do because we’re using the delegate
parameter.
That’s the reason why a special process is used here instead of a regular supervisor.
This file is by far the most complex - and deserve all your attention.
Note that ranch_conns_sup
registers itself to ranch_server’s ETS Table on its initialization
using ranch_server:set_connections_sup/3
.
We notice that this module maintains a state
record that in particular hold the Parent Pid
of this supervisor -
which is ranch_conns_sup_sup
Pid. It also holds the Transport and Protocol options
as well as the max_conns
parameter and the logger.
2.6. ranch_acceptor.erl
Right at its initialization, this module calls
ranch_server:get_connections_sup(Ref, AcceptorId)
.
I’ll copy-paste the full implementation of the function here:
get_connections_sup(Ref, Id) ->
ConnsSups = get_connections_sups(Ref),
NConnsSups = length(ConnsSups),
{_, Pid} = lists:keyfind((Id rem NConnsSups) + 1, 1, ConnsSups),
Pid.
As a matter of readability, I’ll extend what get_connections_sups(Ref)
is doing,
and rename the parameter Id
to what is really is, i.e AcceptorId
:
get_connections_sup(Ref, AcceptorId) ->
ConnsSups = [{Id, Pid} || [Id, Pid] <- ets:match(?TAB, {{conns_sup, Ref, '$1'}, '$2'})].
NConnsSups = length(ConnsSups),
{_, Pid} = lists:keyfind((AcceptorId rem NConnsSups) + 1, 1, ConnsSups),
Pid.
-
First,
ranch_listener_sup
module will spawnranch_conns_sup_sup
-
ranch_conns_sup_sup
will itself spawnnum_conns_sups
amount of connection supervisorsranch_conns_sup
. -
During their initialization, each connection supervisor will be assigned an unique Id ranging from 1 to
num_conns_sups
and will register its Id to theranch_server
conns_sup
ETS Table.
-
-
Only after that,
ranch_listener_sup
will spawnranch_acceptors_sup
-
ranch_acceptors_sup
will itself spawnnum_acceptors
amount of acceptorsranch_acceptor
. -
During their initialization, each acceptor will be assigned an unique AcceptorId ranging from 1 to
num_acceptors
. and will call the above code with its own attributed AcceptorId. This code will simply pick a connection supervisor randomly (using the euclidian division remainder trick - like before)
-
See the ranch_listener_sup
booting order exposed earlier to understand why these two steps
happen sequentially.
ranch_acceptor
use the connection supervisor? It’s simple - let’s just look at the ranch_acceptor:loop/5
function.-
It calls
Transport:accept(LSocket, infinity)
which is a blocking operation. -
Then, on a new accepted connection, it calls
Transport:controlling_process(CSocket, ConnsSup)
to give control of the socket CSocket to ConnsSup, the connection supervisor previously picked. -
Finally it calls
ranch_conns_sup:start_protocol(ConnsSup, MonitorRef,CSocket)
- that will basically start the connection process.
3. Summary
We can now finalize our schema. It will include all the parameters and logic we’ve explicited
in the two previous chapters.
As a matter of simplicity, we’ll consider that the user only uses ranch:start_listener/5
once.
In the next chapter, we’ll focus on the files we haven’t studied before.
4. The other files
The other files are all related to the other ranch
modules API.
They are implementing the features other than simply calling ranch:start_listener/5
.
In the first version of this document, the analysis of those other functions is not included.
Feel free to go on and contribute to this guide. You are more than welcome to add an issue our the GitHub repository and send a pull-request!
Conclusion
1. What is Ranch?
Ranch is a "Socket acceptor pool for TCP protocols".
4. To be continued
The analysis of what happens when calling other API functions than ranch:start_listener/5
has not been done yet.
You’re welcome to open issues and send pull-request to Ranch: what’s under the hood @ GitHub
.