Web cluster - experience of real application. And also in the boxed version of the Bitrix Web cluster service

Greetings, dear accomplices!

This article is about how we implemented a web cluster for a news portal (with a peak of 130 thousand unique visitors per day - that’s 7TB of traffic in 3 days - elections and 2 subsequent ones. Now on average the cluster distributes 35-40 TB of traffic per day month), about how programmers and journalists understand the same tasks differently, about how you can achieve the same goal by taking different paths.

It will be of interest to those who want to build an easily scalable, geographically distributed web cluster without investing astronomical amounts in equipment (and by television standards, the amounts will be generally ridiculous).

I am more than sure that marketers pushing uber solutions for newly released products with the words “scalable web cluster” or “horizontal infinite scalable web cluster” in their names will hate me.

I am more than sure that our clients' competitors will be surprised by the simplicity of the solution we used.

I won’t provide banal configs that can be found in any tutorial on setting up PHP, Nginx and Firebird (the latter, strictly speaking, has nothing to configure - everything works right out of the box) and in general I will talk about the essence decisions, not about which version of PHP is better.

Experienced designers are unlikely to be interested - they already know everything, but for those who are just starting out in the difficult field of system design, it’s more difficult than “Hello, World!” there will certainly be something to learn - the solution has been in combat mode for almost 2 years now, and no architectural problems arose (although two hard drives on two of the three nodes failed at once, but no one noticed anything - the sites opened a little slower , than usual).

So, a few words about how it all began. Once upon a time there was a website of a group of independent journalists who really dreamed of becoming a real television (looking ahead, I will say that they succeeded - they created their own, quite successful television with “blackjack and...” - further in the text). Our country is small, nothing terrible happens (and we are happy about it), but once every 4 years we traditionally have elections to Parliament. Which traditionally does not elect the President in any way. (Don’t be afraid, there will be no politics here, this is just for a general understanding of the moment).

Of course, in the period before the elections and a little after, all online media are very unstable. Some sites don’t just lie there, they are lying around, some periodically try to provide at least a line of text, but the problem is universal and well-known - sites cannot cope with the influx of visitors. I'm talking about regular, text-based sites. And our clients had an unusual website. The fact is that they also had videos - news stories, they produced 10 gigabytes a month - at that time, now they create that many videos a day. On top of everything else (this is the last mention of politics, honestly) these journalists were not particularly loyal to the authorities. They said and wrote what they wanted. Quite insolent, right? We always position ourselves as mercenaries - the client offers a problem, we offer its solution. Everything else does not interest us; we maintain neutrality.

We were given the task of offering a solution for news sites that would not only allow them to withstand an influx of 50-100 thousand visitors, but would also be geographically scattered around the world and controlled from a mobile bunker anywhere in the Universe on the planet. Of course, the geographical dispersion of the cluster nodes was left to us. As a result, we proposed the following scheme (I’m not much of an artist, if you’ll excuse me):

(This is a simplified diagram for November; later almost all servers were transferred to Hetzner, since Netdirekt’s channel was constantly struggling at that time. Now the situation with their network is much better).
Regular visitors see one of 3 servers, while we made it so that all visitors from Moldova pulled “light” content in the form of text and pictures from one of 3, and “heavy” content (video) pulled from server located at a local provider. External visitors simply did not see the Moldovan mirror and pulled all the content from one of the German servers.

This is what we got with visitors as a result (each part of the portal has its own counter):

This scheme allows you to change the control server at any time, it itself checks the availability of cluster nodes, it is easily scalable - Amazon EC was also considered as a backup - moreover, Amazon EC was even used for some time for video streaming (about 4 months), but due to Due to the high cost of traffic, it was decided to move the streaming nodes to the German Hetzner.
Immediately 2 weeks before hour “X” we took backup servers and kept them ready (but users did not see them, since keeping the server active is somewhat cheaper than using it in combat mode - only because of traffic).

How does it all work? Very simple - silently and around the clock;).

On the control server (as I already mentioned, the portal has 2 large “sections”: news in the form of text with pictures and news in the form of a text digest and video; de facto, 2 servers are used, but for simplicity I depicted one) there is something that commonly referred to as a content management system.

The main purpose of this server is to allow journalists to add news. Once at a certain time (3-5 minutes), a script starts that creates... an offline copy of the site. Of course, only pages that have changed or that need to be rebuilt due to crosslinks and dependencies are generated.

This is very easy to implement using sequences and cascading triggers in Firebird - the procedure only needs to make changes to the main page of the site, and cascading triggers will update all dependencies, marking each page that needs updating. The mark is set not in the form of a 1/0 flag, but in the form of a unique number obtained based on the generator. The script for creating an offline version at startup recognizes the new value of the generator, reads the value of this generator from its previous launch and recreates all pages in the resulting range. At the same time, since we use the Firebird transactional mechanism, the script does not care what changes occur during its execution - i.e. We always create a coherent and consistent version of the site, no matter what the reporters do.

Thus, we create a master copy of the portal (or two portals, if you like - text and video). The script (like the admin panel itself) is written in PHP and uses ADODB to work with Firebird - so it can be quite easily rebuilt at the request of the customer*.

(* But we are going to get rid of ADODB soon in all our future projects - its universality is only harmful, since a normal mechanism for working with the database that allows you to use all the features of the Firebird SQL server (as well as others) is not implemented there - for example , it is impossible to catch exceptions when fetching from selective procedures, there is no flexible transaction management and in general, these classes have too much AI - I prefer to decide on my own when I want to roll back a transaction and when to confirm.)

The only thing that had to be changed in the Firebird settings was the value of the database page cache size - since the number of connections to the database is very small (rarely more than 50-60 simultaneous connections), the number of pages was experimentally increased to 2048 (we use the Classic option , for the Super architecture this value can easily be increased by 10 times, since there is a shared page cache. In the upcoming version of Firebird 3.0, the developers promise one SMP-friendly architecture with a shared cache, so for it it will be possible to use large values for the page cache settings. ).

Then, using regular rsync, the difference in changes is scattered across mirrors, which are ordinary nodes for distributing static files based on Nginx. I think there is no need to tell you what a 4-core server with 12 GB of RAM is capable of when distributing only static data? ;-)

At the same time, 10% of the channel of each node (this is exactly 10 or 100 megabits, depending on the specific connection) is reserved for synchronization. Synchronization occurs in 2 stages - first, “heavy” content is synchronized - pictures and videos, then - text (html/xml/js).

Sometimes (when the channels from the control server to the mirrors are busy), the visitor may see small inconsistencies in the form of non-loading pictures and/or videos - since round-robin DNS is used, the user can receive the text of the page from one mirror, and the link to the video from another . This does not prevent the portal from working - the text is always there, and a picture or video will appear sooner or later.

Since the sites have dynamic forms - for example, subscribing to a newsletter - these forms are processed by a separate dedicated server (it is not shown in the diagram, but this does not change the essence). Even if we assume that all visitors rush to subscribe to the news at the same time and this server goes down, nothing bad will happen - the forms are loaded into the iframe and the absence of these forms does not affect the availability of the news.

Adding a new node is simple - first the new mirror is synchronized with the main copy (this happens in parallel with the normal mode of operation of the synchronizer), then the entry is added to the DNS and... no one notices anything.

Removing a node is simpler - simply removing the entry from the DNS. Adding and deleting are quite amenable to automation (this is exactly what we did with the part that was responsible for web streaming of about 1000 megabit streams on Amazom EC), but if you suddenly decide to repeat this, I advise you to first calculate how much the initial data synchronization takes (for us it 2 Gigabytes for the “light version of the portal” and approximately 1 Terabyte for video, stored only for the last few months).

This is why we removed the dynamic addition/removal of nodes from the spare pool after some time in the project's work - the content took up too much and the script turned out to be too paranoid - it removed nodes whenever there was a problem communicating with them.

Separately, it is worth mentioning the calculations of news displays. I got the impression that journalists’ favorite pastime (besides writing/filming reports) is measuring the number of visitors to a particular news story. We had to spend about a liter of blood and a kilometer of nerves to convince the journalistic fraternity that it was not necessary to display counter changes in real time.

Programmers understand perfectly well that it is simply impossible to find out how many people are currently reading an article; you can only count the number of requests for an article from the server, but for journalists these are the same concepts. :)

To count views, we contacted Kiril Korinsky (also known as catap), who kindly agreed to add the URL view counting feature to his Nginx branch. Well, then everything is simple - all nodes are periodically polled and page counters are taken into account in the properties of the page itself. Since the counters (i.e. the values themselves) are stored in separate files (currently one file per news item, we are soon planning to make a group of counters in one file to reduce the number of files themselves) - it is not the site pages that are transferred during synchronization entirely, but only the counter file. With a large number of files, this creates an additional load on the disk subsystem - therefore, when using the same approach, immediately think about how to divide the counters into groups - we settled on dividing the counters by news type and date - the files are relatively small and over time they stop changing , since practically no one is interested in old news.

Here, in brief, are the advantages of the solution we used:

Using static sites as web cluster nodes allows you to reduce the administration of the entire cluster to a few routine tasks - updating the operating system of the nodes and paying for traffic. This same point allows you to scatter cluster nodes geographically, which, together with GEO-DNS, can be generally considered as some kind of analogue of a content delivery network (CDN) - in fact, that’s what it is.
Using the transactional mechanism of the database and transferring the logic to the database itself allows you to always have a holistic and logically consistent version of the site - however, I would be very surprised if the “slice” of data from the server was logically inconsistent.
If an influx of visitors is expected, then simply increasing the cluster nodes can easily cope with it. In our case, the complete commissioning of a new node took a little over an hour for the text part of the portal and about a day (you can’t cram in something that can’t be squeezed in) for the video. If you accept partial synchronization of sites and add the rest in the background, then entering a new node for video can also be reduced to an hour.
An administrative server can be made from any of the nodes (if necessary) - just deploy a backup of the database (in compressed form, about a hundred megabytes). All other content is already there.

Well, a couple of minuses, so that not everything seems so rosy:

The solution is not suitable for cases where there are parts of the site that different users should see differently, i.e. when, according to the conditions of the task, pages are generated personally for each user. In our case, this turned out to be unnecessary.
Visitor counters are updated with a lag of approximately half an hour to an hour. Tolerable, but you will have to convince the client of this for a long time.
The biggest problem is synchronization with a local mirror - our providers do not yet sell traffic at a price of 7 euros per Terabyte, and even if they provide 100 honest Megabits to the world, it is at very inadequate prices.
Design a less paranoid system for tracking cluster nodes - ours turned out to be too sensitive, we had to put adding and removing nodes into manual mode.

And literally a small pinch of experience that will add variety to the bland mess of everyday life.

We use the JFS file system to store an offline copy of the site. It has proven itself very well both when working with many small files and when working with large files (in my experience, only XFS can almost instantly delete a file of 200-300GB in size).

So, by default the file system was mounted with default parameters. But since over time we had a lot of files, disk operations began to slow down a little. Since we do not need the time of last access to the file, I added the “noatime” option to the FS mounting parameters. This is what happened - I think you can decide for yourself when to add it:

Let me repeat briefly - for stable operation in normal mode, use:

3 servers for content distribution
2 servers for “admin”
2 servers for DNS and a tracking system for other servers.

The cluster nodes are scattered geographically and are located at different providers.
In case of expected events that attract a large number of visitors, additional servers are connected to distribute content.

About 40TB of traffic is consumed per month, the total volume of content is just over 1 Terabyte, video content is stored for about 3 months.

I will be happy to answer questions from the habra community.

Any new or running project on "" can be presented as a web cluster of interchangeable servers.

The main tasks that this project configuration allows you to solve:

As traffic increases, you can quickly add new servers to the cluster.
If one of the cluster servers fails, the system continues to continuously serve Clients.
Balancing load, traffic, data between several servers.
The system allows you to take backup copies from specially designated cluster nodes without affecting the operation of the site.

"Geographic web cluster"

A geographic web cluster allows you to raise entire groups of servers. Each of these groups has its own master - in data centers independent from each other. Thus, your websites and your business are completely protected from the inaccessibility of the data centers themselves.

"1C-Bitrix: Web Cluster" is a combination of technologies:

Vertical sharding(moving modules to separate MySQL servers)
MySQL replication and load balancing between servers
Distributed data cache(memcached)
Session continuity between web servers(storing sessions in a database)
Web Server Clustering:

File synchronization
Load balancing between servers

Data center independence(in the event of a failure of one data center, another immediately comes into operation, without the need to restore a “backup”)

How does it work

1. Vertical sharding

Dividing one web application database into two or more databases by separating separate modules, without changing the logic of the web application.

The following product modules can be placed in separate databases:

2. MySQL replication and load balancing between servers

The “master - slave” scheme is implemented using MySQL.

The 1C-Bitrix: Site Management platform allows you to flexibly balance the load between servers participating in replication.

Key Features:

flexible SQL load balancing
ease of administration
cheap and fast unlimited scaling
online backup
no modification of web application logic required

3. Distributed data cache (memcached)

"1C-Bitrix: Web Cluster" allows you to use a pool of memcached servers to work with the data cache.

This provides:

high efficiency- due to the centralized use of the cache by the web application
reliability- due to the caching subsystem’s resistance to failure of individual components
unlimited scalability- by adding new memcached servers

4. Continuity of sessions between web servers (storing sessions in a database)

The ability to store user session data in a database ensures session “transparency” for all web servers in the cluster:

After authorization on one of the servers, the user must be considered authorized for all other servers.
And vice versa - the end of a session on any server should mean its end on all servers at once.

With Bitrix24 you can work either with a cloud service or install it as a software product separately within the company.

What is the difference? Boxed version - "1C-Bitrix24"- installed on your server hosted by your company or hosting provider.

You can individually customize the business logic of the corporate portal, modify the interface, integrate with 1C:ZUP and much more.

Compare with "cloud"

Web cluster

Build your Web cluster - increase the performance, scalability and reliability of your portal!

A web cluster allows you to distribute one portal over several servers, thereby solving several problems: ensuring high availability of the portal; its scaling under conditions of increasing load; balancing load, traffic, data between several servers.

Virtual machine

“1C-Bitrix: Virtual Machine” is a free software product, a virtual server ready for immediate use, fully configured, tested and adapted for optimal work with both 1C-Bitrix products and any PHP applications.

With this product, you save time and money on deploying and configuring a corporate portal, on purchasing the necessary equipment, and on ensuring project security.

Controller for integration with an external site

A site controller is a fundamentally new technological solution, the task of which is to consolidate in a single place control over many completely independent web projects, built on separate copies of the 1C-Bitrix: Site Management product, regardless of their physical location.

Personal one-time password (OTP) generator

Using one-time passwords will significantly increase your security when working with a website or corporate portal. One-time passwords will not allow an attacker to gain access to your personal data on the site, even if your password is stolen or intercepted.

With Bitrix OTP, you can independently enable or disable the use of a one-time password system for your account on the portal.

Administration of the corporate portal

The boxed version of the service contains an administrative part that allows you to fully manage the structure of the corporate portal, configure its tools, manage access to the system, users, user groups and portal content.

Administrators can work directly from the public part of the portal. To do this, they just need to log in and enable editing mode in the Control Panel. In this mode it is even easier to perform all common operations, for example, adding

Training course: “Bitrix24 service administrator (boxed version)”

Content management (visual page editor)

The visual HTML editor is already built into the 1C-Bitrix24 product, and you do not need to install it. With this editor you can change your pages on the portal in real time - directly through your browser. The editor allows you not only to edit and format regular text, but also to work with visual components.

The visual editor built into the product is compatible with all popular browsers!

Advanced access rights management

The boxed version contains a powerful access control system, which contains several aspects:

access to modules;
access to dynamic content elements;
access to files and folders.

The system allows you to flexibly configure access for each employee within the User Group and this group itself within the entire corporate portal.

Design options and customization

The boxed version of the service comes in two standard design templates: Light and Bitrix24. These options satisfy the vast majority of users. However, some companies want to have their own corporate design.

The course is intended for users administering sites on the 1C-Bitrix: Site Management platform. Well Administrator. Modules allows you to master methods for performing advanced tasks for administering modules not related to commercial activities A separate course has been created for modules related to online trading. .

Gained skills:

methods of working with modules of the non-profit plan system;
working with special tools: Push&Pull, Messaging Service and others;
working with modules that allow users to communicate: forums, blogs, and others;
organization of document flow and business processes;
setting up Web analytics tools;
and others.

Minimum knowledge required to study the course:

basic computer literacy and Windows OS skills;
basic knowledge of WWW and organizing access to a web server;
Content manager We believe that you have already completed this course and know a lot about Bitrix. Therefore, there will be much fewer hints in pop-up windows than in the Content Manager course.
, so as not to get confused in the interface.
knowledge of the system within the course Administrator. Base We believe that you have already completed this course and know a lot about 1C-Bitrix administration. Therefore, there will be much less prompts in pop-up windows, as well as explanations about where and how to perform common administration tasks.

It would be nice to have basic skills in installing and administering *nix systems.

If this course is taken as the basis for offline teaching, then the recommended duration: 4 days (32 academic hours).

Save time and study on the way to work or university? This requires files. We provide two options for offline learning: files in PDF and CHM format, they open on any device. Reading programs are available in the AppStore and Play Market. The files are updated periodically, but will still lag behind the online version of the course. Therefore, we still recommend the online version of the courses, especially since mobile Internet is not a problem now.

The file is not located locally, but on a network resource.
If the file is on the local disk, but the path to it contains special characters (# and others).

How to take the training course?

The Web Cluster module is a combination of technological solutions that allow you to distribute one website across several servers, thereby solving several problems: ensuring high availability of the website; its scaling under conditions of increasing load; balancing load, traffic, data between several servers. Build your own Web cluster - increase the performance, scalability and reliability of your project!

Any new or running project on 1C-Bitrix: Site Management can be presented as a web cluster of interchangeable servers.
The main tasks that this project configuration allows you to solve:

As traffic increases, you can quickly add new servers to the cluster
If one of the cluster servers fails, the system continues to continuously serve Clients
Balancing load, traffic, data between multiple servers
The system allows you to take backup copies from specially designated cluster nodes without affecting the operation of the site

"Geographic web cluster"

“Geographic web cluster” increases the fault tolerance of the project and ensures independence from the data center. Several groups of web clusters located in different cities or countries are combined in different data centers. In the event of a failure of one data center, another is immediately brought into operation, without the need to restore a “backup”.

A geographic web cluster allows you to raise entire groups of servers. Each of these groups has its own master - in data centers independent from each other. Thus, your websites and your business are completely protected from the inaccessibility of the data centers themselves.
"1C-Bitrix: Web Cluster" is a combination of technologies:

Vertical sharding (moving modules to separate MySQL servers)
MySQL replication and load balancing between servers
Distributed data cache (memcached)
Continuity of sessions between web servers (storing sessions in a database)
Web server clustering:
File synchronization
Load balancing between servers
Independence from the data center (in the event of a failure of one data center, another immediately comes into operation, without the need to restore a “backup”)

How does it work

1. Vertical sharding

Dividing one web application database into two or more databases by separating separate modules, without changing the logic of the web application.

The following product modules can be placed in separate databases:

"Web analytics"
"Search"

2. MySQL replication and load balancing between servers

The master-slave scheme is implemented using MySQL.
The 1C-Bitrix: Site Management platform allows you to flexibly balance the load between servers participating in replication.

Key Features:

flexible SQL load balancing
ease of administration
cheap and fast unlimited scaling
online backup
no modification of web application logic required

3. Distributed data cache (memcached)

"1C-Bitrix: Web Cluster" allows you to use a pool of memcached servers to work with the data cache.

This provides:

high efficiency - due to centralized use of the cache by the web application
reliability - due to the caching subsystem’s resistance to failure of individual components
unlimited scalability - by adding new memcached servers

4. Continuity of sessions between web servers (storing sessions in a database)

The ability to store user session data in a database ensures session “transparency” for all web servers in the cluster:

After authorization on one of the servers, the user must be considered authorized for all other servers
And vice versa - the end of a session on any server should mean its end on all servers at once

5. Web server clustering

When dividing a project into several web servers, two problems must be solved:

synchronization of data (files) between servers
load balancing between servers