Data mining. Textbook. Vadim Shmal

p>Data mining

      Textbook

      Vadim Shmal

      Pavel Minakov

      Sergey Pavlov

      © Vadim Shmal, 2022

      © Pavel Minakov, 2022

      © Sergey Pavlov, 2022

      ISBN 978-5-0059-4479-5

      Created with Ridero smart publishing system

      Data mining

      Data mining is the process of extracting and discovering patterns in large datasets using methods at the interface of machine learning, statistics, and database systems, especially databases containing large numerical values. This includes searching large amounts of information for statistically significant patterns using complex mathematical algorithms. Collected variables include the value of the input data, the confidence level and frequency of the hypothesis, and the probability of finding a random sample. It also includes optimizing the parameters to get the best pattern or result, adjusting the input based on some facts to improve the final result. These parameters include parameters for statistical means such as sample sizes, as well as statistical measures such as error rate and statistical significance.

      The ideal scenario for data mining is that the parameters are in order, which provides the best statistical results with the most likely success values. In this ideal scenario, data mining takes place within a closed mathematical system that collects all inputs to the system and produces the most likely outcome. In fact, the ideal scenario is rarely found in real systems. For example, in real life this does not happen when engineering estimates for a real design project are received. Instead, many factors are used to calculate the best measure of success, such as project parameters and the current difficulty of bringing the project to the project specifications, and these parameters are constantly changing as the project progresses. While they may be useful in certain situations, such as the development of specific products, their values should be subject to constant re-evaluation depending on the current conditions of the project. In fact, the best data analysis happens in a complex mathematical structure of problems with many variables and many constraints, and not in a closed mathematical system with only a few variables and a closed mathematical structure.

      Data is often collected from many different sources and several different directions. Each type of data is analyzed and all of that output is analyzed to get an estimate of how each piece of data may or may not be involved in the final result. Such analysis is often referred to as the analysis process or data analysis. Data analysis also includes identifying other important information about the database that may or may not have a direct impact on the results. Often, they are also generated from different sources.

      Data is usually collected from many different sources and many statistical methods are applied to obtain the best statistical results. The results of these methods are often referred to as statistical properties or parameters, and often define mathematical formulas that are intended for the results of each mathematical model. Mathematical formulas are often the most important aspects of the data analysis process and are usually structured using mathematical formulas known as algorithms. Some mathematical algorithms are based on some theoretical approach or model. Other mathematical algorithms use logic and logical proofs as mathematical tools to understand data. Other mathematical algorithms often use computational procedures such as mathematical modeling and mathematical tools to understand a particular problem or data. While such computational procedures may be necessary to complete a mathematical model of the data, such mathematical algorithms may have other mathematical tools that may be more appropriate for the real world. Although these mathematical models are often very complex, it is often easier to develop a mathematical algorithm and model from a mathematical model than to develop a mathematical algorithm and model from an actual data analysis process.

      In reality, there are usually a number of mathematical models that provide a more complete understanding of the situation and data than any one mathematical model or mathematical algorithm. The data is then analyzed and a mathematical model of the data is often used to derive a specific parameter value. This parameter value is usually determined by numerical calculations. If a parameter does not have a direct relationship with the result of the final analysis, the parameter is sometimes calculated indirectly using a statistical procedure that yields a parameter that has a direct correlation with the result of the data analysis. If a parameter has a direct correlation with the result of the data analysis, this parameter is often used directly to obtain the final result of the analysis. If the parameter is not directly related to the result of the analysis, the parameter is often obtained indirectly using a mathematical algorithm or model. For example, if data analysis can be described by a mathematical model, then a parameter can be obtained indirectly using a mathematical algorithm or model. It is usually easier to get the parameter directly or indirectly using a mathematical algorithm or model.

      By collecting and analyzing many different kinds of data, and performing mathematical analysis on the data, the data can be analyzed and statistics and other statistical tools can be used to produce results. In many cases, the use of numerical calculations to obtain real data can be very effective. However, this process usually requires real-world testing before data analysis.

      Agent mining

      Agent -based mining is an interdisciplinary field that combines multi-agent systems with data mining and machine learning to solve business problems and solve problems in science.

      Agents can be described as decentralized computing systems that have both computing and communication capabilities. Agents are modeled based on data processing and information gathering algorithms such as «agent problem» which is a machine learning technique that tries to find solutions to business problems without any data center.

      Agents are like distributed computers where users share computing resources with each other. This allows agents to exchange payloads and process data in parallel, effectively speeding up processing and allowing agents to complete their tasks faster.

      A common use of agents is data processing and communication, such as the task of searching and analyzing large amounts of data from multiple sources for specific patterns. Agents are especially efficient because they don’t have a centralized server to keep track of their activities.

      Currently, there are two technologies in this area that provide the same functionality as agents, but only one of them is widely used: distributed computing, which is CPU-based and often uses centralized servers to store information; and local computing, which is typically based on local devices such as a laptop or mobile phone, with users sharing information with each other.

      Anomaly detection

      In data analysis, anomaly detection (also outlier detection) is the identification of rare elements, events, or observations that are suspicious because they differ significantly from most of the data. One application of anomaly detection is in security or business intelligence as a way to determine the unique conditions of a normal or observable distribution. Anomalous distributions differ from the mean in three ways. First, they can be correlated with previous values; second, there is a constant rate of change (otherwise they are an outlier); and third, they have zero mean. The regular distribution is the normal distribution. Anomalies in the data can be detected by measuring the mean and dividing by the value of the mean. Because there is no theoretical upper limit on the number of occurrences in a dataset, these multiples are counted and represent items that have deviations from the mean, although they do not necessarily represent a true anomaly.

      Data Anomaly Similarities

      The concept of anomaly can be described as a data value that differs significantly from the mean distribution. But the description of anomalies is also quite general. Any number of outliers can occur in a dataset if there is a difference between observed relationships or proportions. This concept is best known for observing relationships. They are averaged to obtain a distribution. The similarity of the observed ratio or proportion is much less than the anomaly. Anomalies are not necessarily rare. Even when the observations are more similar than the expected