A big data predictions system using similarity method

Theoretical value

1. In this study, we study the applicability of various software for the collection and processing of big data. Two cases implemented in business and in the field of weather forecasting were investigated and analyzed.

2. The proposed method of weather forecasting by similarity method can help to improve the General time series forecasting systems with a further increase in the number of characteristics.

Practical significance

1. Based on the results of this work, you can build a weather forecast for the future short-term period of time or improve the current forecast algorithms by similarity using the proposed methodology.

2. The method under Study can potentially be useful for other areas of time series forecasting if there is the same data set in terms of properties and characteristics as for meteorological problems.

1.5 Thesis structure

The problem of definition, understanding and history of the term "big data" directly affects the possibility of using the methods and tools offered by this broad field in humanitarian research. Paradox: despite belonging to the exact Sciences, the digital environment and the IT sphere, the concept of "big data" ("big data") does not have a clear definition. Many authors, organizations, and communities try to interpret the concept in different ways. There are some variations of this term.

In June 2013 Oxford English dictionary (The Oxford English dictionary, OED) added a definition of the term "big data". “Computing (also with capital initials) data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges; (also) the branch of computing involving such data.”

There is one more interesting, the site www.lexico.com which focuses on current, relevant word meanings and practical usage, gives the following definition of "big data»: “Extremely large data sets that may be analysed computationally to reveal patterns, trends, and associations, especially relating to human behaviour and interactions.” The technological term in this interpretation is closely related to the socio-humanitarian component, i.e., in relation to the original definition, there is an evolution of the concept towards its humanization, and the content of the concept changes from "serious difficulties" to practical = opportunities for analysing human activity.

Researcher, entrepreneur and author of the blog whatsthebigdata.com Gil Press in the publication "a Very short history of big data" on the website Forbes.com the big data problem dates back to the appearance of the term "information explosion" in the Oxford English dictionary in 1941. Then there were the first attempts to quantify the growth rate of data volumes. Gil Press also records the first meaningful use of the term "big data" in October 1999 in the digital library of the Association for computer computing (ACM Digital Library) - in an article by NASA researchers on the problems of information visualization.

In addition, in 2001, Doug Laney published a seminal study for Meta Group that identified three key parameters of big data: volume, speed, and variety (the so-called three "V's" : Volume, Velocity, and Variety). Big data is characterized by its gigantic size (there are no exact definitions), high rates of new generation and influx, and heterogeneity and disorderliness. And these three "vvvs" are recognized by all experts in one form or another, which indicates an earlier appearance of the concept and its evolution.

Everyone knows that the essence of Big Data technologies is working with huge amounts of data (which, however, follows from the term itself). But the volume of data doesn't make the weather yet. Analysts have come up with a succinct formula for Big Data - they believe that the definition of a Big Data project should include seven important characteristics, "7 V": Volume, Velocity, Variety, Veracity, Variability, Visualization, Value. That is, the volume, velocity, variety, veracity, variability, visualization, value. At the same time, each "V" is important for understanding the overall picture.

The first three V's are the least problematic: Volume, Velocity, and Variety. Indeed, who would argue that Big Data is primarily volume? The volume of data is growing exponentially: for example, aircraft annually generate 2.5 billion TB of data from sensors installed in engines. At the same time, data is constantly updated, new ones are generated, and the speed of the update (Velocity - the second "V") is also important in order to consider it "large". For example, every minute in the world, almost 2.5 million requests are made to the Google search engine. The challenge of Big Data projects is to cope with the tremendous speed at which data is created and analyze it in real time.

The third " V " is Variety, variety. This means that Big Data projects must include data in a variety of formats: structured and unstructured data, text, graphics, corporate email or social media data, up to video. Each of these data types requires different types of analysis and appropriate tools. Social media can help brand owners analyze customer moods, and sensory data can provide information about how the product is most often used to apply this knowledge to improve it.

Until recently, three V's were enough. But everything changes, including approaches to definition. So analysts added four more "V's" to avoid misunderstandings. So, Veracity, Variability, Visualization, and Value were added to the definition. Let's look at each of these points.

Veracity-Confidence: of course, this characteristic is extremely important, because any analysis will be completely useless if the data turns out to be unreliable. Moreover, it is extremely important for us to make sure that everything is OK with the data, because their inaccuracy can lead to incorrect decisions. The simplest example is contacts with false names and inaccurate contact information.

Variability-Variability: a new trend in the field of Big Data. Here we are talking about the fact that the meaning of the same data may differ depending on the context, for example, the same words on Twitter may have different meanings and reflect different moods. We must consider all the nuances! In order to perform proper mood analysis, algorithms must be able to understand the context and be able to decipher the exact meaning of a word in that context.

Visualization-Visualization: this is a necessary part of analysis, because it is visualization that makes big data accessible to human perception. Visualization of large volumes of complex data is much more efficient and understandable for a person than spreadsheets and reports full of numbers and formulas. Of course, visualization within Big Data does not mean building regular graphs or pie charts: complex graphs may be built that include a lot of variable data, but they will still be clear and readable.

Value-Value: here we are talking about getting the most out of the results of big data analysis. What matters is how you use this data and whether you can turn your organization into an advanced company that relies on ideas derived from data analysis to make decisions.

However, even these seven "V's" are not enough to understand the essence of Big Data: we are talking about the fact that all these seven characteristics must be applied to a complex task, usually with several variables and a non-trivial condition. And a small conclusion in the end: of course, we could not pass Big Data with a clear conscience, so now Prognoz Platform is developing all the technologies necessary for working with big data: support for Hadoop, integration with software and hardware complexes, integration with SAP solutions[].

2.2 Analysis of modern methods of dataprocessing

2.2.1 Implementation in philip morris case

The atmospheric air of the Earth is in constant motion. We feel its movement in the wind, which carries heat from the equator to the poles and moisture from the sea to the land, where it falls in the form of life-giving rain.

The only source of energy that causes the movement of the atmosphere is the sun.

Uneven heating of the earth's surface, which in turn heats the air, creates a difference in atmospheric pressure. The cold air is denser, so it goes down and creates a high pressure area. Wind is the movement of air from high-pressure areas to low-pressure areas.

Thus, constantly formed a certain state of the atmosphere, called the weather. In turn, people are trying to monitor weather processes using meteorology and observe global climate change affecting the local weather in the regions.

Weather forecast -- a scientifically based assumption about the future state of weather at a particular point or region for a certain period. Compiled (developed) by public or commercial meteorological services on the basis of meteorological methods. Here are the various methods used in weather forecasting:

* extra-short-term -- up to 12 hours;

* short-term-- from 12 to 36 hours;

* medium-term-- from 36 hours to 10 days;

* long-term-- from 10 days to the season (3 months);

* extra-long-term -- more than 3 months (a year, several years).

At the end, having been acquainted with the basic knowledge on this topic, its needed to look at the methods on the basis of which the weather forecast is now carried out, as well as to find attempts by some enthusiasts to solve this problem.

3.2 Analysis of existing methods and their probability

In modern meteorology, it's possible to determine three large groups of weather forecasting:

· Synoptic weather forecasting,

· Numerical methods, and

· Statistical methods.

Synoptic weather forecasting

“Synoptic" means that the observation of different weather elements refers to a specific time of observation. Thus, a weather map that depicts atmospheric conditions at a given time is a synoptic chart to a meteorologist. In order to have an average view of the changing pattern of weather, a modern meteorological center prepares a series of synoptic charts every day.

Numerical methods

The numerical method involves a lot of mathematics. Modern weather forecasting is now using the techniques of Numerical Weather Prediction (NWP). This method is based on the fact that gases of the atmosphere follow a number of physical principles. A series of mathematical equations is used to develop theoretical models of the general circulation of the atmosphere. These equations are also used to specify changes in the atmosphere as the time passes on. However, even the most simplified mathematical models involve incredibly large number of calculations to be made. The use of such models could only be possible after the advent of high-speed electronic computers.

Statistical methods:

Statistical methods are used along with the numerical weather prediction. This method often supplements the numerical method. Statistical methods use the past records of weather data on the assumption that future will be a repetition of the past weather.

The main purpose of studying the past weather data is to find out those aspects of the weather that are good indicators of the future events. After establishing these relationships, correct data can be safely used to predict the future conditions.

3.3 Analysis current weather forecasting data sources and ways for getting data

big data similarity digitization

There are many different methods for predicting different metrics. The main and General task is to forecast time series. Time series forecasting assumes that data obtained in the past helps explain values in the future. It is important to understand that in some cases we are dealing with details that are not reflected in the accumulated data. Often, different approaches to forecasting combine to provide the most accurate forecasts, as is the case when forecasting weather conditions.

A time series is a collection of values obtained over a period of time, usually at regular intervals.

If you consider how the value changes from one period to another and how to forecast values, keep in mind that time series data has some important characteristics:

· The base level is usually defined as the average value of a time series. In some forecasting models, the baseline is usually defined as the initial value of the series data;

· A trend usually shows how time series change from one period to another;

· Season fluctuations. Some values tend to increase or decrease depending on certain time periods, such as a day of the week or a month in the year;

· Some prediction models include a fourth characteristic, noise, or error, which refers to random fluctuations and uneven movements in the data. We will not consider noise here.

Autoregressive integrated moving average, or ARIMA, is one of the most widely used forecasting methods for single-factor forecasting of time series data.The problem with ARIMA is that it doesn't support seasonal data. This is a time series with a repeating cycle.

ARIMA expects data that is not seasonal or the seasonal component is deleted, for example, it is seasonally adjusted using methods such as seasonal difference.

There are three trend elements that need to be adjusted. They are the same as the ARIMA model; in particular[17]:

· p: Order of trend autoregression.

· d: the Order in which the trend changes.

· q: is the Trend moving average.

· Seasonal elements

There are four seasonal elements that are not part of ARIMA that must be configured; they are:

· P: Seasonal autoregression order.

· D: Order of seasonal differences.

· Q: Seasonal order of moving averages.

· m: the Number of time steps for a single seasonal period.

On the Figure is result of the auto_arima process.

Figure 20. Build model autoarim, which automatically selects the parameters of the model.

Figure 21. Build model with known parameters.

Figure 22. Final prediction for the next month by the model.

Thus, the forecast of a non-stationary series using the seasonal auto-rate method demonstrated a different method of using data loaded for two years.

4.Chapter four: similarity method

It is the kind of statistical predicting methods. The characteristics of any phenomenon in a group of similar can be obtained using similarity theory.

To facilitate the search for similarity from a huge array of weather data in the first stage, it is advisable to highlight some individual features. To study the characteristics of the weather hypothesis is accepted:

- deterministic component of weather conditions on average for the period;

- random nature of weather changes over a shorter period of time.

Its allows the processing of statistical information using the mathematical apparatus of probability theory.

Statistical characteristics of a random process are sufficiently fully reflected by the following indicators:

· mathematical expectation (average value of weather parameter Xi for n observations):

standard deviation (variance):

autocorrelation function, reflecting the rate of change in the process (in this case - weather changes):

· mutual correlation function (the covariance), showing the relationship of the phenomena (in this case, various parameters of the weather - the night and daytime temperatures, the pressure of the atmosphere, etc.):

Where У_{N-T -} another observed through the interval T setting the weather.

From the given properties and the information available to the authors to reduce the cost of finding analogues in a large amount of weather data, it is advisable to use one of the parameters of the first day of the weekly forecast.

4.1.1 The hyposesis about the weather as random function

The main position of probability theory (Central limit theorem) - the sum of a large number of causal factors gives a normal distribution.

The properties of a random variable are described by the integral distribution function F(x), which reflects the probability that the i observation will be less than a certain value of X (figure 22a), and more clearly by the differential function P(x) - the distribution density characterizing the probability (frequency of manifestation) of Xi values (figure 22B).

In measurements, the normal distribution law most often takes place:

where is the expected value (mean):

· dispersion:

· standard deviation(SD):

Root extraction is a non-linear procedure, and to eliminate the bias of the estimate, the result is (4. 5) multiply by the correction factor depending on n (for n=3...7 > Kn=1.13...1.04).

Figure 23. Integral (a) and differential (b) functions distributions of a random variable.

The Central limit theorem - the distribution of random errors will be close to normal whenever the results of observations are formed under the influence of a large number of independent factors.

The result of the evaluation of statistical characteristics depends on the number of measurements and differs from the hypothetical value of the population. The estimation of the SSR is characterized by an error that also depends on the set of statistical data (for n=3...15 > Ду=50...15%). For the investigated parameter with unknown ME(eng)=MO(rus) and SD(eng) their assessment is made by results of the made measurements and dependence on quantity of these measurements :

In practice, even with high requirements for the accuracy of estimates make 30...50 measurements. The result of multiple measurements can be interpreted as an increase in the confidence probability of the error interval.

ME and SD fully characterize the random function with normal distribution law.

There are methods to test the hypothesis:

· on the normal law of distribution;

· uniformity of observations;

· emissions (miss) of observations.

The probability of getting the result of observation during measurements in the range of values X1-X2 is equal to the area bounded by the distribution curve and the specified interval:

4.1.2 The similarity of the processes

Characteristics of any phenomenon in a group of similar can be obtained by similarity theory.

Similarity is a one-to-one correspondence between two processes, in which the transition function of the parameters characterizing one of them to other parameters is known, and the mathematical description can be transformed into the identical one. This means that the phenomenon under investigation can be obtained by similarly given by such a transformation, when the size of each of its magnitude changes a certain number of times (such a transformation of phenomena).

Deterministic criteria reflect physical similarity in functions, parameters, processes.

Stochastic similarity criteria in a generalized way reflect the presence of random factors, the variation of parameters, etc.

Fuzzy similarity - in the absence of numerical accurate estimates and the use of semantic evaluations of phenomena. The limit of similarity is the identity of phenomena, which can be estimated by simple comparison. However, to process large amounts of data in long-term forecasting, it is necessary to formalize the procedure for determining the similarity.

4.3 The use of covariance and algorythm of fuzzification

In General, the process of implementing this method is not complicated. Initially, you need to get a key to connect to the Meteostat API and select the necessary criteria for requesting a JSON file.

Figure 28. Getting Data through the API.

The next step in processing the received data is to read the daily temperatures for the selected period and divide it into lists of 7 elements, as many as are in one week.

Figure 29. Creation of Week list array.

To begin with, a pre-forecast week was introduced, which had a mathematical expectation calculated.The similarity coefficient described is the Pearson correlation coefficient. Which can be calculated using the pearsonr function.

Figure 30. Pearson correlation coefficient for each week.

After calculating the coefficient for each week, we conduct a comparative analysis. As the first rule of fuzzification is considered to be a necessary condition for a value in excess of 0.625. In the current selection, we have only one value that fits this rule. K = 0.82.

Figure 31. Graph of Kxy correlation between pre-forecast week and its historical analog

By transferring the values of both weeks to the format of two-dimensional coordinates and building a Kxy line, we can calculate the angle of inclination of this line. Arctg(ц) = 1.1, then the angle is ц = 47.7. then it's needed to look through fully apply the fuzzification rule.

Figure 32. Results of final fuzzification rules.

So, finally we have : the Kxy is “Strong” (0.82) and цХУ(47.2) “Average” THEN Similarity “Full”.

In the course of these manipulations, we have identified the most similar analog to the pre-forecast period, which means that its subsequent week can be considered as similar to the predicted week. Therefore, for the elementary calculation of approximate predicted values, based only on a single measurement of air temperature, the difference of two weeks was calculated piecemeal and the result was transferred to the historical analog. As a result, the following results were obtained:

Figure33. Predicted values of weather for searched week.

6.Chapter six : conclusion

In the course of current research, we have studied and applied the method of weather forecasting using the similarity method. We analyzed cases from two different areas of public life to understand the principle of choosing software in a particular case. Moreover, an example of one of the modern methods of forecasting time series using SARIMA was given. Further tactics to improve this method were suggested.