Several Methods for Structured Big Data Computation

All data can become valuable only by getting involved in computing and analytics. The big data makes no exception. The computational capability on structured big data determines the range of practical applications of big data. In this article, I'd like to introduce several computation methods: API, Script, SQL, and SQL-like languages.

API: The "API" here refers to a self-contained API access method without using JDBC or ODBC. Let's take MapReduce as an example. MapReduce is designed to handle the parallel computation cost-effectively from the very bottom layer. So, MapReduce offers superior scale-out, hot-swap, and cost efficiency. MapReduce is one of the Hadoop components with open-source code and abundant resources.
Sample code:
public void reduce(Text key, Iterator<Text> value,
            OutputCollector<Text, Text> output, Reporter arg3)
            throws IOException {
          double avgX=0;
          double avgY=0;
          double sumX=0;
          double sumY=0;
          int count=0;
          String [] strValue = null;
          while(value.hasNext()){
            count++;
            strValue = value.next().toString().split("\t");
            sumX = sumX + Integer.parseInt(strValue[1]);
            sumY = sumY + Integer.parseInt(strValue[1]);
          }

          avgX = sumX/count;
          avgY = sumY/count;
          tKey.set("K"+key.toString().substring(1,2));
          tValue.set(avgX + "\t" + avgY);
          output.collect(tKey, tValue);
        }
Since the universal programming language adopted is unsuitable for the specialized data computing, MapReduce is less capable than SQL and other specialized computation languages in computing. Plus, it is inefficient in developing. No wonder that the programmers generally complain it is "painful". In addition, the rigid framework of MapReduce results in the relatively poorer performance.
There are several products using API, and MapReduce is the most typical one among them.

Script: The "Script" here refers to the specialized script for computing. Take esProc as an example. esProc is designed to improve the computational capability of Hadoop. So, in addition to the inexpensive scale-out, it also offers the high performance, great computational capability, and convenient computation between heterogeneous data sources, especially ideal for achieving the complex computational goal. In addition, it is the grid-style script characterized with the high development efficiency and complete debug functions.
Sample code:

Java users can invoke the result from esProc via JDBC, but they are only allowed to invoke the result in the form of stored procedure instead of any SQL statement. Plus, esProc is not open source. These are two disadvantages of esProc.

The Script is widespread used in Mongo DB, Redis, and many other big data solutions, but they are not specialized enough in computing. For another example, the multi-table joining operation for Mongo DB is not only inefficient, but also involves the coding of one order of magnitude more complex than that of SQL or esProc.

SQL: The "SQL" here refers to the complete and whole SQL/SP, i.e. ANSI 2000 and its super set. Take Greenplum as an example, the major advantages of Greenplum SQL are the powerful computing, highly efficient developing, and great performance. Other advantages include the widespread use of its language, low learning cost, simple maintenance, and migration possibility -not to mention its trump-card of offering support for stored procedure to handle the complex computation. By this way, business value can be exploited from the big data conveniently.
Sample code:
  CREATE OR REPLACE function view.merge_emp()
  returns voidas$$
  BEGIN
truncate view.updated_record;
insert into view.updated_record select y.* from view.emp_edw x right outer join emp_src y on x.empid=y.empid where x.empid is not null;
update view.emp_edwset deptno=y.deptno,sal=y.sal from view.updated_record y
where view.emp_edw.empid=y.empid;
insert into emp_edwselect y.* from emp_edw x right outer join emp_src y on
  x.empid=y.empid where  x.empid is null;
end;
$$ language 'plpgsql';

The other databases with the similar structure to MPP include Teradata, Vertical, Oracle, and IBM. Their syntax characteristics are mostly alike. The disadvantages are similar. The acquisition cost and the ongoing maintenance expenses are extremely high. Charging its users by data scale, the so-called inexpensive Greenplum is actually not a bargain at all - it is way more like making big money under cover of big data. Other disadvantages include awkward debugging, incompatible syntax, lengthy down-time if expansion, and awkward multi-data-source computation.

SQL-like language: It refers to the output interfaces like JDBC/ODBC and only limited to those scripting languages that are the subset of standard SQL. Take Hive QL as an example. The greatest advantage of Hive QL is its ability to scale out cost-effectively while still a convenient tool for users to develop. The SQL syntax feature is kept in Hive QL, so that the learning cost is low, development efficient, and maintenance simple. In addition, Hive is a component of Hadoop. The open-source is another advantage.
Sample code:

SELECT e.* FROM (
SELECT name, salary, deductions["Federal Taxes"] as ded, 
salary * (1 – deductions["Federal Taxes"]) as salary_minus_fed_taxes 
FROM employees
) e 
WHERE round(e.salary_minus_fed_taxes) > 70000;

The weak point of Hive QL is its non-support for stored procedure. Due to this, it is difficult for HiveQL to undertake the complex computation, and thus difficult to provide the truly valuable result. The slightly more complex computation will rely on MapReduce. Needless to say, the development efficiency is low. The poor performance and the threshold time can be regarded as a bane, especially in task allocation, multi-table joining, inter-row computation, multi-level query, and ordered grouping, as well as implementing other algorithm alike. So, it is quite difficult for HiveQL to implement the real-time Hadoop application for big data.
There are also some other products with SQL-like languages - MongoDB as an example - they are still worse than Hive yet.

The big data computation methods currently available are no more than these 4 types of API, Script, SQL, and SQL-like languages. Wish they would develop steadfastly and there would be more and more cost-effective, powerful, practical and usable products for data computing.

A Rich Class Library is the Basis to Speed Big Data Computing and Developing

In developing the big data application, framework plays a very important role. It hides the complex parallel logics and guarantees the system reliability and stability. So programmers can be more focused on the business algorithm.

Important as framework is, it doesn't take us much time to develop, while implementing basic algorithms would cost programmers great efforts. For example, to complete the simplest algorithm of grouping and summing up in MapReduce, the code of Reduce part is shown below:

Even the simplest algorithm as such requires so many codes to implement. We can imagine that the normal algorithms would require definitely much lengthy code. Take the normal problem of finding the top N for example. A part of the MapReduce code is as follows:


There are still some algorithms, join operation for example, which are frequently used but extremely difficult to implement with MapReduce. To implement the join in MapReduce, users will usually have to inherit or implement the Partitioner, Writable Comparator, and Writable Comparable,in addition to the Map and Reduce implementation. The codes are too lengthy to list here.
In addition to the grouping and summing up, top N, and join, there are lots of basic algorithms, for example,filtering, distinct, intersection set, sorting, ranking, yearly link relative ratio, year-on-year comparison, relative position computing, and interval computing. As can be imaged, to implement the real business logics, we must combine these basic algorithms in actual use, resulting in a project of great workload.
Hive and other SQL-like means have been tried to package these basic algorithms in Hadoop. However, the functions available in Hive are far from rich. Users usually have to resort to MapReduce or customized class to implement it. So, theyare unable to improve the development efficiency significantly. For example, to make statistics on the top 10 best sellers for each department with Hive, the below Java code need implementing:


In addition, Hive lacks the overall flow control and the statement for judging and looping. It is weak for data traversal, let alone the business algorithms involving multiple steps or complex logics. The similar algorithms must be implemented through combined uses with MapReduce.
MapReduce plays a vital role in Hadoop development. However, MapReduce lacks the structured data functions, making it hard to speed the development of big data application. To reduce the workload, we must package these basic algorithms into function. The rich library functions will definitely boost the efficiency of big data computing.

esProc is the parallel computing architecture specially optimized for the small and medium sized cluster, featuring the rich library functions. Still the above three examples, the codes of esProc solution are respectively shown below:
Grouping and summarizing: sales.groups(empID;sum(amount))
Top N: counts.top(keyword;10)
Top 10 best sellers in each department: products.group(department). (~. top(quantity;10)

As can be seen, regarding the big chunk of MapReduce code, esProc users only need a few library functions like groups, sum, top, and group to substitute it and the development efficiency would increase remarkably.

To meet the challenge of big data computing, esProc meticulously designed two kinds of complete and practical library functions: The cursor function for the high fault tolerance computing in external storage, and the TSeq function for the high performance computing in memory. These two types of functions can be used together to dramatically reduce the workload on big data development.

Cursor function
Cursor is the specific data type for big data computation. Confronting the big data in the database table or files in the external storage, users can use the cursor to retrieve a small amount of data in batches, and complete the computing over a batch of data in the memory. Once this in-memory computation is completed, go proceed to loop and retrieve the data of the next batch till the computation over all data is completed. The typical cursor functions include: cursor for creating, fetch for fetching, skip for skipping, mergex for merging,joinx for joining, groupx for grouping, and sortx for sorting.

For example, when summarizing by the field amount on the big data file sales.dat, the computing job are to be allocated to multiple node machines. Each node machine will respectively compute one piece of the computing file, and return the result to the summary machine. In which, the data allocated to the node machine may be still too many to be loaded into the memory all at once. In this case, cursors can be used to retrieve the data in batches, with 100,000 pieces for each time of computing. The code for the node machine is shown below:
 
Multiple cursors can be consolidated into a single cursor. Multiple sorted cursors can be merged with mergex, the join for the join operation, and the conj@x for the union operation. In which, the code for merging cursors  is shown below:
 
The join computation can be conducted between multiple cursors, in which case, you can use function joinx. For the big data grouping and summarizing, the result of summarizing might be too great to be loaded to the memory. In this case, cursor is required to present the result, and function groupx can be used for such kind of grouping. The big data must be sorted in the external storage, using the sortx function to return a cursor directly.

TSeq function
The cursor can retrieve the data in batches and then compute, so that the low-efficient computing in external storage can be converted to the high-performance computing in memory. The structured data in memory is just the TSeq.
TSeq is generic and sorted, especially fit for the complex computing related to the orders. Moreover, TSeq inherits the concept of table of the database. Users can also use fields and records to access the data, which is quite ideal and convenient for the computing over structured data. In the above example, sales, counts, and products are all TSeq, while groups, max, and group are TSeq functions. In addition, if summarizing on sales.dat, sum function is also the TSeq function. Then, let’s demonstrate it with more examples:

To search the goods with the lowest total price, we can use minp like this: A1.minp(price*quantity).
To find the teams whose goal difference is greater than 30, we can use select like this: A1.select(F-A>30)。

Firstly, filter out the teams whose goal difference is greater than 0. Then, of these teams, select the ones with the greatest points. In this case, we can use maxif, which is equivalent to A.select().max(), that is,A1.maxif(W*3+D;F>A).

In addition to the direct record filtering, we can also only filter the serial numbers of records, using pos, pmin, pmax, pslect, and other functions alike. For example: To find serial numbers of male employees whose initials are C, just write A1.pselect@a( Gender:"M",left(Name,1):"C")

esProc library functions are rich and diversified. Besides the cursor function and the TSeq functions, there are also functions for the database, remote files, parallel dispatching, mathematics, character strings, time, operating on sets, aggregating, loop computation, positioning, screening, and associating. With the various combined uses of these library functions, the development efficiency over big data computing is boosted dramatically.