Homework2 Sol

CS412 Assignment 2 Ref Answer Question 1: Assume a base cuboid of 10 dimensions contains only three base cells: (1) (a1, b2, c3, d4; ..., d9, d10), (2) (a1, c2, b3, d4, ..., d9, d10), and (3) (b1, c2, b3, d4, ..., d9, d10), where a_i != b_i, b_i != c_i, etc. The measure of the cube is count. 1, How many nonempty cuboids will a full data cube contain? Answer: 210 = 1024 2, How many nonempty aggregate (i.e., non-base) cells will a full cube contain? Answer: There will be 3 ∗ 210 − 6 ∗ 27 − 3 = 2301 nonempty aggregate cells in the full cube. The number of cells overlapping twice is 27 while the number of cells overlapping once is 4 ∗ 27 . So the final calculation is 3 ∗ 210 − 2 ∗ 27 − 1 ∗ 4 ∗ 27 − 3, which yields the result. 3, How many nonempty aggregate cells will an iceberg cube contain if the condition of the 4, iceberg cube is "count >= 2"? Answer: There are in total 5 ∗ 27 = 640 nonempty aggregate cells in the iceberg cube. To calculate the result: fix the first three dimensions as (***), (a1**), (*c1*), (**b3) or (*c1b3), and vary the rest seven ones. 4, How many closed cells are in the full cube? Answer: There’re 6 closed cells in the full cube: 3 base cells; (a1, *, *, d4, …, d10); (*, c2, b3, d4, …, d10) : count 2; (*, *, *, d4, .., d10): count 3. Question 2: (Half open questions, make sure your algorithm and assumptions are correct, no need to be very specific) Suppose a base cuboid has the following tuples: A B C D Count Sales a1 b1 c1 d1 1

6

a1 b2 c2 d1 1

4

a1 b3 c1 d2 1

2

a2 b4 c1 d2 1

10

a2 b3 c2 d3 1

12

1, Show the representative steps to demonstrate how a complete data cube (with Count and SUM(Sales) as measures) is computed by the multiway array aggregation algorithm; Answer (from fang2): Suppose dimensions A, B, C, D are organized into 2, 4, 2, 3 partitions respectively. So in total there are 2*4*2*3 = 48 chunks. The cardinality of dimensions A, B, C, D is 2, 4, 2, 3 respectively, i.e. A and C have the smallest size, followed by D, and lastly B has the largest sieze). From the base cuboid given, we can compute 3D-, 2D-, 1D- and apex cuboids as in the diagram. The chunk scan order is always first along the smallest dimension, then along 2nd smallest dimension, then along 3rd smallest dimension, and so on. For example, when computing

3D-cuboids from the base cuboid, we first scan chunks along the A dimension, then C, D and B, in this ascending order of the size of the dimension. In other words, we aggregates first towards CDB, so only 1 chunk of CDB needs to be held in memory at any one time; then aggregates towards ADB, so only 1 row of ADB needs to be held in memory at any one time, so on and so forth. For computation of 2D-, 1D- and apex cuboids, a similar approach is adopted, where the chunks are scanned first along the smallest dimensions. During computation of a cuboid, both measures (count and sales) are aggregated.

2, Do the same using the BUC algorithm; and Answer (from duan9): First we order the dimensions in descending order by cardinality: BDAC. Then we have the aggregation order in the tree form:

At the beginning of the recursion, we aggregate all the dimensions to get the apex cuboid using the two measures: count and sum of sales. Then we start partitioning the table according to the sequence BDAC as follows:

Through this recursive aggregation and partition process we get the following cuboids: apex, B, BD, BDA, BDAC. Then we traverse back (as part of the recursion) and get BDC, and traverse back further we get BA, BAC and so on. 3, Do the same using the Star-Cubing algorithm. Answer (from duan9): First we order the dimensions as we did in BUC: BDAC. Based on the order, we have the following computation ordering:

Then we construct a Star-Tree for the base table. Since we are actually computing the full cube, there is no star on the star tree. Similarly, there will be no compressed table (or you can do your own assumption and build your own compressed table).

Then we start the aggregation process by looking at a branch of the Star-Tree each time and aggregate simultaneously to four descendant cuboids (with shared dimensions). After the first branch is processed, we get the following four trees:

Then the second branch is processed (traverse back after reach the leaf node of the tree). Since it’s a completely different branch from the first, all the four trees formed in the first step will be output and destroyed. Four new trees will be formed in a similar way. This is traverse will keep going until all the tree nodes are traversed. Then we use the same approach to build BD/BD from BDC/BD, BA/BA from BAC/B and so on, so forth. Finally we get all the cuboids for the full cube. Question 3: (Open questions, following are some possible answers) Consider the three data cube computation algorithms exercised in Question 2, discussed the following: 1, For different skewness of data, discuss the relative computation efficiency of the above algorithms in very large datasets; Answer (from luu1): Multi-way array aggregation: Computation efficiency will be higher with less skewed data. If the underlying data is extremely skewed, some chunks may be too big to fit into the memory (i.e. the dense data). Also, the shared aggregate computation will be done over empty cells in the non-dense part of the data, which is inefficient. BUC: Similarly, computation efficiency will be higher with less skewed data, as evenly distributed data provides greater opportunity for pruning. Star –cube: Star-cubing is robust against skewed data because star-tree is generated only based on the existing tuples and the tree generation or the aggregation process does not depend on the distribution of the data. 2, If the cube is sparse, one may want to compute iceberg cube only (e.g., compute only those cells whose support is more than one), for different skewness of data, discuss the relative computation efficiency of the above algorithms in very large datasets. Answer (from luu1): Multiway array aggregation: Although the chunking technique may be able to compress sparse data array by utilizing direct index accessing mechanism, if the data are very sparse, a large

number of chunks will need to be generated for relatively small amount of meaningful data. Also, sparse data means that the shared aggregate computation will be done over empty cells in many cases, which is inefficient BUC: Sparse data will generate huge number of partitions where many of them will be pruned. However, large number of partition adds computational overhead during recursive computation. Star-cubing: Star-cubing compresses data into star-tree structure, removing the redundant empty cells (or those cells below the water) from aggregation process. Therefore it’s robust against sparse data. 3, The base cuboid in Question 2 has 4 dimensions. Suppose a base cuboid has 100 dimensions (D1, D2, ..., D100), discuss how the high dimensional OLAP can be computed. Let the following dimensions be grouped together in shell fragment cube computation: (Dm, D(m+20), D(m+40), D(m+60), D(m+80)) for m = 1, ..., 20. Discuss how the query (a1, ?, ?, * , c5, * , ..., * ) can be computed efficiently, where ? means the inquired variable and "*" means "do not care" variable. Answer (from fang2): As given, the dimensions are grouped as follows in shell fragment computation: (D1, D21, D41, D61, D81), (D2, D22, D42, D62, D82), … (D20, D40, D60, D80, D100). For the sub-cube query (a1, ?, ?, * , c5, *, ..., * ), D1 and D5 are instantiated dimensions, and D2 and D3 are inquired dimensions. All the other dimensions are irrelevant (“don’t care”). We first identify each relevant dimension in their shell fragment, and use the computed cuboids in the shell fragment to obtain TIDlists. This is illustrated in the table below. The TIDlists are constructed randomly for illustration purpose.

Next, we need to intersect the TIDlists to answer the subcube query. We should first intersect the TIDlists of the instantiated dimensions, as this greatly reduces the size of the intersection. So for D1 and D5, we get a reduced TIDlist (a1,c5): {2,3,5}. Then we further intersect this with D2 and D3 using a top-down depth-first strategy, discarding any empty intersections during the process. After which, we obtain the following TIDlists: {(b1,d1):{2}, (b2,d2):{5}, (b2,d3):{3}}1. These TIDlists can be used to construct a 2D cuboid on dimensions D2 and D3. Such a constructed “base-cuboid” can then be used to compute the 2D cube for the two inquired dimensions trivially, which answers the sub-cube query.

Question 4. A database has 5 transactions. Let min sup = 60% and min conf = 80%. TID items bought T100 M, O, N, K, E, Y T200 D, O, N, K, E, Y T300 M, A, K, E T400 M, U, C, K, Y T500 C, O, O, K, I, E

1, Find all frequent itemsets using Apriori and FP-growth, respectively. Compare the efficiency of the two mining processes. Answer (from luu1): Apriori:

FP-Growth:

- About data scan: Apriori needs to scan the database repeatedly to accumulate a k-item support and check the frequency. On the other hand, FP-growth algorithm needs 2 scan, once to identify frequent1-item set and second to build FP-tree. - About candidate generation: Apriori algorithm generates exponential number of candidate set and the self-join process of candidate generation itself is expensive. FP-growth algorithm does not generate any candidate set. 2, List all the association rules (with support s and confidence c) matching the following metarule, where X is a variable representing customers, and itemi denotes variables representing items (e.g., "A", "B", etc.): for all x in transaction; buys(X, item1) and buys(X, item2) -> buys(X, item3) [ s, c ] Answer (from luu1): ‐ Buys(X, E) and buys (X, O) -> buys(X,K) [60%, 100%] ‐ Buys(X, O) and buys(X, K) -> buys(X, E) [60%, 100%]