Heaps for Priority Queues and O(N Log2 N) Sorting We will next study heaps (today) and AVL trees (later). Both of these are special kinds of trees that have interesting order/structure properties, different from binary search trees. There are similarities when adding/removing values from these trees, and then restoring these properties. Heaps (this term has another important use in Computer Science: memory used for allocating the storage for objects) are almost the perfect data structure for storing priority queues. By using heaps we can add a value in O(Log2 N) and also remove the highest priority value in O(Log2 N): heaps always stay perfectly balanced, unlike BSTs. In the discussion below I will characterize Min-Heaps and the algorithms that process them. We would use Max-Heaps for storing priority queues: their operations follow almost identical rules. Order and Structure Properties: Recall that BSTs had a special order property (FOR EVERY NODE, only smaller values are in its left subtree and only larger values are in its right subtree) and no structure property (trees can take on any shape, from well-balanced to pathological). Min-Heaps have a special order and structure property: Order: Every node must be less than all nodes in its left and right subtrees. This is the same as saying every node must be less than its two children. Which is the same as saying every node must be greater than its parent. Structure: All depths must be filled, except possibly the deepest. If the deepest is not filled, then all the nodes at that depth must appear as far to the left as possible. This property ensures that every heap with N values looks the same as every other heap with N values, only the values inside the nodes can be different (like linked lists, but unlike BSTs). The Heap part of the Special Trees web page reading illustrates such trees. We will now review the algorithms to add and remove-the-minimum values from a heap (and how to restore the heap properties for both). These are both illustrated in the web reading. Then we will learn how to store heaps compactly in arrays. Finally we will learn an "offline" algorithm to build a heap with N values in O(N) time. Insertion: To add a value to a min-heap, first we must place it in a new node in the tree according to the heap structure property, so we put it to the right of the last node at the deepest depth (or if that depth is filled, as the leftmost child at one greater depth). By adding this node, we now have a tree (with one more node) that satisifes the heap structure property, but not necessarily the order property. The value may violate the order property (be smaller than its parent: in fact it may be very small and belong near the top of the heap). To restore the order property, we compare that node with its parent: we stop if they are in the right order (parent smaller than child); if they are in the wrong order (child smaller than parent) we swap them; note that putting a smaller value in the parent will keep the order property in the other subtree of the parent (they were all bigger than the original parent, which has now been replaced with yet a smaller value, so all the values in the other subtree are all still bigger than their new parent). Then we compare the parent (which now has a new value) with its parent and again swap if necessary. We continue comparing and swapping until (a) the value settles into its correct place: we find a parent less than the child, or (b) the added value is now at the root of the tree, where there is no parent to compare/swap with. The tree will then satisfy the heap order property. We call this the "peroclate up" operation. Because of the heap structure property, heaps are perfectly balanced trees, so the height of a tree is at most O(Log2 N) and therefore the add operation is O(Log2 N): comparing/swapping the leaf all the way to the root in the worst case. Deletion of the Mininum Value: We can also remove the smallest value in a Min-Heap efficiently. To do so we save the value from the root (it is guaranteed to be the smallest in the tree by the order property), which will be returned by the "remove" method. Then we take the farthest right node at the deepest depth and remove it from the tree, but first place its value at the root (we have already saved the value from the root, for returning from the method). By removing the node at the bottom of the tree, we now have a tree that still satisifes the heap structure property, but with one fewer value. It is very likely that the value now at the top doesn't belong there; we took it from the bottom, which tends to have the biggest values, yet the smallest value belongs at the root of a heap. To restore the order property, if the root is bigger than either/both of its children, we swap it with its smallest child. Again, if the left/right child is the smaller value, then as a parent it will still be smaller than all the nodes in the right/left subtrees. Note that in doing this comparing/swapping, we might just compare the parent with its left child, or maybe both children. Because of the structure property of heaps, a node will NEVER have JUST A RIGHT CHILD. We continue the process until the value settles into its correct place: we find it is less than all of its children, or there are no children to compare it to. Again, at most we do O(Log2 N) operations, one per height in the tree (and each operation takes O(1) time). We call this the "peroclate down" operation. Actually, because we are comparing its value against up to two children (for percolate up we always compare it to just its one parent), we might have to do two times the work, and it might take twice as long to percolate a value down as it takes to percolate one up. But a constant of 2 disappears from our complexity class analysis, so we can ignore the difference and just count swaps, not comparisons. Compactly Storing Heaps in Arrays: Finally, we will find that we can store a heap compactly in an array, with no explicit pointers to its parent or its left or right child. The simplest mapping goes as follows. We start by storing the root at position 1. If a parent is stored at index i, then its left child is stored at index 2i and its right child is stored at index 2i+1. If a child is stored at index i, than its parent is stored at index i/2 (remember integer division truncates: note that 2*i/2 and (2*i+1)/2 both result in i). With this scheme, we store no value at index 0 in the array (wasting that one storage location). So, storing a heap of N values requires an array with a length of N+1. Look in the web reading for an example of a heap and how to store it in an array. Notice the order in the array is a equivalent to a breadth-first traversal of the tree. If we changed the mapping functions to left child is 2i+1, right child is 2i+2, and parent is (i-1)/2, we can store the root at index 0 and not waste any space. By doing so, computing the left/right child and parent are a bit more complicated/expensive. So, which mapping to use results in a time/space tradeoff. Of course, we can always write the functions left(int i), right(int i), and parent(int i) which take the index of any heap node and compute the indes of it left child, its right-child, or its parent. Using these three functions allows us to change the mapping easily. Note that heaps are good for implementing priority queues. And we can sort a list of N values by adding each to the pq/heap in O(Log2 N) for a total of O(N Log2 N) and then removing each one (they come out smallest to biggest) also in O(Log2 N) for a total of O(N Log N). So we have discovered an O(N Log2 N) sorting algorithm, better than any of the O(N^2) sorts that are easy to write with nested "for" loops. I expect you to be able to draw pictures of heaps (both as trees and arrays) and update them according to these algorithms; you will also write the code for them in Java in Programming Assignment #3. Why not store all binary trees in arrays using this mapping? Well, some BSTs have a very pathological structure, making their storage in arrays inefficient. We can store heaps with N values in an array with N values; but a binary tree with N values can require an array with 2^(N-1)+1 values. For example, the folowing pathological tree requires an array with 17 values: 1 is stored in index 1, 2 is stored in index 2, 3 is stored in index 4, 4 is stored in index 8, and 5 is stored in index 16 (remember that with the simple mapping, index 0 is empty). 1 \ 2 \ 3 \ 4 \ 5 So, this method of storage is poor, unless the trees are close to perfectly balanced; heaps are always perfectly balanced. Building a Heap Offline: Some Interesting Mathematics Related to Algorithms "Online" algorithms receive their inputs one at a time and have to completely update their data structure before the next value is received and processed. Building a heap by adding one value at a time is an example of an online algorithm. We start with an emtpy heap, and after we add each values we get a new heap with one more value in it. "Offline" algorithms receive all their inputs before they are required to process any of them. We can use all these values together to create the data structure, which possibly isn't a heap until the very end. We can write an O(N) algorithm to build a heap of N values if we can get all the values that will be added to a heap BEFORE we start building the heap. This is another really interesting result of the heap order/structure properties. We previously saw an offline algorithm for building a balanced BST from a (sorted) array of values in O(N). Here we will examine how to build a heap in O(N), if we know all the values in the heap before we start. Here is a discussion of this offline algorithm. We will first use "h", the height of a heap with as many nodes as possible, as a metric, not N, the size (number of nodes) in the heap. We will count the number of comparisons needed to build heaps of different heights. Let's start with the smallest height, h=0. A heap of height 0 has only one node in it, so it takes no comparisons to build (that note is the root). We can build a heap of height h=1 (3 nodes) by putting a (it stands for any value) on top of two h=0 heaps (b and c, standing for any values) and then do one pair of comparisons (b compared to c and a compared to the smaller) and at most one swap a with its smallest child (if a is not already less than both its children). Then we have a heap storing these three values and requires at most 2 comparisons and a swap, which is just a contant number of operations. a / \ b c In fact if we have two heaps of height h, then we can efficiently build a new heap of height h+1 by putting the new value x on top of these two heaps and then percolating the value x down into the heap where it belongs. x / \ / h+1 \ + + / \ / \ / h \ / h \ +-----+ +-----+ Let Ph be a heap that is a perfect tree of height h; perfect means every depth is filled. Size(Ph) is just the number of nodes in this heap that is a perfect tree, which we've computed as 2^(h+1) - 1. Let C(Ph) be the number of pairs of comparisons needed to build such a heap with the algorithm outlined above (it is easier to count pairs, and multiply by two: of course multiplying by two doesn't change the complexity class, so we will never even bother with this "correction"). According to the algorithm, we will build a heap Ph by first building two heaps Ph-1 and then putting one value on the top and percolating it down at most h times. Recursively, we will will build each heap Ph-1 by first building two heaps Ph-2 and then putting one value on the top and percolating it down at most h-1 times. Thus the number of comparison pairs is just two times the number needed to build the smaller (by one depth) heaps plus the maximum number of comparison pairs (against each pair of children) needed to percolate the value down to its correct node. We can write this relationship with the following recurrence equations. C(P0) = 0 C(Ph) = 2 * C(Ph-1) + h We could write these equation as a trivial Java method to compute C(Ph) as public static c(int h) { if (h == 0) return 0; else return 2*c(h-1) + h We can summarize this information as follows. h | Size(Ph) | C(Ph) ------+-----------+------------------------- 0 | 1 | 0 1 | 3 | 1 = 2* 0 + 1 2 | 7 | 4 = 2* 1 + 2 3 | 15 | 11 = 2* 4 + 3 4 | 31 | 26 = 2* 11 + 4 5 | 63 | 57 = 2* 26 + 5 6 | 127 | 120 = 2* 57 + 6 7 | 255 | 257 = 2*120+ 7 ... | ... | ... Here, as h gets bigger, Size(Ph) is about the same as C(Ph). To a good approximation, increasing the height of Ph by 1 a bit more than doubles the number of nodes in the tree and requires a bit more than doubling the number of comparison pairs. In fact, the ratio C(Ph)/C(Ph-1) approaches 2 (from above) as h goes to infinity (do the divisions above). In fact, we can write a solution to these recurrence equations exactly as C(Ph) = Size(Ph) - (h+1) = 2^(h+1) - 1 - h - 1 = 2^(h+1) - h - 2 (recall Size(Ph) = 2^(h+1) - 1) Try computing a few values of this function and compare them against the values in the table, computed by the recurence equations. If we assume that this formula is true for C(Ph) we can show that it is true for C(Ph+1) as well. C(Ph+1) = 2*C(Ph) + h = 2^(h+2) - 2h - 4 + h = 2^(h+2) - (h+1) - 3 = 2^(h+2) - 1 - (h+1) - 2 = Size(Ph+1) - (h+1) - 2 Finally, since N (the number of nodes in Ph) = 2^(h+1) - 1. If we discard the -1, to simplify things N ~ 2^(h+1) Log2 N ~ h+1 Log2 N -1 ~ h So we can rewrite our solution as C(N) ~ N - Log2 N - 1, which means C(N) is O(N), because we can drop the Log2 N and constant terms. Thus, the time the algorithm requires is some constant times the number of comparisons it does, which are O(N) so the algorithm is O(N). The algorithm to build small heaps and combine them into bigger and bigger heaps turns out to be trivial to write when the heaps are stored as an array. To be concrete, suppose we are to build a Min-Heap from the following 7 random values: 4, 7, 3, 5, 2, 6, 1 (7 is the number of values needed for a height 2 perfect tree). I will now arrange these values into a tree and its underlying array and illustrate how the algorithm. To start, just put these values, in whatever order they are, into the array representing the heap. It is NOT a heap yet, because even though it satisfies the structure property, it doesn't satisfy the order property. 4 / \ 7 3 /\ /\ 5 2 6 1 0 1 2 3 4 5 6 7 +---+---+---+---+---+---+---+---+ | | 4 | 7 | 3 | 5 | 2 | 6 | 1 | +---+---+---+---+---+---+---+---+ Now, for the values at the deepest depth: 1, 6, 2, and 5 (in that order: indexes, 7, 6, 5, and 4) percolate them down to become heaps. They are already leaves so percolate down immediately stops before comparing to any of their children, because they have no children! Thus, the data structure remains unchanged with a total of 0 comparison pairs. 4 / \ 7 3 /\ /\ 5 2 6 1 0 1 2 3 4 5 6 7 +---+---+---+---+---+---+---+---+ | | 4 | 7 | 3 | 5 | 2 | 6 | 1 | +---+---+---+---+---+---+---+---+ Now, for the values at the next higher depth: 3 and 7 (in that order: indexes, 3 and 2) percolate them down into their left or right subheaps to become heaps of height 1. Each requires just one comparison pair to find the smaller of children and to decide whether to swap (1 and 3 are swapped; 2 and 7 are swapped). So we have a total of 2 comparison pairs. 4 / \ 2 1 /\ /\ 5 7 6 3 0 1 2 3 4 5 6 7 +---+---+---+---+---+---+---+---+ | | 4 | 2 | 1 | 5 | 7 | 6 | 3 | +---+---+---+---+---+---+---+---+ Now, for the value at the next higher depth (the top): 4 (at the root of the tree in index 1) percolate it down into its left or right subheaps to become one final heap. It requires two comparison pairs to twice find the smaller of its children and to decide whether to swap (4 and 1 and then 4 and 3 are swapped). So we have a total of 4 comparison pairs. 1 / \ 2 3 /\ /\ 5 7 6 4 0 1 2 3 4 5 6 7 +---+---+---+---+---+---+---+---+ | | 1 | 2 | 3 | 5 | 7 | 6 | 4 | +---+---+---+---+---+---+---+---+ The result is now a heap, satisfying both the order and structure properties. Notice that we percolated down the indexes 7, then 6, then 5, then 4, then 3, then 2, then 1. Building heaps from the bottom to the top, right to left at each depth. So, we can compactly state the offline algorithm to build a heap of N values: percolate down every index in the array starting and the last index and going backwards until index 1. Simple code and a beautiful result. I invite you to generate your own 7- or 15-valued heaps and hand simulate this algorithm to ensure you understand it. Another way to look at analzing this algorithims is as follows. Imagine Ph is a perfect heap with nodes at every depth from the root to the height. It has N nodes. For each node x, it takes C(x) comparison pairs to percolate that node down to its correct location in Ph. + + / \ / \ / \ / \ +-----+ / \ Ph +-------+ Ph+1 If we instead built Ph+1, the new N leaves (actually N+1) would require 0 comparisons to percolate down. Each of the N internal nodes would require 1 more comparison pair to percolate down than it needed in Ph. So, doubling the number of nodes in Ph (from N to 2N+1) requires doubling the number of comparison pairs (it was N, but is now 2N), which is the signature of a linear process. Note that about 1/2 the values in a perfect tree are leaves, so they need 0 comparison pairs; about 1/4 of the values in the tree (the ones above each pair of leaves) need 1 comparison pairs; about 1/8 of the values in the tree (the ones above those) need 2 comparison pairs. Thus, at each depth, going upward, the nodes need 1 more comparison pair, but there are 1/2 as many nodes at that depth. height For th example above, 4 nodes require no comparison pairs, 2 nodes require one comparison pair, and 1 node requires 2 comparison pairs. In a tree with about 1,000,000 values, about 500,000 would require no comparison pairs, about 250,000 would require 1 comparison pair, about 125,000 would require 2 comparison pairs, ... , 2 would require about 18 comparison pairs, and 1 would require about 19 comparison pairs. Although the # of comparison pairs goes up by 1 each time, the number of nodes requiring that many comparison pairs goes down by a factor of 1/2. The end result is that very few nodes require lots of comparison pairs, and the total number of comparison pairs in an N node heap is bounded by O(N). Finally, sorting with a heap is still O(N Log2 N), but building the heap should be faster. The complexity to build a heap (offline) and remove all its values is now O(N) + O(N Log N) which is still O(N Log N), but the algorithm should run faster. Generalizing Heaps: d-heaps We can generalize binary heaps to d-heaps, where d represents the maximum number of children of any node. So, the heaps representing the binary trees that we have just studied can also be called 2-heaps. The ordering property in d-heaps remains the same, and the structure property too, although we need to apply it with more children. A 3-heap with 9 nodes must have the following structure. A / | \ / | \ B C D / | \ / | E F G H I Note that an array storing this heap would look as follows (using the most compact storage scheme, starting at 0). 0 1 2 3 4 5 6 7 8 +---+---+---+---+---+---+---+---+---+ | A | B | C | D | E | F | G | H | I | +---+---+---+---+---+---+---+---+---+ with the access functions 1stChild(i) = 3i+1 2ndChild(i) = 3i+2 3rdChild(i) = 3i+3 parent (i) = (i-1)/3, using truncating division The storage scheme and access functions easily generalize to 4-heaps, 5-heaps, etc. In fact, given a d-heap we can write just two functions that each have d as a parameter nthChild(n,i,d) = d*i+n where 1<=i<=d: this is the nth child of i in a d-heap parent (d,i) = (i-1)/d, using truncating division The percolate up operation is the exactly the same: switching children and parents. Its complexity is O(Logd N), aka Log base d of N. The percolate down operation is a bit more complicated because in a Min-d-Heap each node must be swapped with the minimum of its children (if it is bigger than any child). This requires d operations at each node, so at most O(d Logd N); but since d is a constant for any given heap, the complexity class is still logarithmic. How much difference is there between logs of different bases? We can compute some numbers easily for Log4 N because Log4 N = .5 Log2N So Log2 1,000 ~ 10 and Log4 1,000 ~ 5; Log2 10^6 ~ 20 and Log4 10^6 ~ 10. but remember that each node percolated down would require 2 comparisons instead of 1 to determine which of its children is smallest. So, for percolating down the time would probably be about the same, but percolate up would be twice as fast. Merging: Finally, there is one operation common to priority queues that heaps do not do optimally: merging two priority queues into one. A simple way to do this for any heap implementation is just add every value from one heap to another: this would be an O(N Log2 N) operation (assuming each heap had N values). Of course O(N Log2 N) is good complexity class. Another way to merge is to put both heaps into an array and then use the offline technique above to build a heap from their contents. This is an O(N) operations: putting the two heaps in an array big enough to hold both is O(N) and then doing the offline heap construction algorithm is also O(N), so the resulting complexity is O(N). There are more advanced implementations of priority queues that merge more quickly (while still quickly doing inserts and remove-min operations): leftist, skew, binomial, etc. heaps. The key is to create order and structure properties that constrain the data enough (but not too much) to allow all the operations to work quickly.