Algo Blog

Hey everyone,

Thanks for still keeping tabs on me. I’ll post something tomorrow morning. Stay tuned!

Suppose you are a real estate developer. These days, the market is getting tight so you have decided to turn to algorithms to give you an edge over the competition. You have purchased an irregularly shaped plot of land which you have divided into a square grid. For each square, you have calculated a cost. For example, building where the ground is less stable will cost you more. However, building on a square also has some value to you. For full generality, each square could have its own value as well. For example, a square overlooking the sea might be great to build on. As a simplifying assumption, you’ve decided that the value of a house is the value of the sum of its squares.

You wish to build a rectangular house with the greatest net value (value - cost). How can you do this efficiently? Here’s an example where we’ve pre-combined cost and value:

The max sum is highlighted and it’s 86.

Think for a few minutes before reading below…

(more…)

A question often asked in interviews is: what’s the best way to store a set? There are many different ways to store well-ordered data so that it can be queried quickly. Many people find that a hash table gives them the best performance characteristics, so much so that if you are in an interview and answer with anything else, you will immediately raise the suspicions of the interviewer. However, assuming you have a good hash function, a hash table still has one key flaw: it really doesn’t tell you anything about your relative neighbors. This means that you can’t easily extract an ordered list out of a hash table. You could query all possible keys in order, but this will give you abysmal performance. Furthermore, if you use e.g. a dictionary (strcmp) ordering of strings, you will have an infinite number of keys between any two finite keys. That makes exhaustive search impossible.

But besides extracting an ordered list, there are other uses for having neighbor access. Suppose, for example, you had a set of non-overlapping time intervals which represent different policies in effect at Company X. Each time interval has some starting time as well as an ending time. Now, given a moment in time, you would like to find the time interval that contains it, if there is one; this corresponds to the policy which was in effect at that time. (Assume for now that if the interval starts at a query moment, it contains that moment.) What is a good data structure which would allow you to efficiently add new time intervals and query moments in time intervals?

Think for a few minutes before reading below…

(more…)

I’ve made a few tweaks to the blog. There are a few minor cosmetic changes. Also, comments are now protected using WP HashCash. Contact me via email if you have trouble.

I’m preparing a new algorithm post which should be ready within 12 hours. Stay tuned!

So, I’ve totally been writing this blog for almost a week, but so far I haven’t talked about how an actual algorithm works. Here’s a classic algorithm.

Let’s say you have an array of n elements a[0] through a[n-1]. You would like to generate a random permutation of this array. This means that any ordering of the array is equally probable, which means that if you look at any position in a, any element has an equal chance of being there: 1 in n (1/n). We assume that we have the function r(n) which generates a uniform random integer between 0 and n-1, inclusive. (In C, you could do this with random() % n, and in Java you’d do (int)(Math.random() * n).)

On two occasions while interviewing, I have been asked for an algorithm to permute an array. Both times the interviewer was amazed when I showed them this algorithm. Many people assume that a second storage array is necessary for remembering numbers you have picked, but you can do it in place with no memory. The algorithm (usually credited to Knuth, but see below) is quite simple:

• For i = 1 to n-1:
  • Let j = r(i+1)
  • Swap a[i] and a[j]

My guess as to why more people don’t know it is due to its interesting use of induction. Software engineers generally learn of induction in college, fear it, learn how to parrot back enough to pass the classes that use it, then promptly forget it. But recursion is ubiquitous and unavoidable. Whether explicitly recursing with a function calling itself, or implicitly in a loop, any time later steps assume that earlier steps have done their work over part of the data, you need induction. Instead of fearing induction, learn how it works and learn how you can rely on it to do more with your data.

So how does the algorithm use induction? Suppose we are in iteration i. Assume inductively that a[0]…a[i-1] are a random permutation of that part of the array. As a base case, a[0] starts as being a permutation of itself. Now consider what the array will look like after the swap. Element i has an equal chance of being in any position 0…i. What about the remaining elements? Each one has a (i-1)/i chance of standing still and a 1/i chance of being put in the ith spot. If it stands still, it had a 1/(i-1) chance of being there by the induction hypothesis. But multiplying this by the stand-still probability, we get a 1/i chance of it being there after the swap. Together with the swap probability, this means that it has a 1/i chance of being in any spot, which is the definition of a uniform permutation.

As a side note, Wikipedia gives a different version:

• For i = 0 to n-2:
  • Let j = i + r(n-i)
  • Swap a[i] and a[j]

This has the advantage that as soon as the i-loop visits a location, its value is finalized so you can output the permutation as it is being created. The disadvantage is that it’s “less inductive” in that, after the ith iteration, you don’t have a true permutation of the first i elements, but only part of a permutation of all n elements. Also, if you are simply generating a random permutation of the numbers 0…n-1 (or analogously the result of some function), you can replace a[i] with i (or f(i)) in the first version and you don’t need to initialize the array. This doesn’t work in the second because of its “future swaps”. This is the version that most people give. A proof of its correctness is given here.

What are some uses of random permutations? They can be good for generating test data, e.g. for a sorting function or any other function which should be able to deal with random data. They can be good as a starting point for a hill-climbing algorithm, e.g. with Traveling Salesman. And, of course, they’re great for asking about in interviews.

Update: I’m overwhelmed by the response! Thanks for all the replies, on here and Reddit. Here are some clarifications:

• I agree that random() % n is not perfectly uniform, but neither is (int)(Math.random() * n) due to the finite precision of the floating point number. Both of them are pretty close for small values of n. We could do better, but it’s outside the scope of this post. Unlike rand, random is supposed to have good entropy in the low order bits.
• Swapping i with r(n) is a simplification. This doesn’t give you a uniform distribution, however. As one reader mentioned, this page gives the lowdown, as do several commenters on reddit.
• Someone took issue with the first algorithm. However, they cited that same page, which is only about the i-with-rand(n) issue (and a[i] remaining stationary). If you can find a fundamental flaw with my proof, please tell!

Again, thanks, and I hope you keep reading!

We all use algorithms every day, but what is an algorithm, exactly? By my definition, an algorithm is a formalized procedure to accomplish a task (this is essentially the same as Wikipedia’s).

Not everything you do is an algorithm. Some things we do have no known good algorithm, such as writing a blog entry. On the other hand, there’s an algorithm out there to write computer science papers. Other things do have an algorithm but we typically don’t explicitly use one. For example, when I vacuum a carpet, I generally don’t think about how to cover the area using the minimal amount of effort, I just do it.

An algorithm exists for any problem which we can state formally, as long as there is a finite search space. In this case, we can always “try all possibilities”. Prolog will do this for you, but not necessarily very efficiently. The fun comes when we want to create the most efficient algorithm possible according to some metric.

A key feature of an algorithm is that it’s deterministic. That is, given exactly the same input, it will always give the same output. I’m almost fibbing: some algorithms use randomness. The most frequently used of these is probably quicksort, a method of sorting which uses randomness to help it divvy its work into roughly equal piles. However, we may view the random stream used by this process as a part of its input. Given the stream, we know exactly how the algorithm will execute. When we prove the correctness of a random algorithm, we assume that the random string is drawn from a certain distribution.

Sometimes it’s hard to tell if something you do has an algorithmic basis. Is how much you love someone determined algorithmically? Some people believe that the universe itself is governed by an algorithmic process. This concept is sometimes known as digital physics. In that case, love, hate, and everything under the sun is the product of an algorithm. If this were true, I would argue that it wouldn’t detract from the universe’s beauty, richness, or importance, any more than the physical medium of a painting detracts from the painting itself.

Most scientists, I think, believe that the totality of the rules behind the universe are describable. But it’s well known that it’s impossible for the state of the universe to be observed. The Heisenberg uncertainty principle is another example of this: you can’t know both the exact position and velocity of a particle.

But suppose we somehow had the whole state of the universe. There’s still something else: the information which we get every time we observe certain quantum phenomena, such as the spin of an electron. What if this stream were completely unknowable until after the fact? This could very well be the case if the many-worlds interpretation is correct. Essentially, every time the universe needs a “random” bit, it splits into two universes; which universe you perceive yourself in is completely arbitrary. This means that even if the rules behind the universe are fixed and even if all possible information is revealed to you, its behavior is not predictable, not even in a statistical sense. Can we still call it an algorithm?

At this point, it all becomes a little too philosophical for me to bother with. There are too many interesting, concrete algorithms out there for me to spend much time on this topic.

Ask.com has recently launched an ad campaign centered around “the algorithm”. They are obliquely referring to their search algorithm “Teoma” when they say “The algorithm is from Jersey”. They even ran a full-page ad in various major newspapers written in the style of an academic article. The ad is reproduced at thealgorithm.com. The only fact I could glean from it was that they use something resembling HITS in that they give each page a hub and authority value. This is reflected in the Wikipedia entry for Teoma, which states that it was inspired by IBM’s “CLEVER” search engine, which in turn used Kleinberg’s HITS algorithm. Other than that, it’s all marketing fluff designed to make Ask’s core technology seem complicated and cool, despite the fact that it apparently hasn’t changed for a decade. (Of course, the proof of Fermat’s last theorem is older than that, and it’s about as complicated as things get around here.)

I view this campaign as being kind of like how Starbucks is to coffee. Small coffeehouses love to bash Starbucks. However, Starbucks has raised the awareness of coffee as a gourmet beverage and the small coffeehouses profit because of it. So, I am glad that Ask has raised people’s awareness of algorithms. Maybe it will make me seem more valuable.

Howdy. This is my blog about algorithms.

Posts will fall into the following categories:

1. Algorithms assigned as homework problems. These will be tagged as ‘edu’. I welcome submissions via email. I will always follow these guidelines:
   • I will only post about problems accessible from the “general web”, by which I mean that Google can find it.
   • I will only post about problems which were due at least a month ago. This should cover extensions.
   • I will always paraphrase the problem (or maybe even rewrite it as an equivalent problem) and I won’t link back to any source. This makes searching more difficult and allows the problem to be reused.
   • If you are a teacher and don’t want me to use your problems, please email me. You will have to give me a convincing reason to delete existing posts.
2. Fairly recent results in algorithms. These will be tagged as ‘new’.
3. Classic algorithms, tagged as ‘classic’.
4. Algorithms as mentioned in popular culture or the media. These will be tagged as ‘pop’.
5. Philosophical stuff, ‘philosophy’
6. My own personal ideas, ‘idea’.

I’ll add to this as I see fit.