You can change the layout, colors, etc for the generated word clouds. It also provides a ‘randomize’ feature where you can randomly choose the format for the word cloud.

Here is a word cloud of navBurner product blog.

(http://wordle.net/)

Amazon Elastic Block Store is a persistent, high performance and highly available block level storage that can be attached to a running Amazon EC2 instance.

Amazon’s Web Services team recently launched Amazon Elastic Block Store and made it available to all it’s user base. Persistance Storage has been the missing (and previously talked about) piece in Amazon’s cloud computing platform and now with the introduction of EBS, that hole has been filled to provide basic building block for a variety of solutions.

Before Amazon EBS, the storage in Amazon cloud was local to a running instance. If the instance got terminated, all the data stored in that instance was lost. Solutions to safeguard the data from being lost included scheduled backups to Amazon S3, and in high change environments, frequent backup strategies introduced tedious processes.

With the introduction of Amazon EBS, virtual machines on the cloud have become more like traditional traditional physical servers. If an EC2 instance goes down, the storage volume can be made available ‘quickly’ by attaching the volume to another EC2 instance. Thus, Persistent storage can act as the machine’s file system that will contain all the data even after restarts.

The separation of storage from the machine has made it possible to attach the same volume to any other EC2 instance. EBS provides the capability to snapshot the entire volume to S3, which makes it easy to create clones in the same and/or other availability zones. Applications can take advantage of this to provide redundancy across availability zones (making sure one creates a new volume from the snapshot in the respective availability zone).

Now that one can have persistent filesystem (of any type) and raw block level access, certain class of applications that deal with large datasets, use filesystem heavily for processing and need to be performant will be easier to build on top of the platform.

LATimes page with Ancillary Blocks highlighted

To increase the quality of the returned results, the ancillary blocks can be removed at the indexing time. There are various approaches to remove the non-informative content from these web pages that have been discussed in the literature (Entropy-threshhold based approach by Lin and Ho, ContentExtractor and FeatureExtractor by Debnath et al). There is also another approach that is very straight forward and is used by perl module HTML-Content-Extractor and has been ported to PHP and is described here.

The product I am working on currently is based on Java and Python, and unfortunately there is no port for Python. Thus, I wrote one in python that uses the ideas above. In short, it does the following three things

1. Heuristics - Looking across web pages, it is evident that certain HTML tags do not create any informative content. These include tags like img, script, link, input, hr etc. These can be deleted from the DOM.

2. Feature based block detection and removal - Identifying ancillary blocks based on text to link ratio.

3. Remove all the HTML tags from the remaining DOM.

So far, I have just tried it on content pages from CNN and LA Times. As I try more pages and find other enhancements / issues, I’ll keep posting.

Solr has a set of default SearchComponent like MoreLikeThisComponent, HighlightComponent, etc and this month, another has been added to the list - SpellCheckComponent. This was in works for sometime, but now is out to be used.


DELETE FROM Table1
WHERE id
IN ( SELECT id FROM
(SELECT A.id FROM Table1 A
LEFT OUTER JOIN Table2 B ON B.id = A.other_column WHERE B.id is null)
AS BOGUSTABLENAME)


The AJAX Libraries API is a content distribution network and loading architecture for the most popular open source JavaScript libraries…….Google works directly with the key stake holders for each library effort and accept the latest stable versions as they are released. Once we host a release of a given library, we are committed to hosting that release indefinitely.

Whenever I take a look at the pages of our new product (n private alpha) using YSlow, two things always rank as ‘F’. First is using a Content Distribution Network (CDN) and second is the expires header (see screenshot below).

The expires header is something that we need to work on but then there is CDN. At initial stages, it’s difficult for developers / development group to get access to a Content Distribution Network. But with Google’s AJAX Libraries API, developers can get access to Google’s CDN to deliver their Javascripts.

All you have to do is to identify the version you are using and replace the script tags with the following.

<script src="http://ajax.googleapis.com/ajax/libs/prototype/1.6.0.2/prototype.js"></script>

Doesn’t Solr already provides faceted search on top of Lucene?

Collect all the paths you can think of in a pile, pull out the 8 paths that 80% of your visitors come looking for, and that’s your home page.

An interesting approach to create Home Page containers for navigation front ends. See full article at Signal vs. Noise

Whoever has worked on a new product knows that the content in these pages evolve overtime. Frequently, during the initial stages, and (hopefully) less often afterwards. The question that I was looking an answer for, was - “How can you setup things so that the developers do not get involved in making the content changes?”

I started looking for how to push the content addition and modification to marketing as opposed to the product dev guys. Apparently, Django framework have an amazingly simple application for exactly this purpose - called - ‘flatpages’. You can add new pages and modify existing pages through the Django admin interface without the need to change the code once flatpages has been installed.

To restrict the marketing guys to only change the static content, create a Marketing Group that has access to only flatpages model in Django admin. That way, they can change the content, and the developers can focus on things that moves the product forward!