There is a theory which states that if ever anybody discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened. – Douglas Adams

What is the algorithm behind the Facebook News Feed aggregation of stories around a particular keyword?

Facebook says the average user has access to about 1,500 posts per day but only looks at 300. (A user who scrolls endlessly will eventually see every post from their friends and a smattering of posts from Pages they follow.)

To ensure that those 300 posts are more interesting than all the rest, Facebook says it uses thousands of factors to determine what shows up in any individual user’s feed. The biggest influences are pretty obvious. How close you are to a person is an increasingly important metric, as judged by how often you like their posts, write on their Timeline, click through their photos or talk with them on Messenger, Facebook’s chat service. The post-type is also a big factor, as Facebook hopes to show more links to people who click lots of links, more videos to people who watch lots of videos and so forth. The algorithm also assumes that content that has attracted a lot of engagement has wide appeal and will place it in more people’s feeds.

But there are other, less intuitive factors to the algorithm. Use a phone with a slow mobile connection and you may see less video. Writing “congratulations” in a comment signals the post is probably about a big life event, so it will get a boost. Liking an article after you clicked it is a stronger positive signal than liking before since it means you probably read the piece and enjoyed it.

The exact calculus that determines how all these factors coalesce is in constant flux, according to the company. Engineers are also continually running multiple experiments with about 1% of Facebook users in an attempt to boost engagement. During a week in mid-May, for instance, one test was giving greater preference to tagged photos that included close friends, while another was boosting the ranking for stories people spent more time looking at on the iPhone. Experiments that are deemed successful by News Feed’s product managers are quickly rolled out globally to all users. Big updates are noted on a News Feed blog, but smaller changes occur without fanfare. Two to three changes typically occur every week.

The new team of human raters, which Facebook calls the “feed quality panel,” are key to surfacing this meaningful content. Each day a typical panellist rates 60 stories that would actually appear in their own News Feeds on a 1 to 5 scale, judge how interesting they found the posts. They also reorder their own feeds to show how their priorities differ from the algorithm’s, providing what Facebook calls a transposition score. And they write paragraph-long explanations for why they like or dislike certain posts, which are often reviewed in the News Feed engineers’ weekly meetings. Facebook also regularly conducts one-off online surveys about News Feed satisfaction and brings in average users off the street to demo new features in its usability labs.

Some of the changes sparked by these methods have already reached you on Facebook, such as an April tweak that placed increased priority on posts by close friends and a June change that added time spent reading a post as an algorithm factor. The company sees these kinds of improvements, which can’t necessarily be measured through basic engagement metrics, as crucial. Facebook says the feed quality panel is currently representative of its U.S. userbase, but the company has plans to expand the initiative globally !!!

How does  this work 

1)  language models based on publicly available corpora for our entity extraction. Based on this data we can extract topics at various levels of confidence. To answer your question, yes, it can figure out terms like “arrested development” out of a normal text. It can also disambiguate between words like “Apple” the fruit, and “Apple” the computer company.

2)  A second level of infrastructure that tries to use other data to increase accuracy. Generally, you can think of this adding more context into the equation, whereas the first level only takes into account the text of a message.

3)  heuristic Implementation decide to show a particular cluster. Generally, this is a combination of trying to filter out noise from the extraction system and deciding when something is newsworthy enough to show. Two of your friends talking about bananas, for example, is pretty uninteresting.


One of the  Magical things about twitter  is , It opens a wide open space to real time Monitoring, An event happened anywhere in the planet and seconds after the Event / During the happening it is shared with People through twitter Feed.
Consider the Example various events and their twitter feed shared in the form of tweets across tweets .


When each of these events happening , people instantly came to twitter and in particular , Tweet search to Discover whats Happening! , From search and advertising perspective these events pose several challenges for various situations.


Before we drive into the system , here is an overview of how the system works

  1. First we Monitor for Search queries that are popular under a variety of events happening.

Now we Run and analyze Statistical data of search queries asa topology.

Example– The query Icetea may have zero searches a day but Iphone7launch may be a popular search query .

2.As soon as we discover new popular search query we will send it to Human evaluation systems , which predicts a variery of questions about the query .

Now the topology detects the query has sufficient popularity which connects the API with search query and dispatches the Results .

Example -As we notice icetea, we will ask human judges to categorize the query and provide the information Related to the query (like interesting pictures based on event , query about an event ,relevant data , tweet analytics,  twitter ads etc )

Once the  data is received we push the data  to backend systems , so that next time a user Searches a query, our Machine Learning models will make use of additional information where human Judges will tell that ice tea is related to Food , when next time  a User performs a search we can get the data based ads by user tweets not monitored ads through twitter.

Monitoring Popular queries –

Storm is a distributed system for real-time computation. In contrast to batch systems like Hadoop, which often introduce delays of hours or more, Storm allows us to run online data processing algorithms to discover search spikes as soon as they happen.

In brief, running a job on Storm involves creating a Storm topology that describes the processing steps that must occur, and deploying this topology to a Storm cluster. A topology itself consists of three things:

Tuple streams of data. In our case, these may be tuples of (search query, timestamp).

Spouts that produce these tuple streams. In our case, we attach spouts to our search logs, which get written to every time a search occurs.

Bolts that process tuple streams. In our case, we use bolts for operations like updating total query counts, filtering out non-English queries, and checking whether an ad is currently being served up for the query.

Here’s a step-by-step walkthrough of how our popular query topology works:

  1. Whenever you perform a search on Twitter, the search request gets logged to a Kafka queue.
  2. The Storm topology attaches a spout to this Kafka queue, and the spout emits a tuple containing the query and other metadata (e.g., the time the query was issued and its location) to a bolt for processing.
  3. This bolt updates the count of the number of times we’ve seen this query, checks whether the query is “currently popular” (using various statistics like time-decayed counts, the geographic distribution of the query, and the last time this query was sent for annotations), and dispatches it to our human computation pipeline if so.

One interesting feature of our popularity algorithm is that we often rejudge queries that have been annotated before, since the intent of a search can change. For example, perhaps people normally search for “Clint Eastwood” because they’re interested in his movies, but during the Republican National Convention users may have wanted to see tweets that were more political in nature.

Human evaluation- At Twitter, we use human computation for a variety of tasks. (See also Clockwork Raven, an open-source crowdsourcing platform we built that makes launching tasks easier.) For example, we often run experiments to measure ad relevance and search quality, we use it to gather data to train and evaluate our machine learning models, and in this section we’ll describe how we use it to boost our understanding of popular search queries.

So suppose that our Storm topology has detected that the query “Big Bird” is suddenly spiking. Since the query may remain popular for only a few hours, we send it off to live humans, who can help us quickly understand what it means; this dispatch is performed via a Thrift service that allows us to design our tasks in a web frontend, and later programmatically submit them to crowdsourcing platforms like Mechanical Turk using any of the different languages we use across Twitter.

On our crowdsourcing platforms, judges are asked several questions about the query that help us serve better ads. Without going into the exact questions, here are flavors of a few possibilities:

What category does the query belong to? For example, “Stanford” may typically be an education-related query, but perhaps there’s a football game between Stanford and Berkeley at the moment, in which case the current search intent would be sports.

Does the query refer to a person? If so, who, and what is their Twitter handle if they have one? For example, the query “Happy Birthday Harry” may be trending, but it’s hard to know beforehand which of the numerous celebrities named Harry it’s referring to. Is it One Direction’s Harry Styles, in which case the searcher is likely to be interested in teen pop? Harry Potter, in which case the searcher is likely to be interested in fantasy novels? Or someone else entirely?

Turkers in the Machine

Since humans are core to this system, let’s describe how our workforce was designed to give us fast, reliable results.

For completing all our tasks, we use a small custom pool of judges to ensure high quality. Other typical possibilities in the crowdsourcing world are to use a static set of in-house judges, to use the standard worker filters that Amazon provides, or to go through an outside company like Crowdflower. We’ve experimented with these other solutions, and while they have their own benefits, we found that a custom pool fit our needs best for a few reasons:

In-house judges can provide high-quality work as well, but they usually work standard hours (for example, 9 to 5 if they work onsite, or a relatively fixed and limited set of hours if they work from home), it can be difficult to communicate with them and schedule them for work, and it’s hard to scale the hiring of more judges.

Using Crowdflower or Amazon’s standard filters makes it easy to scale the workforce, but their trust algorithms aren’t perfect, so an endless problem is that spammy workers get through and many of the judgments will be very poor quality. Two methods of combatting low quality are to seed gold standard examples for which you know the true response throughout your task, or to use statistical analysis to determine which workers are the good ones, but these can be time-consuming and expensive to create, and we often run tasks of a free-response research nature for which these solutions don’t work. Another problem is that using these filters gives you a fluid, constantly changing set of workers, which makes them hard to train.

In contrast:

Our custom pool of judges work virtually all day. For many of them, this is a full-time job, and they’re geographically distributed, so our tasks complete quickly at all hours; we can easily ask for thousands of judgments before lunch, and have them finished by the time we get back, which makes iterating on our experiments much easier.

We have several forums, mailing lists, and even live chatrooms set up, all of which makes it easy for judges to ask us questions and to respond to feedback. Our judges will even give us suggestions on how to improve our tasks; for example, when we run categorization tasks, they’ll often report helpful categories that we should add.

Since we only launch tasks on demand, and Amazon provides a ready source of workers if we ever need more, our judges are never idly twiddling their thumbs waiting for tasks or completing busywork, and our jobs are rarely backlogged.

Because our judges are culled from the best of the crowdsourcing world, they’re experts at the kinds of tasks we send, and can often provide higher quality at a faster rate than what even in-house judges provide. For example, they’ll often use the forums and chatrooms to collaborate amongst themselves to give us the best judgments, and they’re already familiar with the Firefox and Chrome scripts that help them be the most efficient at their work.

All the benefits described above are especially valuable in this real-time search annotation case:

Having highly trusted workers means we don’t need to wait for multiple annotations on a single search query to confirm validity, so we can send responses to our backend as soon as a single judge responds. This entire pipeline is design for real-time, after all, so the lower the latency on the human evaluation part, the better.

The static nature of our custom pool means that the judges are already familiar with our questions, and don’t need to be trained again.

Because our workers aren’t limited to a fixed schedule or location, they can work anywhere, anytime – which is a requirement for this system, since global event spikes on Twitter are not beholden to a 9-to-5.

And with the multiple easy avenues of communication we have set up, it’s easy for us to answer questions that might arise when we add new questions or modify existing ones.

U.S Election Polls – Data Set 2016

This dataset is a collection of state and national polls conducted from November 2015-November 2016 on the 2016 presidential election. Data on the raw and weighted poll results by state, date, pollster, and pollster ratings are included.


There are 27 variables:

  • cycle
  • branch
  • type
  • matchup
  • forecastdate
  • state:
  • startdate
  • enddate
  • pollster
  • grade
  • samplesize
  • populaion
  • poll_wt
  • rawpoll_clinton
  • rawpoll_trump
  • rawpoll_johnson
  • rawpoll_mcmullin
  • adjpoll_clinton
  • adjpoll_trump
  • adjpoll_johnson
  • adjpoll_mcmullin
  • multiversions
  • url
  • poll_id
  • question_id
  • createddate
  • timestamp

Inspiration-The original dataset is from the FiveThirtyEight 2016 Election Forecast and can be downloaded from here. Poll results were aggregated from HuffPost Pollster, RealClearPolitics, polling firms and news reports.