How Twitter tweets your tweets with open source

when Twitter recently joined The Linux Foundation. You couldn't tweet about your dinner, your latest game, or the newest political rumor without open-source software.

Chris Aniszczyk, open-source manager at Twitter, explained just how much Twitter relied on open source and Linux at LinuxCon, the Linux Foundation's annual North American technology conference. “Twitter's philosophy is to open-source almost all things. We take our software inspiration from Red Hat's development philosophy: 'default to open.''”

Specifically, according to the company, “The majority of open-source software exclusively developed by Twitter is licensed under the liberal terms of the Apache License, Version 2.0. The documentation is generally available under the Creative Commons Attribution 3.0 Unported License. In the end, you are free to use, modify and distribute any documentation, source code or examples within our open source projects as long as you adhere to the licensing conditions present within the projects." Twitter's open-source software ware is kept on GitHub.

You're welcome to use this code. Indeed, Aniszczyk strongly encourages others to use and build on it.

Twitter itself is famous, or infamous in some circles, for having been built on Ruby on Rails. Today though Aniszczyk said, Twitter has moved to Java and a list of open-source programs longer than your arm.

If Unix and Linux are operating systems that are made of many utilities loosely coupled than Twitter is a social network made up of many open-source programs loosely couped together. Some parts will be familiar to anyone in Linux or Web development circles.

Twitter's core operating system is Linux 2.6.39 and for its core database it uses MySQL. To manage the source code for the rest Twitter uses Git. Linus “Linux” Torvalds' other software baby.

But, let's cut to the chase, what actually happens when you tweet?

First unless you've never used “The Twitter,” you know that a tweet is a short of 140 characters or about 200 bytes. When you send this tweet it will soon be “fanned out” to the people who read your tweets. Sound easy right? “Wrong!” Proclaimed Aniszczyk.

The problem is the Twitter's scale. Twitter handles 2.8-billion tweets during a typical year. That counts to 5,000 tweets a second on average. But, Aniszczyk said, things aren't always average. When someone noticed the singer Beyonce showing a baby bump, traffic went up to 8,800 Tweets per second (TPS). The last SuperBowl? 12,000-plus TPS, and when someone got the idea that everyone should go see an anime movie and then tweet about it, Twitter faced one of its greatest challenges: 25,088 TPS.

What happens with each of these tweets is they put are registered as a status update. Then each one is given a unique ID using a program called snowflake. Next, it's geolocation data is noted by Rockdove, a program that hasn't been made open-source yet.

Each tweet is then checked by a combination URL shortener and spam detector called t.co. Once past this stage, each tweet is stored in MYSQL by Gizzard, a flexible sharding framework for creating eventually-consistent distributed datastores. Now, and only now is an HTTP 200 signal, meaning all has gone well, to your Web browser.

Of course at this point your tweet hasn't gone out to the world. First, your tweets get started on their way to Bing and other search programs using the Firehose application programming interface (API). Finally, your tweets are ready for fanout, that is heading to your friends, family, and fans.

The actual process is handled by FlockDB. This is an open-source graph database that sits on Gizzard and pulls data from MySQL. FlockDB contains all of Twitter's users and their relationships to one another. Now, armed with the your followers addresses your tweets are finally on their way.

The average time all this takes? About 350-milliseconds. Not bad for a system handling 5,000 TPS every day, 24-hours a day.

Twitter may be causing some of its would-be partners grief with tighter API rules, but the company itself does an exceptional job of delivering thousands of messages every moment of the day with open-source software.

Twitter Joins Linux

Twitter uses and builds a fair amount of open-source software, so it wasn’t too shocking when we read in our inboxes this morning that the social media startup has joined the Linux Foundation.

“Not only is Twitter built on Linux, but open source software is core to its technology strategy,” said a Linux Foundation rep to VentureBeat via email.
“It’s investing even more in the platform now as the company evolves and positions itself for the future. Linux has become even more dominant among web-based companies as the ‘hacker way’ has become pervasive among the newest generation of startups.”
Ah, yes, the Hacker Way. Or should we say, the “Hacker Way.”
Espousing open-source ideals, at least in spirit, has become increasingly common among web startups, especially in the Bay Area. Facebook CEO Mark Zuckerberg famously wrote a “hacker way” mini-treatise into his company’s SEC IPO filing.
But in fact, while companies like Facebook and Twitter rely on open-source technologies and programming languages to get their various jobs done, their businesses are conceptually based on proprietary software, not open-source software.
As famous hacker Eric “esr” Raymond pointed out in a recent interview with VentureBeat, the true hacker way means “to give control to the individual, to respect his or her privacy, to create tools for autonomy and liberty, and to encourage creative re-use of software” — only parts of which are built into Twitter’s products.
While Twitter has open-sourced some of its software — a load-balancer called Iago and a design framework called Bootstrap, for example — vast expanses of Twitter code remain under lock and key, and the company’s recent and coming API changes means it’s getting farther away from anyone’s definition of free and open-source software.
This is a problem. Specifically, it’s a recruitment problem.
Twitter needs to continue to pull in the best, brightest, neckbeardiest developers the world can offer, and it can’t do so without some commitment to open-source communities. The company actually recently hosted an open-source event with thinly veiled recruitment mechanisms built in just for this reason: great developers and open-source software go together like peanut butter and jelly, and the more you can convince a great developer that your company believes in open-source, the more likely you are to recruit great developers in a highly competitive hiring environment.
All that being said, Twitter does have a vested interest in helping to advance the cause of Linux in particular, and some participation in open-source communities is better than none at all.
As a web-based business, Twitter, like every other web service, is supported by tens of thousands of Linux servers. In a statement on today’s news, the company said it intends to partner with the Linux Foundation to promote and protect Linux, the open-source operating system.
“Linux and its capability to be heavily tweaked is fundamental to our technology infrastructure,” said Twitter open-source manager Chris Aniszczyk in the statement.
“By joining the Linux Foundation, we can support an organization that is important to us and collaborate with a community that is advancing Linux as fast as we are improving Twitter.”

Open-source attitude in the Internet Era

The road of history appears to be leading steadily towards a more open-sourced vision

The term ‘open source’ comes from computer programming. It refers to a computer program that isn’t owned by any company and is freely available to the general public. Microsoft Word, by contrast, is ‘closed source’ — the Microsoft Corporation owns the code for its software and will never make it available or give it away for free.

A little-known program called Open Office is a freely available alternative to Microsoft Office with many of the same features. A loose group of programmers around the world created Open Office and constantly tinkers with it to make it better. They do this for free with no benefit besides the pleasure of providing a useful service for anonymous users.

The motivation for developing open-source software can be puzzling, but they are becoming increasingly ubiquitous. Many phone users, for instance, opt for closed-source versions of Apple iPhones or BlackBerries. However, people using Samsung and other devices are running an open-source operating system — Android. (Google leads the programming effort, but doesn’t own the source code.)

Phones running Android operating systems tend to be cheaper because no money must be earmarked for the development costs. Of course, many people are quite happy to pay a little more for the features of closed-source iPhones and BlackBerries.

Freedom to innovate
But increasingly, both BlackBerry and the Apple iPhone flirt with open-sourced systems — allowing independent programmers leeway to produce apps for their closed-source platforms. In fact, in many other areas open-source alternatives are increasingly embraced.

Wikipedia, for instance, continues to grow in popularity while Encyclopedia Britannica recently announced the end of its print edition. The open-source Wikipedia allows users to modify any entry, but a team of editorial volunteers tends to keep material fairly accurate and well-balanced.
Only the editors of the closed-source Encyclopedia Britannica can make changes to their entries, meaning they should be more accurate. However, some research has found that Wikipedia and the encyclopedia both contained roughly the same amount of erroneous material. While it does have some drawbacks, Wikipedia is praised for the speed with which it updates information as well as its insistence on the verification of information.

Websites such as YouTube and Digg (which allows users to rate the value of new stories) represent open-source versions of television networks and newspapers. While a select few gatekeepers once made decisions about what to air or publish, these sites and many others now allow users to collectively make the decisions. The impact of these open-source sites on ‘traditional media’ cannot be understated.

Online alternatives
Many prestigious universities are now promoting open-source research. Academics are avoiding the costly peer-reviewed journals where research is kept behind a paywall in favour or free, online alternatives. The move will help equalise research globally, allowing poorer nations to have access to the same information as richer countries.

While some may debate whether open-source or closed-source systems are more beneficial, the road of history appears to be leading steadily toward a more open-sourced vision. The benefits of an open-sourced systems include an increased acceptance of new ideas and a quicker pace toward innovation — far more so that in closed systems.

Open-source projects also tend to benefit from the ‘wisdom of the crowd’ with ideas and innovations that could be missed in closed environments. Most importantly, open-source systems are transparent — nothing is hidden from view, allowing anyone to offer their input equally.
Even nations seem to fall on one side of the open-source versus closed-source structure. US Secretary of State Hillary Clinton recently remarked that countries would no longer be divided along lines such as East or West or religious beliefs. Rather, they would be judged on their relative approach toward openness.

She said that countries closed to “change, ideas, cultures and beliefs that are different from theirs will quickly find that in an internet world they will be left behind”.

Her reference to the internet is an important point — societies could more easily afford to be ‘closed-source’ before the internet. Many attribute the recent upheaval in the Arab world to an increase in the open-source information flow afforded by YouTube, Twitter and Facebook.

The UAE is already making efforts to be more open-source. The government of Abu Dhabi, for instance, has pledged to create a transparent regulatory environment as part of its 2030 economic vision.
And, as Federal National Council (FNC) member Noura Al Kaabi has noted, the FNC is also leading the way toward openness. The FNC often raises critical issues into the sphere of public discussion and provides added transparency and accountability to workings of the UAE government.

But the UAE could do more to increase its openness. For instance, some government agencies make decisions without much scrutiny or explanation leaving residents in the dark or puzzled.

Similarly, many government and business spokespeople aren’t empowered to truly provide answers and information to the public. In effect, they’re trapped in a ‘closed-source’ mentality.

Hopefully, the UAE will address these problems and continue making strides toward being more ‘open-sourced’.

Much of the world seems ready to embrace the wisdom of the crowd.

Twitter Details Google Summer Of Code Participation

Google’s Summer of Code program is about to get started and Twitter couldn’t be more stoked. This is the first time that Twitter has joined in the program to mentor students and help them learn more about programming.

Google’s Summer of Code focuses on open source technology which is a great fit for Twitter. Not only are they now a sponsor of the Apache Foundation, but they have been open sourcing a lot of the software used to power the inner workings of Twitter. Working with a company as well-known as Twitter must be pretty awesome and we wish them the best.

Twitter chose three students to help them develop code over the summer. All three students will be working on various open source projects that Twitter is developing. Besides the students getting an invaluable education, Twitter gets free help with their code since Google pays the students who participate in the program.

The first student, Federico Brubacher, has been programming since he was 6. If that’s not impressive enough, he is also on his way to finishing up his MS in computer science at ORT Uruguay. During his time with Twitter, he will be “building scalable, online machine learning algorithms on top of Storm.” Storm is the software that powers Twitter’s Analytics platform.

The second student, Kirill Lashuk, is currently studying math and computer science at Belarusian State University in Minsk. For his summer project, he will be adding more localization capabilities to TwitterCLDR. TwitterCLDR “uses Unicode’s Common Locale Data Repository to format certain types of text into their localized equivalents.”

The third and final student, Ruben Oanta, is also on his way to finishing up his MS in computer science at DePaul University. His job will be adding MySQL support to Finagle, a “protocol-agnostic library that abstracts the complicated details of asynchronous RPC communication.”

It looks like Twitter has some fantastic students working on some really important stuff. It’s nice to see a company giving students proper jobs in programming. Twitter is a friend to the open source movement so here’s hoping they instill that same love for open source in the next generation of programmers.

The End of Twitter?

If you work in an internet related field (programming, designing, administrating, etc) you’ve probably heard of the “micro-blogging” service Twitter. For those who don’t know what it is, Twitter is a social networking site where users share “Tweets” (140 character messages) with each other and the world at large. As is the case with many new internet technologies however, Twitter has it’s shortfalls, and it might not be long before newer micro-blogging services take over.

One such new micro-blogging service is identi.ca. Even though identi.ca doesn’t have as many users or features as Twitter (yet!), it does some things Twitter does not.

1) REALLY easy setup.

Identi.ca supports OpenID*. Since I already have an OpenID set up, I used it to register my new Identi.ca account. All I had to do was enter my OpenID, approve identi.ca, and choose a nickname. That’s IT! No adding mundane details like my name or e-mail address… all that was pulled from my OpenID automatically.

Of course, if you don’t have OpenID you can still register at Identi.ca normally, and the process looks simple and easy.

2) IM and Text Message support.

When I first started using twitter, I could “tweet” using an instant messaging program. Unfortunately, this functionality soon disappeared, with little explanation as to where it had gone or when it was coming back. Likewise, one of the strongest features of Twitter is the ability to receive tweets from people you’re following on your cell phone as text messages. In the USA this works fine, but here in Canada Twitter has had to disable the service due to excessive costs from the cell phone companies.

Identi.ca seems to have neatly avoided both problems. As of right now at least they have an IM service that works and they’re using e-mail to SMS for their text messaging. e-Mail to SMS doesn’t cost them anything - the fees are passed on to the user. I looked into it and for me it would be a $5 a month upgrade, or 15 cents per message received. This is a bit of a pain, but at least the option is there.

3) Interfaces with Twitter.

Identi.ca interfaces well with twitter, so if you’re switching over you don’t need to worry about people missing your Tweets. Just add your twitter username and password to Identi.ca, and anything you post to identi.ca is copied to Twitter. Win / Win!

4) Identi.ca is Open Source.

Rather than being a closed platform like Twitter, Identi.ca is built on open source software and works with an open source license. This, combined with support for FOAF files*, Microformats*, and OAuth* means that other webmasters can set up their own micro-blogging systems, and fully integrate back to Identi.ca.

In short, Identi.ca takes sharing what you’re doing right now to a whole new level, and its smart business plan and open licensing agreements mean that it will continue to expand long after Twitter has died. If you’re just thinking about getting into the micro-blogging world, take Identi.ca for a spin. As with Facebook / Myspace, sometimes in the internet world “the second mouse gets the cheese”.

Read More...