Let me give you an update on what significant milestones I've reached in the past six months.
First of all Directlyrics has seen double digit growth every month with in August over 23 million page views. Doubling my previous record from 2007 of an ex-site. Thanks go out to Eminem, Katy Perry and Rihanna with their enormous single successes.
I've also released a complete re-design of the website together with a daily updated music blog which already attracts 30.000 page views per day thanks to Kevin and his work.
With the maturity of site, I’ve also negotiated lyrics publishing rights from the major publishing companies through a deal with Gracenote. Where previously the songwriters did not retrieve any compensation for their work, I’m currently rolling out the official lyrics to the site which guarantee the correct lyrics and compensate the artists for their work.
Next to the publishing rights, I've partnered with BUZZMEDIA for exclusive brand ad sales representation on Directlyrics. This should attract premium advertisers to the site, hence increase user experience and revenue to keep the site running. Sears currently has a campaign running.
And last but not least, I've been developing Fanity for the past few months together with Raoul. A lot of backend work has been done, tons of data is coming in and algorithms are trained to handle it all. We’ve just launched a simple invitation interface to allow alpha users of the site to login and try the site out, while others (you!) can get an invitation for the upcoming launch. Expect something big!
Facebook and MySpace fixed this quickly after being notified.
As a application developer on Facebook, I usually run into certain walls that limit my application functionality. But I don't give up easily, and only recently I found a solution to one of my function limitations. Surprisingly, when looked into more carefully my solution allowed full access and control to the Facebook user account that accessed my application. Did I mention this would also be untraceable since exploit actions would happen from the users IP and own domain cookie?
Lets walk through it along some clarifying images. Flash applications run on a users' computer. A Flash application is able to load data into its environment. This is done by a request of the application, where the user loads a certain URL. Luckily - just with browser AJAX requests- a flash application hosted on domain X is unable to open a file on domain Y. If this would be possible, domain X is able to access content on domain Y, and when the user is logged in on domain Y retrieve and post back any personal data.
In certain cases this could limit a flash application capabilities. A relevant example: an application wants to display public Facebook user thumbnails. The application is on domain X, the thumbnails on domain facebook.com. To resolve such issues, Adobe (Flash's developers) introduced a "crossdomain.xml" file which could allow certain domains accessing another domain, leading to cross domain access by certain or all domains.
While indeed Facebook locked the front door from any non-facebook domain access via Flash, a simple subdomain change allowed any flash application (domain="*") to access it's domain data:.
This wouldn't be a big deal if the subdomain only hosts images, but unfortunately this domain hosts the whole Facebook property, including a facebook user session.
If you have auto-login enabled on facebook, you might recognize your fullname [update: its a screen cap now] in the snippet above (and the keys to do actions from the accounts credentials).
A huge problem that leads to full access and control of a user account whom has "auto login" enabled, and who hasn't?
But how does MySpace fit in this story? You would be surprised if I found a similar back door on not one, but two of the top 10 websites online, right? Well a quick look at the MySpace crossdomain.xml file shows again a locked door, except for one element: the domain farm.sproutbuilder.com was enabled to access myspace.com data.
A look at "sproutbuilder" showed a application builder (which indeed has a module able to load MySpace data: news updates) but more disturbing an upload function allows anybody uploading ".swf" files, the file extension of Flash applications. The location of the uploaded file? farm.sproutbuilder.com [exploit closed], exactly the domain that is allowed access to MySpace data.
You don't need much time to think of all the ways this could be exploited. All what has to happen is a active session, or a "auto login"-cookie and a URL which hosts a exploiting Flash file. For example when accessed, a automatic "post update" could be made, that would lure friends of the user to access the exploit URL, and the exploit would spread virally. An more invasive and hidden exploit could harvest all the users personal photo's, data and messages to a central server without any trace, and there is no reason why this wouldn't be happening already with both Facebook and MySpace data.
News item featured in various publications:
This week I got my Android G1 phone in the mail. I already have the iPhone 3G and a Nokia so it was not because I needed another phone; I got it because I had an urge to develop a custom application for the Android! This urge was sparked from looking at the wide variety of applications available for the Android and more importantly; Android promised a nice developer framework in JAVA where an app can take advantage of the multimedia build into a phone like wifi, GPS, camera and more uniquely to the G1 a compass (bringing in semi-augmented reality).
Things are changing: where only a few years ago websites – another form of an application a.k.a. app - were build by nerdy people in nightly hours, the current generation of nerds are at the core of building billion dollar businesses like Google and Microsoft. Nerds rose from unsocial creatures to Hollywood movie heroes (e.g. Harry Potter, Peter Parker). Today you’re cool if you know how to CSS hack your MySpace page. Nearly everybody has a online social network account. It has also become almost normal to talk about internet stuff on social occasions, something you shouldn’t have attempted only 5 years ago.
In my opinion these are signs of a trend that will continue due to a need to gain full control over computers – in specific by programming them – becomes mainstream. With computer devices from laptops to phones everywhere it make sense that an evolution will go from using available applications to creating your own applications adjusted to your specific demands. The basis of this is local and niche problems can bring local and niche solutions. For example I want to instantly share my French homework with my classmates by making picture of it with my camera phone, transform the image-to-text, spell check it online with an API, push it automatically over Bluetooth to my classmates and have them pay 10cents for the homework copy; a strange app that won’t be made by anyone any time soon, but why wouldn’t they create that app themselves? The boundaries that have prevented this from happening are disappearing fast:
- Knowledge: developing applications has been a very specific quality. But with computer devices all around us the ability to create a custom application for these devices makes programming a more common need. Schools started educating on computer usage and have already included basic programming courses (pun intended). And to be honest programming is no rocket science; it’s actually very logical but just like being good in math or not some people might be better in other stuff.
- Costs: with developing came high costs due to the highly trained personal. But quality guides on programming languages are freely available online and the numbers of books on any programming language are in the thousands. Next to that; where previously websites were made in notepad, WYSISYG website editors and SDK’s like Eclipse (free!) provide some very strong frameworks to work with and help novice developers to create whatever they have in mind.
- Publish & Spread: it is possible for anybody to develop a website and have it available online within seconds. There are no boundaries that prevent you from publishing an webapp online. The Apple app store (15.000 apps) and Google Android market (~1000 apps) provide a platform to publish and spread your app to the world. These apps are not made by big companies, but mostly by people who had an idea (bad or good), developed it, and released it for others to use.
- Money: previously it was hard to monetize your own developed app due to the lack of a (popular) platform that handled (micro) payments. For the web Adsense solved this for website developers by providing advertising on any webpage. The last year or so mobile app stores opened the possibilities to monetize on mobile application development. A thousand downloads of your 1$ application is a nice $1000 dollar, or if you want more you could’ve entered this competition that gave away over 10 million (25k to 225k each).
One major hurdle - in specific for mobile development - are the numerous development environments: JAVA, C++, Objective-C, .NET, J2ME. Don’t think you can make an iPhone app without having a Macintosh, and also don’t think a Nokia 6500 can have the same app code as a Nokia N95. The only solution right now is to create your app for every language there is since everybody still has different phones: if you pick ten random people you get ten different phones types with 6 different development environments.
There is so much more to say about development, but for this post I can summarize it to this: In the near future programming knowledge is widely educated, most current boundaries are resolved and only creativity with some excess time will limit people to make great custom apps for the world to use.
I've gathered my personal best practices to secure my home-coded websites against security exploits via XSS, SQL injection, and CRSF. I am not a security expert at all, and I'll probably never be because I am to lazy to sanitize every single variable coming in and out my websites. Even companies with millions to spend like Facebook, MySpace, Google and other big names have/had exploits that exposed them against malware distribution, abuse of user accounts, data loss and other security issues. Even if you might not have sensitive information on your website, an exploit could target getting ownership of your domain or server: think of a fake cpanel login on your site by sending the webmaster to a exploited url. A good example to read and learn from is this story that describes how someone exploited myspace by easily circumventing basic security patches. While it's hard to close every security gap – some hackers go a long way – the tips below are an understandable introduction to programming security and the code examples will steer you in the right direction to fix them.
PHP: Clean up all the user input
One of the most common exploits are the result of unintended user input. User input by URL, forms and cookies has to get cleaned up from any exploitable input before doing anything with it. Most importantly you want html characters (like <,>) to be encoded to their harmless html representative and ', " escaped by a slash to exclude external code to be forced into your site. This script code below runs through the array $_GET, $_POST and $_COOKIE, and cleans up the values passed from the user. Please force integers on user input e.g. ID's by stripping out any other character. It's best to always also have Mod security installed.
These filters (filter_var) only work in PHP5, but with a good regular expression it can be also run in other versions. It's also good to truncate a string to a maximum number of characters or else you could exposed to this. In this script the string allowed is limited to 100 characters this could break alot of systems (like long user comments) so be careful with that.
PHP: Working with forms
Never put any settings in hidden form fields, and expect them to not be exploited. Easily exploitable is the example below where a form value identifies the user by UserID.
These hidden form fields can be set to any value. Look at this example on a major bank website. Another tip is to add a token to a form field that is based on the clients encrypted IP, session and/or cookie so the form is harder to temper with by CRSF exploits. More on tokens and why it's needed in my upcoming post.
Also dropdowns can best be setup like the example below to restrict unexpected values from the user in $dropdown.
PHP: uploading files
Never let people upload stuff to your server, unless you know what you are doing. Best way is to store the uploaded file in a folder unavailable via a url (e.g. next to your public_html folder) and have a file act as a proxy that grabs the uploaded file and forces a innocent extension like .jpg (only when it's an jpeg of course).If you still want to host the uploaded file on your server available through a direct url make sure it doesn't make flash cross-domain available or other content like text, html, or executable files that could compromise the whole server. Always wonder if you are ready to host anything that isn't yours?
Could easily be exploited by creating a url /?username=name'+alert(1);+foo+=+'bar. The cleanArray() function above helps some against these problems since it adds slashes, truncates the string and encodes characters.
User management: don't store anything critical in a cookie
It seems tempting to store certain values in a cookie, saving time to retrieve settings from a database for example. For example to give a user administration rights:
Also saving $passwords, $usernames and $userID's (especially incremental $userID's. UserID = 1 having admin capabilities?) in a cookie is a bad idea, cookies are as easy to tamper with as changing a variable in a URL. Best way is to crypt or MD5 settings to identify a user:
When a user has a cookie, and you want to check it's credentials. From the cookie $useridentifier you generate the MD5 by combining your encrypt key and user IP and match that with the users cookie set.
PHP: Header forwards
An not widely known behavior on a well known practice: A header forward, without a exit() or die() could continue to load the page if the browser (or an exploiter) continues to load the page.
Mysql: limit your updates and deletes
When performing sql queries that contain dynamic variables always limit your query to 1 (of course unless you need to update or delete a bunch).
Apache: Mod Security
Install mod security. It has a nice black list of rules that block certain (common) exploit attempts from the external environment e.g. blocking exploits in the user agent string, bogus image uploads, injecting XSS, SQL injecting, commands, and url request string. In your .htaccess you set the mod_security to on on most apache installs:
And last be very careful when installing external scripts like Wordpress, Gallery or even Cpanel (web based server control panel) on your server. These scripts could have a security hole that can be exploited throughout your domain and even on the whole server. Always update to the latest version, and don't install in the standard directory so crawl bots won't find it easily.
Search Engine Optimization (SEO) has always been my prime pillar to build traffic to my websites. If search engines love your site, you’ll bump competitors in the search results aside, and traffic that flows from the search engines to your website will increase almost exponential with every jump to the top. Finally at the top for your relevant search keywords, after weeks, months or even years you relax… but I’ve learned that at that time, the fight just begun. Being at the top isn’t the endpoint:
- Competition in the search engine listing is huge and always on your tail. With a very good reason: real money is being earned from the traffic. Thousands of dollars a day could be the difference between being #1 or #2. If you find a great way to out rank your competitor, you can count on them imitating your strategy. And not much stands in their way, because it’s hard to hide your SEO strategy with tools like Yahoo! SiteExplorer or Technorati that reveal your (in-)linking strategy. Or notepad that shows your complete page markup (HTML) . While page markup and content are important, my experience shows in-links are the main ingredient to success (note: that’s not a secret). All these success factors can be revealed and adopted by the competition.
- Search engines are always tweaking their ranking algorithms. Your strategy that got you to the top, could as easily bring you back to the bottom again. If you are #1 yesterday, and #1 today, doesn’t mean you will be #1 tomorrow. I’ve learned that the hard way by now…
You end up with a constant craze of tweaking your markup, adding content, revising changes based on a possible relation between your edit and rankings, expand your site with functions, restyle to minimalism, focusing on high traffic keywords, focusing on niche keywords, link exchanges, no-following, etc. Whatever your personal fad is.
But in the end, the search engines are a blackbox, and you can only hope that you’re lucky enough to come out on top.
My thoughts went out to search for another way to make money with online traffic without search engines as middlemen who determine my online and offline faith. I don’t want to invest in (ad-) campaigns, or do low margin arbitrage on buying and selling traffic, nor do I want to game the system since that wouldn’t be a long term strategy either. I need a strategy that is low maintenance, fully automated, little competition, providing quality ad views/leads and viable as a respectable long term business. It maybe sounds lazy, but that’s not my intention, these qualities will make a scalable business with low operating costs.
And there the answer is: 404. File not found. Dead links. Expired domains. Misspellings. Typos. Dead ends. All traffic that goes to waste and ends on a blank page. I might be able to come up with a variation of what businesses are already doing with dead-end traffic:
- Millions of people mistype domains. Even a small percentage of 0.01%, could lead to billions of pageviews on the wrong place. Instead of focusing on the first part of a domain name (e.g. googgle.com) by buying all these misspellings (typo squatting is a well known ‘business’ practice) for a few dollars each, this company made a deal with the country of Cameroon – owner of the .cm extension – to handle all typos in the last three characters of the domain name: (“domain.cm”). This deal leads any domain + .cm to their ad page.
- Instead of getting the standard not found page, or could not connect from your browser, internet provider Verisign hijacks the user and redirects them to a related ad page instead. A nice traffic flow leading to a good revenue bonus for an internet provider. But they are not unique: browser toolbars are also in the game with a ‘service’ of fetching a dead link, and redirecting a search results/ad page to the browser. Google, Adobe, Yahoo, Microsoft are all forwarding dead end traffic to their businesses. Google’s new browser chrome has it started turned on.
So my brain brainstormed on my next possible project that didn’t need Google, or any other search engine: Own 404! Spider the internet in a smart way, get loads of data from browsers and webpages. See how internet traffic moves from click to click, and focusing on dead ends: a page that isn’t there anymore. When logged, get control of the page (yes, legally), push to a relevant ad page. When no owner of the dead end (domain) exists get the domain. Instead of the dead end 404 page, people get my ads!
With the internet getting older and bigger by the day, more dead ends pop up, and I’ll be happy to take them over. No more high traffic website with content, but a high traffic end-point with ads:
5 visitors per page/day, control 200,000 previously dead-end pages, gives 1 million page views/day, CPM 3 dollar, $3000/day * 365 = 1,095,000 a year.
While the project might be viable… my current and upcoming projects are still bound by the search engines and their judgments and I’ll have to accept it... for now.
With use of the internet movie database I was able to dig into the future releases of Hollywood. With some hacking skills (Google) I was able to grab all upcoming releases in the ‘script only’ stage, to post-production status. With possibly a little over 2000 movies coming out between October 2008 and 2013, we have a lot coming our way. But when you see the future releases list you can’t help but feel like puking. The sheer lack of creativity out in the Hollywood hills is mind boggling. The general idea of a ‘new’ movie is something like this: Just take a popular (comic) book character, hire a hot female cast member, add something with terrorist threatening the world (which in most movies means New York) with a some sort of global warming bomb and get the script ready for a possible sequel. People will be swarming to
download on bittorrent buy a ticket at the box office!
Below are Hollywood’s more popular ideas listed by category:
Games are hot
After a long period of books becoming a movie (Harry Potter), currently comic books characters become movies (Spiderman, The Hulk) our future holds… drum-roll… games that become movies! ‘Tombraider’ set it all off with taking in $300M worldwide. The movie was a big hit, although not really because of its script quality, but rather due to Angelina’s - hard-to-ignore - front assets. Hollywood is now attempting to achieve the same thing with numerous computer games listed below:
|Prince of Persia: The Sands of Time||
16 June 2009
15 October 2008
|Metal Gear Solid||
|Gears of War||
|The Sims: The Movie||
Sequels, Triquels and more
Hollywood loves sequels. Sequels provide some kind of guarantee of people coming to the theaters, thus bringing in the money. For example take American Pie. Everybody loved it, but while part 1 brought in ‘only’ $102M, the horrible sequel was somehow cashing 40% more (over $145M). It gets worse with the ‘Scary Movie’ series that reached a global total of $800M with 5 movies. Some noteworthy sequels:
|Transformers: Revenge of the Fallen – There just can’t be enough Megan Fox movies.||
24 June 2009
|The Incredible Hulk – So it’s like the Hulk, but than even more incredible?||
6 June 2008
|Sin City 2 – Doubt it will be nice since Jessica has become a mom.||
|The Brazilian Job – Like the Italian job/heist but now in Brazil. How did they come up with it?||
Every time I hope the moviemakers will just give up after yet another bad sequel, but somehow you guys can’t seem to get enough of paying money for a movie that really isn’t worth sitting through if you were the one getting paid to see it! Anyways it leads us to triquels, fourquels…
|Harry Potter and the Half-Blood Prince – Lord Voldemort! (hehe, I said it out loud)||
19 November 2008
|High School Musical 3: Senior Year – I’ll wait it out till Vanessa’s college freshmen years.||
22 October 2008
|Quantum of Solace – With a $230M budget is must be good. Right?||
31 October 2008
|Spider-Man 4 – I hope they get back together.||
|Harry Potter and the Deathly Hallows: Part I – So like… another one?||
19 November 2010
|Jurassic Park IV – They should’ve stopped four movies earlier.||
|Superman: Man of Steel – Our yearly Superman movie release…||
|Toy Story 3 – Wasn’t part 2 released in your local supermarket VHS section?||
18 June 2010
|Shrek Goes Fourth – They better keep Puss in boots!||
Destined for disaster
Sometimes you just need a tiny bit of information to predict if a movie will be hot or not. Somehow these movies just don’t sound like box office successes.
|Fast and Furious – Come on Vin, couldn’t you make the title more original?||
4 June 2009
|Watchmen – Based on a comic book.... next.||
5 March 2009
|Jennifer's Body – edit: wrongly listed here since it has Megan Fox||
|Justice League: Mortal – Official summary “Superman, Batman, Wonder Woman, Aquaman, the Flash and other superheroes unite to to fight against evil forces.” Enough said.||
|The First Avenger: Captain America – We’ve all had enough with superheroes by the time this gets released. We – the people - want movies based on supermodels or playmates.||
Retro = cool
Didn’t you love the A-team? Of course you did, everybody did! So Hollywood decided to make a movie of it (sound familiar?). Too bad all characters are overweight now, so they have to bring in other actors (won’t be the same without the real BA “I ain't gittin' on no plane!" Baracus will it?). Anyways go over the list below and decide what will go on your 2009-calendar.
|Dragonball – I’ve been practicing “Kamehameha” since.||27 March 2009|
|G.I. Joe: Rise of Cobra – Why the F is Brendan Fraser in this movie?||6 August 2009|
|X-Men Origins: Wolverine – Soon to be released in a $3 dvd-movie collection near you.||29 April 2009|
|The A-Team – Let’s hope that with Bruce Willis the movie will work out.||11 June 2009|
Money, money, money!
So what would you do, if you had 100+ million dollars? Apparently the producers of the movies below got that question and answered by saying “We will make a movie!”.
|Land of the Lost||
15 July 2009
27 March 2009
|The Curious Case of Benjamin Button||$100M|
22 May 2009
|Quantum of Solace||
31 October 2008
|Harry Potter and the Deathly Hallows: Part I||$250M|
Last week I visited the beautiful country Thailand, from Koh Pi Pi (island in the south) via Bangkok to Chang Mai and Pai (up north in the jungle) back to Bangkok: a huge city that spans as far as the eye can see from the 84th floor of the highest building.
Thai are overall very friendly and enthusiastic people, which provides a nice break from European so-called ‘hospitality’ (be careful for the elephants though!).
Next to a great experience of a first time in Asia, I’ve had to post something on what’ve experienced about domains in this non-english country. Although only a limited group of Thai are able to read, write and talk in English all website references go to english / ASCII characters .com (or .th)-domains. Not only companies servicing foreigners in tourism and transportation but also (local) governmental sites refer to .com-domains on billboards while the description is fully in the Thai language.
For me this emphasizes that although the number of domain top-level names will be increased significant soon and IDNs are targeted at a big portion of the web from Asia, a “ascii-domain.com” is strong and widely used even on the other side of the world.
PS: I wish I had taken some pictures as examples, but l had to pretend I had no work on my mind to my girlfriend.
PS2: Get a blue-ray player and buy the “Planet Earth” documentary from the BBC. 550 minutes of jaw-dropping entertainment.
While watching BBC’s Earth: The Power of the Planet I was pleasantly surprised I wasn't confronted with the typical nature show patronizing speech on how our actions have caused global warming (or cooling? Lets say climate change) and how very bad we are in using (fossil) energy. Instead of all this, the TV series presenter ended with a very insightful speech which I feel captures the whole ‘we should all save the planet’ debate into what it really is all about:
Our planet is really tough and there is nothing to suggest that it is going to change anytime soon in the long run. Earth can cope with anything we can throw at it.
We could clear all the jungles, but a jungle can regrow over a few thousand years. We could burn all earths’ fossil fuels, flooding the atmosphere with carbon dioxide. But even then, it will take the planet only a million years or so for the atmosphere to recover.
Even the animals we are wiping out will eventually be replaced by others equally rich in diversity. As a relentless work of evolution continues, it’s only a question of time. The earth will be just fine.
It’s not to say rapid changes we force on earth don’t matter, that is because humans operate on a different time scale. We have evolved to life in a world as it is now.
So in changing this world, we are altering the environment that has allowed the human race to thrive. We could be creating conditions that threaten that long term survival of our civilization.
So all this stuff about saving planet earth, well that is not the problem. Planet earth doesn't need saving - earth is a great survivor - it’s not the planet we should be worrying about, it’s us.
(transcribed: Dr Iain Stewart )