notes, learnings and reflections for future self reference
it is very important to know the type of opportunity you are tackling and to employ the right strategy. Opportunity shifts overtime between categories
This graph was originally referenced from article written by Casey Winters. While his article is primarily focused on how to improve the way growth teams operate, it also documents the phases a product will necessarily need to go through.
The sections in red are traditionally handled by the following roles:
Contrary to popular beliefs, if you listened very carefully between the lines of this 1 hour long lecture with the MBA students in MIT, Jobs did spend a lot of time and energy deeply listening to and understanding customer needs.
It would not have been possible to gain the insight and develop that very informed go to market strategy otherwise
There is always that 5 year window where the market is not really ready yet, where you need to spend a lot of time and effort to push the window open. Beyond that 5 year window, you have around a 5 year period to exploit the market before the next technology curve comes about.
If you hang around MIT or other research labs, you can kind feel it in your bones. Different pieces of technology that come together to make what was not possible before possible. Based on historical observation these things tend to converge in clumps. They are not linear.
The key challenge now for computing is the two segments of conflicting needs. The need for portability versus the need for power. Morse Law and technology innovation makes it inevitable that they converge.
The concept of GetData.IO was first conceived back in November 2012. I was rewriting one of my side project (ThingsToDoSingapore.com) in NodeJS back then. Part of the rewrite required that I wrote up two separate crawlers each for a different site which I was getting data for.
Very soon after I was done with the initial rewrite, I was once again compelled to write a third crawler when I wanted to buy some stocks on the Singapore stock exchange. I realized while the data for the shares were available on the site, they were not presented in a way that facilitated my decision making process. In addition to that, the other part of the data I needed were presented on a separate site and unsurprisingly not in the way I needed.
I was on my way to write my fourth crawler when it occurred to me, if I structured my code by cleanly decoupling the declaration from underlying implementation details, it is possible to achieve a high level of code re-use.
Two weekends of tinkering and frenzied coding later, I was able to complete the first draft of the Semantic Query Language and the engine that would interpret this query language. I was in love. Using just simple JSON, it allowed anybody the ability to declare the desired data from any parts of web. This includes data scattered across multiple pages on the same site or data scattered across multiple domains which could be joined using unique keywords.
Five years have past since, during this time, I brought this project through an incubator in Singapore with my ex-co-founder, tore out and rewritten major parts of the code-base that did not scale well, banged my head countless times on the wall in frustration due to problems with the code and with product market fit, watched a bunch of well-funded entrants came and went. To be honest, quite a few times I threw in the towel. Always, the love for this idea would call out to me and draw me back to it. I picked up the towel and continued ploughing.
It’s now June 2018. Though it has taken quite a while, I am now here in the Bay Area, the most suitable home for this project given to the density of technological startups in this region. My green card was finally approved last month. I have accumulated enough runway to allow my full attention on this project for the next 10 years. Its time to look forward.
The vision of this project is a multi-sided market place enabled by a Turing complete Semantic Query Language. The Semantic Query Language will be interpreted and executed upon by a fully decentralized data harvesting platform that will the capacity to gather data from more than 50% of the world’s websites on a daily basis.
Members can choose to participate in this data sharing community by playing one or more of the 4 roles:
From this vantage point, given its highly decentralized nature, it feels appropriate to deploy the use of block chains. The final part that needs to be sorted out prior to the deployment of blockchain to operate in full decentralized mode is figure out the “proof of work”.
Operations available in other database technologies will get ported over where appropriate as and when we encounter relevant use cases surfaced by our community members.
More as I dwell in this space, I see very clearly why it is only going to become increasingly important to have this piece of infrastructure in place. There are namely 3 reasons for this.
The next phase of our computing will rely very heavily on machine learning. It is a very data intensive activity. Given that established data siren’s like Facebook, Google, Amazon and Microsoft have over the past years aggregated huge tons of data, this have given them a huge unfair advantage which might not necessarily be good for the eco-system. We need to level the playing field by making it possible for other startups to gain easy access to training data for their machine learning work.
GDPR is a cumulation of concerns of data ownership that has been building for the past 10 years. People will increasing want to establish ownership and control over their own data, independent of the data siren’s use to house them. This means a decentralized infrastructure which people can trust to manage their own data.
Demand for engineering talent will only continue to increase as the pervasiveness of computing in our lives increase. The supply of engineering talents does not seem like it will be catching up and short fall is projected to continue widening till 2050. A good signal is the increasingly high premium paid to engineering talents in the form of salaries over the recent years. It’s just plain stupidity as a civilization to devote major portions of this precious engineering resource to the writing and rewriting of web crawlers for the same data sources over and over again. Their time should be freed up to do more important things.
Based on historical observation, I believe we are on the cusp of the very first inning in this space. A good comparison to draw upon is the early days of online music streaming.
Napster versus the music publishers is similar to how the lay of the land was back 5 years ago when Craigslist was able to successfully sue 3Tap.
Last year, LinkedIn lost the law suit against folks who were scraping public data. This is a very momentous inflection point in this space. Even the government is starting to the conclusion that public data is essentially public and Data Siren’s like any of the big Tech should have no monopoly over data that essentially belongs to the users who generated them.
Drawing further upon on the music industry analogy, the future of this space should look like how Spotify and ITunes operate in the modern day online music scene
…
If you want to scale beyond your own physical efforts, you will need to be able to convince others the importance of what you are doing. When you are successful at that you will be able to elicit their muscles to work for your own cause.
To be able to elicit their muscles, you will need to be able to tell a good story. If you read the book titled “Sapiens” by Yuval Harari, story telling is a technological innovation by the human species that has enabled large numbers of people to coordinate their efforts based around a single endeavor. It is one of the main causes for the human species’ predominance in our environment.
This innovation is enabled by our limbic brain which understands things based on narratives. Compelling narratives elicit an emotional response resulting in human motivation and corresponding action.
As a product manager working on the core product as opposed to growth, the primary focus is the narrative as opposed to the metrics.
The primary job of the executive is to “fight” for resources to further his agenda. The way he does so is by telling a compelling narrative to the company.
As opposed to a demagog, a good product manager tells a story, backs it up with data and delivers his promise. The demagog gets his resources by telling a story but never delivers anything of substance.
Learn to tell a story. That is a very important skill set acquire if you haven’t done so.
Inspired by conversations with Ved.
References