This fall I am excited to be talking about Big Data at the Informatica Maximize Return on Big Data roadshow events across North America.

The theme of the roadshow is centered around solving the return on data equation which Informatica defines as the value of data divided by the cost of data:

Return on Data = Value on Data / Cost of Data

Specifically I am talking about how this applies to what is quickly becoming the most important and fastest growing data domain – “Big Interaction Data”. It is your customers’ blogs and tweets, your competitors pricing information and the data on your partners’ portals. It is unstructured, rapidly changing and provides unique insights that can turn your company into the market leader.

From being almost non-existing a few years back, Big Interaction Data is today growing explosively, with most of that happening on the internet and available as “Web Data”. My previous blog Dynamic pricing with automated competitive price monitoring provides a good example of how the use of Interaction Data can create huge differentiation, drive topline growth and turn companies into winners in their industry.

For many, understanding the huge importance of Big Interaction Data is just in its infancy, but I can assure you this is quickly changing. In my presentation, I give a few other examples of how companies leverage Interaction data. One example is about, Fiserv, a large financial services company who had to monitor point-in-time cash position across 300 partner banks for regulatory compliance. They had no other way of getting the data other than manually logging in to 300 portals and retrieving bank balances. Being able to get this type of data in an automated way was game changing for this organization. So as you can see Big Interaction Data is far more than just social media and provides significant value.

The other side of the equation is the “Cost of Data”. If the data is impossible to get, or very expensive to get, or cheap but out-of-date, then the return will be lower.

The most successful companies will be the ones that put data first, the most relevant data, and leverage technologies to get all that data at a low cost – in real-time.

Here is a snippet video of the presentation I gave in New York earlier this month:

On November 7th I will be presenting again in Atlanta, but you can also meet my colleagues at the San Francisco event on the same day. I will also be speaking in Dallas on November 13th if you happen to be in that area.

Come see me and let’s discuss the topic.

Stefan

Sales reps spend a lot of time searching for leads and sourcing critical information on their prospects. This information includes contact details, new hires, recently launched products, new locations, etc. According to a recent survey, “CSO Insights 2012 Impact of Data Access on Sales Performance Report: Why Big Data Should Be a Big Deal for Sales”, sales reps search and review on average as many as 15 different sources before calling a prospect. Among them business-oriented membership groups such as LinkedIn are used more than traditional internal information sources. Other sources include search engines, press releases, social networks and financial statements. More than 80% of participants feel challenged by the amount of data available and the difficulty of accessing it. Think about how much time is being wasted by sales reps searching for data instead of making calls.

At the same time, the “decision window”, defined by Aberdeen’s BI practice as the period of time during which information can materially impact a business decision, is shrinking. That decision window can range all the way from real-time to weeks or months. When asked how that window is changing, two-thirds of Aberdeen’s surveyed decision makers reported it is shortened indicating the increased urgency for relevant data.

According to CSO Insight survey, nearly 90% of sales organizations feel they are missing opportunities because of inability to effectively leverage all information available on prospects. It all comes down to getting the right data into the hands of the frontline managers who make daily critical decisions that impact the top-line – such as dynamically changing product prices ahead of the competition, monitoring social media to prevent a major product recall or predicting market trends for profitable investment decisions. According to McKinsey director David Court, in his interview “ Putting big data and advanced analytics to work”, powerful but simple tools for managers are a key success factor for transforming companies to exploit data that can provide a broader view on the business, operations and customers.

Are you missing opportunities or are you putting the right business intelligence into the hands of the people responsible for growing your topline?

By: Hila Segal, Director, Product Marketing

The age of connectivity is here. Smartphone and tablets have given consumers the power to connect with businesses from nearly anywhere. Companies that take advantage of the heightened connectivity are at a significant advantage, according to a Forbes article written by Marita Scarfi, CEO of digital ad agency Organic. The main challenge is that social media Big Data is unstructured, and, traditionally, it hasn’t been easy or cost effective to use that social media information to measure consumer engagement. However, more and more consumers are beginning to interact with brands through social media websites, making them a valuable source of business intelligence.

“To meet those ROI needs, we need to turn social media into actionable data or we may end up harming the brand instead of helping it,” Scarfi wrote. “The Holy Grail of marketing has always been creating great emotional engagement with utility. Social media has the power to connect consumers with brands in ways that go far beyond even the best word of mouth. But we still need to know if that emotion is turning into profit.”

Social web data such as blogs, forums, community sites and customer review sites provide extremely valuable insight about customers like and dislikes, brand loyalty and other related sentiment information. At the same time, these portals and web-based data sources are typically difficult to access and integrate due to the lack of available APIs.

With only 0.0017% of all websites having APIs – integrating valuable external web-based big data for your business becomes a key challenge.

By: Kapow Software Marketing

Since the Beijing Olympics, the number of Facebook users has surged to 900 million from just 100 million. There are over 500-million active users on Twitter, compared to just 6 million in 2008. The explosion in the use of social media and smartphones in the past four years means the London 2012 Olympics will be the most liked, tweeted, pinned, blogged and reported event in history. Like no other medium, the social channels will provide fans with access to athletes, results, special events and details about the Olympic Games.

This is the new world of informed consumers, empowered with information and freed from the reliance on traditional media and news providers. Now not only can I instantly know who won the gold in 200-m freestyle right when it happens but I also have access to information about this athlete’s previous wins, exact times for the last 3 Olympic Games and maybe even his favorite ice-cream flavor. Wouldn’t it be great if I can get all that with a single click? Pulling in athlete’s tweets, maybe weather data for the day of the race, info from Wikipedia and other sources about competing athletes — without entering these sites one by one. That would be golden….

But it can even get better. Automatically getting information on my favorite athletes, updates on their results, press mentions and links to photos and videos of key performances. Information collected for me and delivered via my preferred choice of delivery like text or email.

As the world becomes always-on always connected, we will see consumers demanding self-service access to information via tools that require less time, less training and less clicks. Let the games begin!

By: Hila Segal, Director, Product Marketing

In his recent book, “Thinking Fast and Slow”, Princeton University Nobel Laureate, Daniel Kahneman, introduces us to the principle of Theory Induced Blindness – the adherence to a vulnerable belief, even though a counterexample may exist, about how something works that prevents you from seeing how it REALLY works. So once you have accepted a theory, it is extraordinarily difficult to notice its flaws, trusting instead the community of experts who have already accepted it.

Over the years, it seems that the realm of the Corporate IT community has been rooted in protecting ‘Technology Theory’ for the company. As the experts of design, specifically for application development (AD) and information & integration access, the long standing belief was that stitching together applications and processes required precise, well-defined access points, usually in the form of APIs. Today, even though the business users have been pained and slowed by the scarcity of this restrictive practice, the blindness still prevails for corporate IT and (sadly) its customers, as they struggle onward.

In truth, corporate IT organizations have never really had to engage on a competitive landscape for their services. As the once empowered purveyors of corporate information shielded by exclusionary practices, IT has had a nice run without much disruption. But even after so many years of Big Budgets, Big Hardware, Big Applications, Big Consulting and the intense scrutiny from company employees and executives, the reoccurring question persists…

Why are there still so many disconnects in the enterprise?
The reality check has arrived in the form of real competition. Unfortunately for IT, the challenge has come from within – their own customers, the employees of the company – the same group that they once served are now fiercely engaged, combative and driving change.

Mass Amateurization of IT

The high technology barriers and transactional costs that once made the IT organization a mystical entity requiring an ‘IT Blessing’ for even the most remedial of tasks is being dismantled. A new kind of employee is onboarding into the company. A more technology savvy millennial worker is charting a new course for the company and the ‘Known World’ of information access and sharing is proving to be a more expansive ocean of knowledge than was previously taught.

Encompassing social, mobile and cloud norms, BIG Data has burst onto the enterprise, seemingly as a shock to IT. Suddenly from the rank and file of the company, we see the rise of very qualified, amateur IT “professionals” – the line of business workers – seeking immediate answers. This BYOD employee is challenging long entrenched IT values and expertise while intrinsically focused on creating innovation and competitive upside for the company.

This is a defining moment for IT as the demystification of technology (and IT) occurs. The blunt realization that mass amateurization has swept through one’s profession should be a transformational event for IT workers, because the deconstruct of traditional corporate IT organizations is underway.

Connect Value

Falling transactional costs and technology barriers will allow for more experimentation by the business users and there will be failure as a matter of course, but this will be part of the learning process. The ability to ‘Fail Fast & Learn From’ is what these millennial workers expect. IT must recognize and support this new and important business style because if not, these workers will find new like-minded partners that do.

The real stewards of the corporation are its trusted employees. If IT organizations are to become agents of change, they must come to grips with the business requirements of their fellow employees to connect to any source, in near real-time, without the limitations of API dependencies.

In parallel, the intuitive nature for people to ’self-serve’ is a fundamental belief of modern, web-based applications. ‘No-Dependency’ models are critical for timely information access and sharing. So there is a reasonable expectation that Integration-as-a-Self-Service™ will move closer to reality, as business users strive for more autonomy to experiment.

There is some good news for the competitive IT Professional. When real, impactful innovation occurs, it raises the baseline knowledge set and standards of the company and thus the need for genuine, highly-skilled specialists move to the forefront. Those IT Professionals that add high value to the business will be in the greatest demand and will become leaders and invaluable assets in transforming the company.

But in order to lead, IT leaders must first prove they have cured their blindness by finding the ‘counterexample’ platform for agile access and collaborative information exchange.

Posted by: John Yapaola, CEO
Thanks to Colin Camerer, for the title inspiration.

I’m flying back from the InformaticaWorld event in Vegas where we have announced a strategic partnership with Informatica and the introduction of Informatica PowerExchange for Kapow Katalyst, for harvesting even bigger web data for strategic gain. Great conversations with leading companies from multiple industries, including retail, hospitality, financial services, insurance, pharmaceuticals, manufacturing, media and entertainment, consumer goods — you name it! In today’s hyper-competitive business climate all of these companies are quickly recognizing the huge importance of information from the broader web to get 360 views of their customers or deeper insight into market trends.

In a recent interview, Filippo Passerini, the CIO of P&G, talked about the importance of Big Data as a key differentiator. He also recognizes the value of extracting key business indicators and predictors from the data that will allow P&G be more accurate in the way they go to market with new products. But with that recognition, companies are also faced with the difficulty to access and integrate data from what we call the “long-tail” of web-based data sources.

When you look at the social web landscape as depicted here, beyond Facebook and Twitter, each category is a goldmine of its own from blogs and sharing sites to community platforms. These sources are the real treasure trove in the full sense of the term as they are “gold found hidden underground,” and getting it is difficult and requires going on a real “treasure hunt,” mostly because they do not have any available APIs.

This is the core value proposition of our partnership with Informatica: extending the reach of Informatica PowerExchange and the broader Informatica portfolio with the “hard-to-get” but extremely precious big web data that enables the “social enterprise.”

By: Hila Segal, Director, Product Marketing

GigaOm guru Om Malik goes so far as to call data without proper context “dirt” in a recent blog post, and truly some of the large piles of data gathered by companies today are nothing more than a waste of storage space. It’s a tough job for a BI system – and the analyst working with it – to make sense of data that has lost its accompanying metadata. Especially with huge amounts of data, it is critical that it can be processed in an automated fashion without requiring human intervention or interpretation for every single datum.

There are many products emerging in the big data space where the focus is on driving huge volumes of data into your (local or cloud-based) storage systems. Much of this data is erroneous (due to “smart” algorithms that often turn out not to be less smart than advertised) and isn’t annotated with the relevant metadata that could make the difference between relevant information and garbage data.

Along the same lines, Paul Michaud of Nebility discusses the Garbage In = Garbage Out (GIGO) principle. Analysis becomes meaningless if it is done on top of inconsistent or incorrect data. Don’t get lost in the “big data” frenzy; data isn’t just data. It has to be meaningful, annotated, and consistent.

This requires a powerful ETL tool such as Kapow Katalyst that has rich transformation capabilities and can acquire the proper metadata as the data is extracted, or even add new relevant metadata based on complex business rules. And rather than just pulling random petabytes of data in, you should carefully consider what you wish to accomplish with the data and what requirements that poses on the quality of the data, the number of data sources that must be combined, and the current degree of consistency between these data sources.

It’s tempting to measure your big data strategy by the volume of data consumed, but that would be about as meaningful as judging gourmet restaurants by calories per serving. Focus on accessing, transforming, and delivering the highest quality of data for superior decision-making and competitive advantage.

By: Anne-Sofie Nielsen

It’s Friday morning in sunny California. I’m heading to the office, listening to some good music. Usually I avoid the busy US-101 going south to Palo Alto and take the nicer I-280 route. This Friday I guess I was feeling lucky or simply tired of sifting through traffic reports and data feeds. I simply wanted the answer: 101 or 280? I took 101 based on a gut feeling. I was wrong. What usually takes 30 minutes became 1 hour due to a gravel spill on the road. There went 30 precious minutes all for making a “non-data-driven” decision.

As I was stuck in traffic regretting my decision to take 101, it made me think about data and how it is becoming the new currency. There is so much more data at our disposal as individuals for the decisions we make every day from traffic to shopping to booking a hotel or selecting a physician – which makes our lives better by saving time, money or simply feeling better informed. But at times this can be a little overwhelming when all we want are the answers.

When you think about what’s at stake for organizations it’s no wonder Big Data is becoming such a headliner and companies such as the world’s largest retailer, Wal-Mart, are turning to long-tail data on the web and social networks to glean insights into what consumers want in order to better compete with its online rival Amazon. But how much data do you really need to be able to make smarter decisions? How do you bridge the gap between increasing volumes of information and getting the right answer to the right question at the right time that will make you smarter and more successful?

In his article published in Forbes, Christopher Frank, Vice President in Global Marketplace Insights at American Express, talked about the “IWIK” (“I wish I knew”) discussion to identify wants and needs with respect to Big Data while uncovering the strategic, essential questions that can be answered through this information. I couldn’t agree more with Frank’s bottom line: “In the new reality of big data, hyper-competitive businesses are linking and labeling perception information, transactional data and layering in social behaviors and then using the IWIK to define the data rails to shape analytic-based decisions. Therefore, the focus should be less on ‘big’ data and more on ‘essential’ data”.

I wish I knew about the gravel spill on Friday…. Let’s start focusing on the data that matter the most for our strategic decision making and less about the volumes.

By: Hila Segal, Director, Product Marketing

My good friend Mik just gave his wife an iRobot for Christmas, and she loves it. The iRobot is intelligent enough not to vacuum the same place twice, avoid furniture, and recharge itself in the wall when running out of power. No more vacuum cleaning and instead Mik’s wife can spend her time painting, which is her great talent and passion.

Those of us with the latest iPhone 4S, love Siri. The kind female voice that helps you automatically perform tasks like sending a text reminds me to turn the oven on when I get home, for example, and even learns to do personal tasks for me like calling my wife.

Then there is the new service on the web, IFTTT (If This Then That). What a cool service! We don’t get a lot of rain in California, but when we do I’d like to know the day before — automatically. This is what IFTTT does very well. All I had to do was go to www.ifttt.com, select “rain in Palo Alto” for “If this”, and then “send me an email” for “then that”, and now I always know a day ahead when it’s going to rain.

iRobot, Siri, and IFTTT are just the beginning of a growing number of “intelligent robots” which will help us with our lives both in the physical and virtual world of the Internet.

Just as the Industrial Revolution automated the tedious tasks of weaving clothes, hammering hot iron into a horseshoe, and even building a new car, the information overflow and busy life of today has created demand for a next-generation Industrial Revolution, one built on services driven by intelligent software and smart devices with powerful CPUs.

I just read an article on AppleInsider about future updates to Siri. The article gives a hint to where Apple is going, and I predict that soon a next-generation Siri will become our 24/7 personal assistant, monitoring anything we want on the Internet and automatically performing actions such as buying a ticket for the next football game when one becomes available, or notifying us where to get the cheapest gas nearby when driving down the highway.

Soon all of the manual processes derived from the increasing information overflow — A. I. and the Next American Industrial Revolution on ptucker’s blog at the World Future Society website.

You might find this article interesting as well: Why it’s China’s turn to worry about manufacturing on the Berkley Blog.

Let’s go paint a beautiful picture.

By: Stefan Andreasen

There has been so much hype around Big Data and now finally there is a $ amount associated with it – Jeff Kelly form Wikibon, an open-source-style community of industry analysts, delivered the much anticipated forecast estimating Big Data market will reach the $50 billion mark worldwide by 2017, growing from $5 billion in 2012. IDC with a more modest forecast still predicts 40% a year growth rate reaching close to $17 billion by 2015.

Wikibon expects a significant growth for Big Data as “the new definitive source for competitive advantage across all industries”. There are plenty of examples of where data-driven decision making has made a significant impact in retail, financial services, logistics, consumer packaged goods and even sports with the well-known Moneyball Big Data story. Shipping companies use data on delivery times and traffic patterns for route optimization while retailers leverage data analysis for predictive targeting to increase number of shoppers, store visits and spend per visit.

Our customers are turning blogs, forums and social media commentary as actionable predictors for stock performance allowing them to make significant economic gains by being the first to buy certain stocks before prices go up. Others in the logistics space are leveraging real-time data from multiple sources to better compete for business in a timely and more economical way. The possibilities for agility, innovation and overall business growth are infinite and will drive more investments, as indicated by the analysts’ estimates.

While a lot of the focus in Big Data is driven by the new economical ways to store and process large volumes of data, getting the data is the first step to any Big Data initiative and can be equally challenging. This data is often scattered across unlimited number of internal and external data sources. These include a long tail of social feeds, review sites and news sources, cloud applications, government web-based applications (federal regulations, public data on housing, marriages, foreclosures etc.), channels, suppliers and competitor’s sites. A majority of these sources are difficult to connect to and the data they contain is constantly changing.

Removing this barrier with access to relevant data regardless of source or format will open up even greater opportunities for growth in this new wave of technology revolution called “Big Data”. I can’t wait to see what other game-changing use cases and applications are on the horizon.

By: Hila Segal, Director, Product Marketing