Kapow Katalyst Blog

There has been so much hype around Big Data and now finally there is a $ amount associated with it – Jeff Kelly form Wikibon, an open-source-style community of industry analysts, delivered the much anticipated forecast estimating Big Data market will reach the $50 billion mark worldwide by 2017, growing from $5 billion in 2012. IDC with a more modest forecast still predicts 40% a year growth rate reaching close to $17 billion by 2015.

Wikibon expects a significant growth for Big Data as “the new definitive source for competitive advantage across all industries”. There are plenty of examples of where data-driven decision making has made a significant impact in retail, financial services, logistics, consumer packaged goods and even sports with the well-known Moneyball Big Data story. Shipping companies use data on delivery times and traffic patterns for route optimization while retailers leverage data analysis for predictive targeting to increase number of shoppers, store visits and spend per visit.

Our customers are turning blogs, forums and social media commentary as actionable predictors for stock performance allowing them to make significant economic gains by being the first to buy certain stocks before prices go up. Others in the logistics space are leveraging real-time data from multiple sources to better compete for business in a timely and more economical way. The possibilities for agility, innovation and overall business growth are infinite and will drive more investments, as indicated by the analysts’ estimates.

While a lot of the focus in Big Data is driven by the new economical ways to store and process large volumes of data, getting the data is the first step to any Big Data initiative and can be equally challenging. This data is often scattered across unlimited number of internal and external data sources. These include a long tail of social feeds, review sites and news sources, cloud applications, government web-based applications (federal regulations, public data on housing, marriages, foreclosures etc.), channels, suppliers and competitor’s sites. A majority of these sources are difficult to connect to and the data they contain is constantly changing.

Removing this barrier with access to relevant data regardless of source or format will open up even greater opportunities for growth in this new wave of technology revolution called “Big Data”. I can’t wait to see what other game-changing use cases and applications are on the horizon.

By: Hila Segal, Director, Product Marketing

Last week I attended a CIO Executive Leadership Summit here in Palo Alto. It was a great chance to interact with Silicon Valley IT execs driving transformation, innovation, and value creation in our rapidly changing connected world. Speakers included Tim Campos (CIO of Facebook), Kimberly Stevenson (CIO of Intel), Ralph Loura (CIO of Clorox), Mark Egan (CIO of VMWare), and Tom Keiser (CIO of Gap) just to name a few.

The event is structured to help IT leaders learn what it means to be transformational – to guide company-wide projects that result in significant changes to company strategy, value creation, revenue generation, and innovative products and services.

With the convergence of mobile, social, cloud and Big Data, CIOs cannot afford to ignore the importance IT has on the strategy, innovation, and ultimately business success of the companies they lead. Technology is core to a vast majority of initiatives that drive change.

Consumerization of IT is a passion of mine (and in particular the consumerization of integration), and there was no shortage of CIOs focused on the urgent need to make enterprise work environments equal to, if not better than, what many of their employees enjoy at home. Imagine that. Trying to be more convenient, efficient, and easy to use than what you can put together yourself from off the shelf products from your local electronics store.

But as IT departments look to provide a “consumer-like” enterprise environment for their employees, customers, and partners, why is there not a similar effort to make IT tools and platforms equally as easy to use? Whereas the iPhone “redefined” the smartphone market by resetting the consumer expectations of what a smartphone could be, when is someone going to “redefine” and “consumerize” the way IT departments integrate Big Data, Cloud, Mobile, and Social Media data and applications so enterprises can scale their businesses to not only keep up with the rapid pace of change their customers, partners, and employees demand of them, but to drive competitive advantage to innovate new products, develop new channels of revenue generation, and outwit their competition?

Transformational CIOs require transformational software. It exists, you just have to find it.

By: Rick Kawamura

In 2007, James Governor penned an article, Why Applications Are Like Fish and Data Is Like Wine, depicting how data gets better with age, while apps, like fish, begin to smell over time.

Earlier this year, Chuck Hollis from EMC offered his own wrinkle on the topic, also making a case for keeping the data (wine) and dumping those ‘lumbering apps’ (stale fish).

Both offer interesting reads, but too quickly dismiss the value of apps. Fish and wine go well together and depending on how you pair them, can make or break the experience or value of the meal. The same is true in the modern enterprise – the fish (application) is equally, if not more important than the wine (data), especially when the apps are kept “fresh and simple” with “many varieties to choose from…”

Consider if all the data trapped within applications were easily accessed and made even easier to interact with other applications.

There should be a straightforward way to interact and automate as many ‘fresh’ application sources as possible, in the shortest amount of time. IT organizations can no longer remain ‘comfortably numb’ in their avoidance of these agile Line Of Business (LOB) activities.

So, does the LOB really want a customary IT application integration fix or are they asking for something different? When the LOB says, “I really need access to this data now!” what they are really asking for is to interact with applications (or websites) in an impromptu manner in order to keep pace with emerging business dynamics and acquire key data advantages for the business.

The blistering pace and expansion of social media access, in all its forms (brand management, blog monitoring, anti-piracy, competitive intelligence, analyst research, prospect & customer mapping, consumer behavior analysis, marketing research, partner & reseller communication, on-line videos, risk compliance, legal monitoring, corporate reputation, background checks, R&D innovation research, consumer & customer research, sentiment analysis, predictive analytics, social CRM) makes the LOB integration demand on IT seem like an insurmountable task. The intensification of interactions from sources without API coverage (or even marginal API compliance) can no longer shackle the LOB data integration needs.

The traditional corporate IT blueprint has been to deploy long established application integration methodologies. This old design is already showing the strain of abandonment. I would propose a modern application integration approach consisting of the following components:

APPLICATION INTEGRATION PILLARS:

All four pillars are key to providing real-time integration, but they must also move toward a new form of LOB automation – a self-service component for the LOB. Ubiquitous access with the ability to quickly prototype and test application interactions are the most critical components to this offering. Applications need to be integrated in order for the LOB to interact and drive time-to-value (TTV) in the enterprise.

Adding to the turmoil for IT organizations are the hundreds (often thousands) of in-house applications constructed over the years. The vast majority of these have not been SOA enabled. Those ‘lumbering apps’ are also inclusive of the LOB need. There’s valuable data trapped in those in-house apps that must be freed.

Integrating to applications will require more frequent and varied connections with real-time and on-demand communications. These integrations may be more permanent links to applications or have ‘throw-away’ integration conditions.

The LOB is forcing the application integration challenge to the forefront of the IT stack. Applications are the real-time component of the raging BIG DATA frenzy. The day is coming, when the LOB will self-serve most of their application and website interactions.

The data wine cellars of the enterprise will be cared for by the IT stewards, but the Line Of Business wants FISH.

By: John Yapaola

IT  Executives  -  Dictators or the Army?

These are spectacular and exciting, historical times. Real-time, social/mobile connectivity is proving to be the defining catalyst for unparalleled (democratic?) change for the world. The ‘discovery costs’ for like-minded groups to find each other has been eradicated because of social tools like Twitter and Facebook, energizing powerful, collective action in near real-time.

The end is drawing near for dictatorial regimes that control their populace by muting innovation and slowing change which results in lasting and painful legacies for their countries.

Stretching the analogy into the Corporate IT Enterprise, we can see the influence of these real-time connectivity tools beginning to impact work-flow in corporations. The continued focus on top-down IT initiatives which impose stifling policies and regimented system architectures has led to the perception of IT inflexibility, and in some cases resulted in failed deliverables for their companies. In defense of IT, this present day world of rapid change is unprecedented and the impatient (intolerant even) business climate, driven by the needs of the LsOB (Lines of Business) for ‘instant’ information and connectivity has caught IT off guard.

‘Hold On’

The erosion of IT support, driven by the LsOB, is inspired by the demands for varied interconnectivity and real-time information access points, all of which force transformational change on the corporation. The chronicled, ‘Accepted IT Standards’ dogma is being challenged by these new market dynamics.

The Enterprise revolution has begun and will only gain momentum. If your employees, customers and partners are embracing and demanding these new methodologies for information sharing, then why aren’t you?

‘Taking it to the Streets’

The LsOB are now crowd-sourcing their demands on Enterprise IT and for the first time the LsOB have options. Twitter and Facebook provided the rails for change in Egypt and other countries to follow. Similarly, the Cloud Enterprise provides a conduit of independence for the LsOB in an effort to maximize their throughput. Social/Mobile/Cloud platforms are fast becoming the new “IT Standards”.

These are bottom-up initiatives with groundswell support of the LsOB, Enterprise Partners, Enterprise Customers and Vendors. The continued issuance of IT-driven mandates will only have diminishing returns for the Corporation, as the collective action of LsOB and their customers find more efficient ways to engage.

‘Revolutions can devour their children’

IT holds a significant role in the potential success of the Company. Just like the Armies of these modern-day revolutions, the IT organization can become the great facilitator for change by converging and adopting the demands of the LsOB and providing for orderly transitions. The ability to build ‘Employee Capital’ for the Company is a key building block for a successful and sustainable Corporation.

However, SPEED is the accelerant. As in Egypt, in just 18-days a stable 30-year dictatorial regime was easily overthrown. These are NOT merely ‘evolutionary’ times for IT.

Timing is as important as action.

…for the Times they are a Changin.

By: John Yapaola

“Your old road is rapidly agin’
Please get out of the new one
If you can’t lend your hand
For the times they are a-changin.”
-  Bob Dylan  1963

Fast forward almost 50 years to 2010 and this Dylan verse offers a modern-day perspective on recent developments in the corporate enterprise, as business users set down new demands (expectations) for the access and use of data.

Economic downturns can be painful, but often have a positive effect on companies by forcing clarity of business execution.  Hardened product/service deliverables and timelines suddenly become indisputable for managers and employees.  Expensive IT projects and initiatives previously thought to be valuable to the Company are ‘rudely’ terminated in favor of genuine efficiency demands, with the goal of lowering transactional costs of the Company.  The Lines of Business (LsOB) and IT organizations are given very specific marching orders: ‘Do more with less and do it faster and better than before!’

Are you “Waiting on IT” in an impatient business climate?

‘Clouds’ have gathered in the wake of the 2008 financial crisis and the perfect storm for change is imminent.  An Economic Downturn, Cloud Computing, and Social/Mobile media are driving rapid and transformational change in the way the LsOB will consume and distribute information in the modern enterprise to their partners and customers.

The business users’ insatiable need for access to data in all of its forms and from as many sources has created enormous friction for IT.    As the traditional gate-keeper of corporate data, IT’s role is to provide the means to make data-delivery easier for the business with integrated data being an expectation of the business user.

So why has data integration in the enterprise fallen so far behind in recent years?  In Clay Christiansen’s book “The Innovator’s Dilemma” there is a passage on new product development in the construction industry and the use of older machines prior to a disruptive innovation.  The excavator specifically triggered an analogy for me, as it relates to the current state of data integration in the enterprise.

The excavator is a massive piece of machinery to acquire, operate and maintain.  It requires highly-skilled specialists to run with the primary function of moving massive amounts of earth, in bulk.

As you can see in the graphic, the excavator approach to data integration (purple area) worked well, keeping pace with the distribution of data up until the late 90’s.  From there, the explosion of data from web apps commenced (red triangle) and has continued to accelerate year after year.   The data requirements of the business user changed, and as a result, IT has been overrun with rapidly expanding data demands as new data sources are discovered on what seems like a daily occurrence.  Using traditional, excavator-like machinery for these modern data integration projects has proven to be just too expensive and cumbersome.

In fairness to IT, the use of an excavator to address these dynamic integration projects was (and still is) an unfair disadvantage for IT.   Someone needed to invent the modern data integration equivalent of the BOBCAT, an agile, industrial strength, versatile, data integration platform that is inexpensive to operate and maintain; that can be quickly deployed for on-demand integration needs and used for the continuous progression of enterprise-extensible application lifecycles.

Today, Kapow introduces the Kapow Extraction Browser TM, the first and only web browser purpose built to extract, transform, integrate and migrate data from apps in the enterprise, on the web, and in the cloud.  Data is extracted from any layer in the application stack:  database, app logic and/or presentation layer.

Kapow KatalystTM – Browser-based data integration – a pragmatic new approach to data integration – No APIs required – No dependencies

The emergence of the browser as the defacto standard for data viewing has defined the linkage for business users to their data.  Brisk, mobile adoption and social media amplification are driving new data demands to the forefront of importance for the LsOB as the corporate enterprise ecosystem expands.   IT must begin to anticipate and adapt to these powerful business trends.

Business Users are fast-tracking the extensibility of the corporate enterprise by surfacing new applications for delivery on newer and less encumbered devices.  They will need assistance in dealing with these very real, dynamic and complex data movements. The opportunity for IT to step up and bridge this modern integration chasm for the LsOB has arrived.  Browser Integrated Data is the new road traveled – but get out of the way…

“…If you can’t lend your hand
For the times they are a-changin.”

By: John Yapaola

Social Media and BI are the sweet and sour, yin and yang, oil and vinegar topics of interest in BI these days.  Can the real-time, user-generated, free flowing tweets and online conversations of social media benefit traditional enterprise BI?

In the past week, Information Management published the following, Social Media Will Play a Big Part in BI’s Future.

No doubt the volume of social media is growing exponentially.  And surely, this data contains valuable information on competitive intelligence, product feedback, customer service, and even market trends.

But there’s a gap in social media data access.  Traditional BI tools can’t access all this unstructured data and present it in a usable format, let alone filter out all the noise.

What’s needed is an automated, flexible way to access hundreds or even thousands of sites in real-time, extract only the relevant content, add structure to the data, and load it easily into a database.  What’s needed is Web Data Services, and it exists today.

Social Media Data Access:

With hundreds of sites to monitor (most having no API access) and an already overburdened IT department, accessing social media data becomes the foremost hurdle to overcome.  With Web Data Services, all of this can be achieved with no coding.  Kapow robots (automated data collection processes) are easily created with visual point-and-click technology eliminating the need for complex, time-consuming coding and scripting.  If you can see the data in a web browser, Web Data Services can extract it.

Enriching Unstructured Data:

The trick is taking disparate text based tweets, comments, blog posts, online conversations, etc. and structuring them in a way that lets your analyst understand when it occurred, who said it, and how it applies to your keywords or hypothesis.  But getting there is harder than you might think.  Web Data Services surgically transforms unstructured social media web data to provide superior data quality without the noise.  Included, but not often talked about, is the ability to perform regular expressions (through a graphical interface), encoding and decoding, date formatting, string calculations, conditional expressions, numeric calculations, and multiple language support.

Making the data readily available:

Web Data Services makes it easy to output the structured social media data into multiple formats, such as a SQL database, vendor hosted database, Java or C# data structure, SOAP or REST Web service, RSS, CSV, or XML.

Social media is BI 2.0. It opens the doors to listen in on what people are saying about your brand, products and services, and also taps into untapped market opportunities and customer pain points.  So rather than reacting, you are out in front predicting future events and gaining first mover advantage.

By:  Rick Kawamura

Kate Gosselin on Dancing with the Stars Photo Credit: ABC

Can Social Media be used to predict the outcome of Reality TV shows such as American Idol and Dancing with the Stars?  We created Reality Buzz based on our real-time automated web data collection platform to find out.

Jennifer Zaino over at Semantic Web wrote a nice article that captures the essence of Reality Buzz and our process of using real-time social media web data to build intelligence in to predictive analytics:  Taking Sentiment Analysis to Dancing with the Stars and American Idol

Check it out.  And if you have the need to automate the access, collection, harvesting, scrubbing, grabbing or scraping or real-time web data to improve market or competitive analysis to improve your strategic decision making, we’re here to help.

By:  Rick Kawamura

While web data, especially social media data, grows exponentially, the vast array of opportunities for using real-time web data to improve analysis and decision making is limited only by your imagination.  These days, companies must incorporate Web data into their intelligence and analysis tools in order to compete. In some industries it’s a matter of survival.

Real-time data is where the answers are.  It’s where market and customer trends are immediately identifiable.  It’s where deals will be won and where winners will claim their trophies.

Reality Buzz

We recently built something 30M Americans can relate to – a way to predict American Idol and other reality show results based on data harvested from popular social media sites.

Every week on American Idol, contestants perform, their fans dial-in their support for their favorites, and the next day contestants are voted off the show.  During the performances, and for several hours after, fans tweet about and discuss their favorites online, showing support for the ones they want to see voted through to the next show.

We scrape thousands of pieces of web data from twitter, Facebook, forums and discussion sites around the web, apply sentiment analysis, analyze the data, and make predictions about the person(s) to be eliminated from the show, all in the span of a few hours.

We built the robots (automated web data collection processes) in a matter of hours.  Now they are automated to collect the data, transform unstructured data into structured data, and load it into a MySQL database.

In the last two weeks starting with the top 12 contestants, we’ve successfully predicted the American Idol contestant to be eliminated hours before the elimination show aired.  For more information on our latest predictions and to learn more, please visit Reality Buzz on Facebook.

Imagine what you could do for your business with Kapow’s Web Data Server and a few hours creating Kapow robots.  Real-time web data can fuel predictive analytics capabilities to give your company an unfair advantage.

Over 400 Kapow customers are jumping in with both feet.  What’s stopping you?  Learn more on our Kapowtech.com website, or contact us for a Free Trial of the Kapow Web Data Server.

By:  Rick Kawamura