Kapow Katalyst Blog

In 2007, James Governor penned an article, Why Applications Are Like Fish and Data Is Like Wine, depicting how data gets better with age, while apps, like fish, begin to smell over time.

Earlier this year, Chuck Hollis from EMC offered his own wrinkle on the topic, also making a case for keeping the data (wine) and dumping those ‘lumbering apps’ (stale fish).

Both offer interesting reads, but too quickly dismiss the value of apps. Fish and wine go well together and depending on how you pair them, can make or break the experience or value of the meal. The same is true in the modern enterprise – the fish (application) is equally, if not more important than the wine (data), especially when the apps are kept “fresh and simple” with “many varieties to choose from…”

Consider if all the data trapped within applications were easily accessed and made even easier to interact with other applications.

There should be a straightforward way to interact and automate as many ‘fresh’ application sources as possible, in the shortest amount of time. IT organizations can no longer remain ‘comfortably numb’ in their avoidance of these agile Line Of Business (LOB) activities.

So, does the LOB really want a customary IT application integration fix or are they asking for something different? When the LOB says, “I really need access to this data now!” what they are really asking for is to interact with applications (or websites) in an impromptu manner in order to keep pace with emerging business dynamics and acquire key data advantages for the business.

The blistering pace and expansion of social media access, in all its forms (brand management, blog monitoring, anti-piracy, competitive intelligence, analyst research, prospect & customer mapping, consumer behavior analysis, marketing research, partner & reseller communication, on-line videos, risk compliance, legal monitoring, corporate reputation, background checks, R&D innovation research, consumer & customer research, sentiment analysis, predictive analytics, social CRM) makes the LOB integration demand on IT seem like an insurmountable task. The intensification of interactions from sources without API coverage (or even marginal API compliance) can no longer shackle the LOB data integration needs.

The traditional corporate IT blueprint has been to deploy long established application integration methodologies. This old design is already showing the strain of abandonment. I would propose a modern application integration approach consisting of the following components:

APPLICATION INTEGRATION PILLARS:

All four pillars are key to providing real-time integration, but they must also move toward a new form of LOB automation – a self-service component for the LOB. Ubiquitous access with the ability to quickly prototype and test application interactions are the most critical components to this offering. Applications need to be integrated in order for the LOB to interact and drive time-to-value (TTV) in the enterprise.

Adding to the turmoil for IT organizations are the hundreds (often thousands) of in-house applications constructed over the years. The vast majority of these have not been SOA enabled. Those ‘lumbering apps’ are also inclusive of the LOB need. There’s valuable data trapped in those in-house apps that must be freed.

Integrating to applications will require more frequent and varied connections with real-time and on-demand communications. These integrations may be more permanent links to applications or have ‘throw-away’ integration conditions.

The LOB is forcing the application integration challenge to the forefront of the IT stack. Applications are the real-time component of the raging BIG DATA frenzy. The day is coming, when the LOB will self-serve most of their application and website interactions.

The data wine cellars of the enterprise will be cared for by the IT stewards, but the Line Of Business wants FISH.

By: John Yapaola

IT  Executives  -  Dictators or the Army?

These are spectacular and exciting, historical times. Real-time, social/mobile connectivity is proving to be the defining catalyst for unparalleled (democratic?) change for the world. The ‘discovery costs’ for like-minded groups to find each other has been eradicated because of social tools like Twitter and Facebook, energizing powerful, collective action in near real-time.

The end is drawing near for dictatorial regimes that control their populace by muting innovation and slowing change which results in lasting and painful legacies for their countries.

Stretching the analogy into the Corporate IT Enterprise, we can see the influence of these real-time connectivity tools beginning to impact work-flow in corporations. The continued focus on top-down IT initiatives which impose stifling policies and regimented system architectures has led to the perception of IT inflexibility, and in some cases resulted in failed deliverables for their companies. In defense of IT, this present day world of rapid change is unprecedented and the impatient (intolerant even) business climate, driven by the needs of the LsOB (Lines of Business) for ‘instant’ information and connectivity has caught IT off guard.

‘Hold On’

The erosion of IT support, driven by the LsOB, is inspired by the demands for varied interconnectivity and real-time information access points, all of which force transformational change on the corporation. The chronicled, ‘Accepted IT Standards’ dogma is being challenged by these new market dynamics.

The Enterprise revolution has begun and will only gain momentum. If your employees, customers and partners are embracing and demanding these new methodologies for information sharing, then why aren’t you?

‘Taking it to the Streets’

The LsOB are now crowd-sourcing their demands on Enterprise IT and for the first time the LsOB have options. Twitter and Facebook provided the rails for change in Egypt and other countries to follow. Similarly, the Cloud Enterprise provides a conduit of independence for the LsOB in an effort to maximize their throughput. Social/Mobile/Cloud platforms are fast becoming the new “IT Standards”.

These are bottom-up initiatives with groundswell support of the LsOB, Enterprise Partners, Enterprise Customers and Vendors. The continued issuance of IT-driven mandates will only have diminishing returns for the Corporation, as the collective action of LsOB and their customers find more efficient ways to engage.

‘Revolutions can devour their children’

IT holds a significant role in the potential success of the Company. Just like the Armies of these modern-day revolutions, the IT organization can become the great facilitator for change by converging and adopting the demands of the LsOB and providing for orderly transitions. The ability to build ‘Employee Capital’ for the Company is a key building block for a successful and sustainable Corporation.

However, SPEED is the accelerant. As in Egypt, in just 18-days a stable 30-year dictatorial regime was easily overthrown. These are NOT merely ‘evolutionary’ times for IT.

Timing is as important as action.

…for the Times they are a Changin.

By: John Yapaola

“Your old road is rapidly agin’
Please get out of the new one
If you can’t lend your hand
For the times they are a-changin.”
-  Bob Dylan  1963

Fast forward almost 50 years to 2010 and this Dylan verse offers a modern-day perspective on recent developments in the corporate enterprise, as business users set down new demands (expectations) for the access and use of data.

Economic downturns can be painful, but often have a positive effect on companies by forcing clarity of business execution.  Hardened product/service deliverables and timelines suddenly become indisputable for managers and employees.  Expensive IT projects and initiatives previously thought to be valuable to the Company are ‘rudely’ terminated in favor of genuine efficiency demands, with the goal of lowering transactional costs of the Company.  The Lines of Business (LsOB) and IT organizations are given very specific marching orders: ‘Do more with less and do it faster and better than before!’

Are you “Waiting on IT” in an impatient business climate?

‘Clouds’ have gathered in the wake of the 2008 financial crisis and the perfect storm for change is imminent.  An Economic Downturn, Cloud Computing, and Social/Mobile media are driving rapid and transformational change in the way the LsOB will consume and distribute information in the modern enterprise to their partners and customers.

The business users’ insatiable need for access to data in all of its forms and from as many sources has created enormous friction for IT.    As the traditional gate-keeper of corporate data, IT’s role is to provide the means to make data-delivery easier for the business with integrated data being an expectation of the business user.

So why has data integration in the enterprise fallen so far behind in recent years?  In Clay Christiansen’s book “The Innovator’s Dilemma” there is a passage on new product development in the construction industry and the use of older machines prior to a disruptive innovation.  The excavator specifically triggered an analogy for me, as it relates to the current state of data integration in the enterprise.

The excavator is a massive piece of machinery to acquire, operate and maintain.  It requires highly-skilled specialists to run with the primary function of moving massive amounts of earth, in bulk.

As you can see in the graphic, the excavator approach to data integration (purple area) worked well, keeping pace with the distribution of data up until the late 90’s.  From there, the explosion of data from web apps commenced (red triangle) and has continued to accelerate year after year.   The data requirements of the business user changed, and as a result, IT has been overrun with rapidly expanding data demands as new data sources are discovered on what seems like a daily occurrence.  Using traditional, excavator-like machinery for these modern data integration projects has proven to be just too expensive and cumbersome.

In fairness to IT, the use of an excavator to address these dynamic integration projects was (and still is) an unfair disadvantage for IT.   Someone needed to invent the modern data integration equivalent of the BOBCAT, an agile, industrial strength, versatile, data integration platform that is inexpensive to operate and maintain; that can be quickly deployed for on-demand integration needs and used for the continuous progression of enterprise-extensible application lifecycles.

Today, Kapow introduces the Kapow Extraction Browser TM, the first and only web browser purpose built to extract, transform, integrate and migrate data from apps in the enterprise, on the web, and in the cloud.  Data is extracted from any layer in the application stack:  database, app logic and/or presentation layer.

Kapow KatalystTM – Browser-based data integration – a pragmatic new approach to data integration – No APIs required – No dependencies

The emergence of the browser as the defacto standard for data viewing has defined the linkage for business users to their data.  Brisk, mobile adoption and social media amplification are driving new data demands to the forefront of importance for the LsOB as the corporate enterprise ecosystem expands.   IT must begin to anticipate and adapt to these powerful business trends.

Business Users are fast-tracking the extensibility of the corporate enterprise by surfacing new applications for delivery on newer and less encumbered devices.  They will need assistance in dealing with these very real, dynamic and complex data movements. The opportunity for IT to step up and bridge this modern integration chasm for the LsOB has arrived.  Browser Integrated Data is the new road traveled – but get out of the way…

“…If you can’t lend your hand
For the times they are a-changin.”

By: John Yapaola

Social Media and BI are the sweet and sour, yin and yang, oil and vinegar topics of interest in BI these days.  Can the real-time, user-generated, free flowing tweets and online conversations of social media benefit traditional enterprise BI?

In the past week, Information Management published the following, Social Media Will Play a Big Part in BI’s Future.

No doubt the volume of social media is growing exponentially.  And surely, this data contains valuable information on competitive intelligence, product feedback, customer service, and even market trends.

But there’s a gap in social media data access.  Traditional BI tools can’t access all this unstructured data and present it in a usable format, let alone filter out all the noise.

What’s needed is an automated, flexible way to access hundreds or even thousands of sites in real-time, extract only the relevant content, add structure to the data, and load it easily into a database.  What’s needed is Web Data Services, and it exists today.

Social Media Data Access:

With hundreds of sites to monitor (most having no API access) and an already overburdened IT department, accessing social media data becomes the foremost hurdle to overcome.  With Web Data Services, all of this can be achieved with no coding.  Kapow robots (automated data collection processes) are easily created with visual point-and-click technology eliminating the need for complex, time-consuming coding and scripting.  If you can see the data in a web browser, Web Data Services can extract it.

Enriching Unstructured Data:

The trick is taking disparate text based tweets, comments, blog posts, online conversations, etc. and structuring them in a way that lets your analyst understand when it occurred, who said it, and how it applies to your keywords or hypothesis.  But getting there is harder than you might think.  Web Data Services surgically transforms unstructured social media web data to provide superior data quality without the noise.  Included, but not often talked about, is the ability to perform regular expressions (through a graphical interface), encoding and decoding, date formatting, string calculations, conditional expressions, numeric calculations, and multiple language support.

Making the data readily available:

Web Data Services makes it easy to output the structured social media data into multiple formats, such as a SQL database, vendor hosted database, Java or C# data structure, SOAP or REST Web service, RSS, CSV, or XML.

Social media is BI 2.0. It opens the doors to listen in on what people are saying about your brand, products and services, and also taps into untapped market opportunities and customer pain points.  So rather than reacting, you are out in front predicting future events and gaining first mover advantage.

By:  Rick Kawamura

Kate Gosselin on Dancing with the Stars Photo Credit: ABC

Can Social Media be used to predict the outcome of Reality TV shows such as American Idol and Dancing with the Stars?  We created Reality Buzz based on our real-time automated web data collection platform to find out.

Jennifer Zaino over at Semantic Web wrote a nice article that captures the essence of Reality Buzz and our process of using real-time social media web data to build intelligence in to predictive analytics:  Taking Sentiment Analysis to Dancing with the Stars and American Idol

Check it out.  And if you have the need to automate the access, collection, harvesting, scrubbing, grabbing or scraping or real-time web data to improve market or competitive analysis to improve your strategic decision making, we’re here to help.

By:  Rick Kawamura

While web data, especially social media data, grows exponentially, the vast array of opportunities for using real-time web data to improve analysis and decision making is limited only by your imagination.  These days, companies must incorporate Web data into their intelligence and analysis tools in order to compete. In some industries it’s a matter of survival.

Real-time data is where the answers are.  It’s where market and customer trends are immediately identifiable.  It’s where deals will be won and where winners will claim their trophies.

Reality Buzz

We recently built something 30M Americans can relate to – a way to predict American Idol and other reality show results based on data harvested from popular social media sites.

Every week on American Idol, contestants perform, their fans dial-in their support for their favorites, and the next day contestants are voted off the show.  During the performances, and for several hours after, fans tweet about and discuss their favorites online, showing support for the ones they want to see voted through to the next show.

We scrape thousands of pieces of web data from twitter, Facebook, forums and discussion sites around the web, apply sentiment analysis, analyze the data, and make predictions about the person(s) to be eliminated from the show, all in the span of a few hours.

We built the robots (automated web data collection processes) in a matter of hours.  Now they are automated to collect the data, transform unstructured data into structured data, and load it into a MySQL database.

In the last two weeks starting with the top 12 contestants, we’ve successfully predicted the American Idol contestant to be eliminated hours before the elimination show aired.  For more information on our latest predictions and to learn more, please visit Reality Buzz on Facebook.

Imagine what you could do for your business with Kapow’s Web Data Server and a few hours creating Kapow robots.  Real-time web data can fuel predictive analytics capabilities to give your company an unfair advantage.

Over 400 Kapow customers are jumping in with both feet.  What’s stopping you?  Learn more on our Kapowtech.com website, or contact us for a Free Trial of the Kapow Web Data Server.

By:  Rick Kawamura