Kapow Katalyst Blog

I’m flying back from New York where I presented “The Moneyball Approach to Big Data – Creating an Unfair Advantage” at the Wall Street Technology Association’s Hot Technologies Forum. Big Data is an area technologists are curious about, but I’m concerned there’s a “wait and see” approach. My job is to create value for my customers, and I’d hate for any of you to miss out on this opportunity.

Skepticism or “late adopter” mentality is understandable – if you want to forego a low-risk, high-reward opportunity and let your competition gain the advantage. Everyone is benefitting from Big Data in some form or another – most probably don’t even know it. But believe me, there are 100s of scenarios I could walk you through that could save your company millions of dollars, grow revenue double digit percentages, create more personalized products that delight your customers, automate real-time feedback on your brand, products, and competitor prices, create your own custom research that gives allows you to see trends before your competitors, and overall make you a much more agile business that scales with your new found vigor and growth.

What’s the secret to Big Data rewards? “Relevance”, “Access”, “Intelligence” and “Action”.

The most common definition I’ve seen for Big Data relates to the 3 Vs:

• Volume: it’s Big – terabytes and petabytes
• Variety: it comes in many forms – internal, external, structured and unstructured
• Velocity: it grows fast and changes quickly – making real-time capture and action hugely important

And this is always supported by numbers showing how gynormous Big Data is:

• The New York Stock Exchange creates 1 terabyte of data per day (InformationWeek)
• 10,000 payment card transactions are made every second around the world (American Banker)
• 30 billion pieces of content shared on FB every month (McKinsey)
• Twitter feeds generate 8 terabytes of data per day (InformationWeek)

Before you go out and buy more storage, consider what you want to do with it. If there are 200M tweets a day equaling 8 terabytes of data, but only 1000 of the tweets relate to your product or company, do you need to store and analyze all 8 terabytes every day? Although Big Data is big, don’t get caught up in all the massive numbers. Stick with what’s relevant to your business.

Forrester Analyst Brian Hopkins made a great point in his blog “Big Data will help shape your markets next big winner”, stating that Forrester estimates enterprises use only 5% of their available data. So the playing field is wide open for anyone to quickly take advantage of the 95% they’re currently ignoring.

But slow down there pardner. Sybase published Big Data, Big Opportunity that stated, “for the median fortune 1000 company… a 10% increase in usability of data translates to an increase of $2.01B in total revenue per year, [and that] a 10% increase in accessibility to data translates to an additional $65.67M in net income per year.” So don’t think you have to go from 5% to 100%. You really only need to go from 5% to 5.5%.

The internet plays a huge role in the rapid growth of Big Data, giving individuals the ability to post and upload immense amounts of pictures, text, video, and mobile data, and businesses the channel to offer access to customers and partners through web-based applications (think Oracle, salesforce.com, social media, procurement, logistics, publishers, and so on).

In reviewing other articles about Big Data, despite all the discussion around the massiveness of Big Data, I didn’t find a single article mentioning the difficulty of accessing the data spread throughout all these applications. This is a HUGE POINT to understand because you are SOL if you can’t access the data you need. If I told you I could guarantee any app or data you can see in your web browser (customer data, bank transactions, twitter, blogs, supply chain vendors, government data, competitor prices, etc.) could be automatically accessed and loaded into the app, database, or spreadsheet of your choice, how many game-changing Big Data projects could you think of? Point-in-time cash position understanding of billions of dollars across 300 banks? No problem. Monitor competitor pricing on 50,000 SKUs every day? Simple. Automate a twenty-three step manual invoicing process to get paid millions of dollars 2 days faster? Done. Real-time, automated access to the data you need is the key to success with Big Data. Lest you think this all fantasy, learn how Kapow Katalyst Application Integration Platform provides real-time access to Big Data.

There’s huge difference between “I have terabytes of data – videos, satellite pictures, social media conversations, and research reports” and “I know where Osama Bin Laden is”. It’s Data vs. Intelligence. Data is useless if you can’t extract meaningful intelligence from it. And the quality of your intelligence is most likely much less dependent on the volume than the relevance and ability to access it.

And the whole point of having relevant, accessible, intelligent Big Data is that it is actionable. Otherwise it’s just a recommendation or a strategy without execution What’s incredibly cool about Big Data and the web-based nature of so much of it is that just as easily as you can access anything you can see, you can just as easily transform the data, perform an operation on it, and automate a resulting action for you. Huh? Here’s an example. You know consumers and even your B2B purchasers research prices online and that loyalty to any one vendor has deteriorated as buyers have more pricing knowledge a search and mouse-click away. But you are smarter than your competitors because you’re already doing the extra 10%. So you set up automated monitoring of your competitor’s pricing, and when their price drops below yours your Big Data Integration Platform calculates the difference plus 10%, logs into your ecommerce site and adjusts your prices automatically, all within a few ticks of the clock.

And the beauty is that this can all be set up in hours, if not a few days, and you don’t have to bring in an army of developers or consultants to create custom code to do any of this.

So let the Big Data party begin. Kapow Software is here to help. To learn more about Big Data Solutions or to set up a Big Data Sales Consultation, click either link, because you’ve read this far and deserve it!

By: Rick Kawamura

In 2007, James Governor penned an article, Why Applications Are Like Fish and Data Is Like Wine, depicting how data gets better with age, while apps, like fish, begin to smell over time.

Earlier this year, Chuck Hollis from EMC offered his own wrinkle on the topic, also making a case for keeping the data (wine) and dumping those ‘lumbering apps’ (stale fish).

Both offer interesting reads, but too quickly dismiss the value of apps. Fish and wine go well together and depending on how you pair them, can make or break the experience or value of the meal. The same is true in the modern enterprise – the fish (application) is equally, if not more important than the wine (data), especially when the apps are kept “fresh and simple” with “many varieties to choose from…”

Consider if all the data trapped within applications were easily accessed and made even easier to interact with other applications.

There should be a straightforward way to interact and automate as many ‘fresh’ application sources as possible, in the shortest amount of time. IT organizations can no longer remain ‘comfortably numb’ in their avoidance of these agile Line Of Business (LOB) activities.

So, does the LOB really want a customary IT application integration fix or are they asking for something different? When the LOB says, “I really need access to this data now!” what they are really asking for is to interact with applications (or websites) in an impromptu manner in order to keep pace with emerging business dynamics and acquire key data advantages for the business.

The blistering pace and expansion of social media access, in all its forms (brand management, blog monitoring, anti-piracy, competitive intelligence, analyst research, prospect & customer mapping, consumer behavior analysis, marketing research, partner & reseller communication, on-line videos, risk compliance, legal monitoring, corporate reputation, background checks, R&D innovation research, consumer & customer research, sentiment analysis, predictive analytics, social CRM) makes the LOB integration demand on IT seem like an insurmountable task. The intensification of interactions from sources without API coverage (or even marginal API compliance) can no longer shackle the LOB data integration needs.

The traditional corporate IT blueprint has been to deploy long established application integration methodologies. This old design is already showing the strain of abandonment. I would propose a modern application integration approach consisting of the following components:

APPLICATION INTEGRATION PILLARS:

All four pillars are key to providing real-time integration, but they must also move toward a new form of LOB automation – a self-service component for the LOB. Ubiquitous access with the ability to quickly prototype and test application interactions are the most critical components to this offering. Applications need to be integrated in order for the LOB to interact and drive time-to-value (TTV) in the enterprise.

Adding to the turmoil for IT organizations are the hundreds (often thousands) of in-house applications constructed over the years. The vast majority of these have not been SOA enabled. Those ‘lumbering apps’ are also inclusive of the LOB need. There’s valuable data trapped in those in-house apps that must be freed.

Integrating to applications will require more frequent and varied connections with real-time and on-demand communications. These integrations may be more permanent links to applications or have ‘throw-away’ integration conditions.

The LOB is forcing the application integration challenge to the forefront of the IT stack. Applications are the real-time component of the raging BIG DATA frenzy. The day is coming, when the LOB will self-serve most of their application and website interactions.

The data wine cellars of the enterprise will be cared for by the IT stewards, but the Line Of Business wants FISH.

By: John Yapaola

Do you dare to step out of your comfort zone? The world is changing at a faster and faster pace, especially in the world of IT.

Driven by the explosion of internet connectivity, distribution of data sources, and the number of legacy and networked applications, impatient business demands for IT projects are growing both in numbers and in complexity.

Line of Business needs projects delivered yesterday and with data as fresh as in the original data source. “Rotten” data is no longer acceptable.

In a situation like this, the traditional methods of working often fall short, something that can be very difficult to realize when changes happen gradually. Things don’t get out of control from one day to the next, but over months and years, until it’s too late.

I tried to illustrate the situation in the 3D graph above. The area of control (blue) is gradually turning out of control (red).

• Data demand has moved from batch to real-time
• Complexity has increased with the distribution of applications and data source
• Business urgency requires unrealistic time-to-market goals
• Number of IT projects is growing, but the budget is not

In a changing environment like this, we constantly need to challenge our processes and tools.

So my question to you is: “Are you ready to change the way you do things? Do you dare to step out of your comfort zone?”

If not, you’ve got a problem. You might not see this problem on a daily basis; it’s one of those problems that slowly sneaks up on you until one day it’s too late to fix.

It’s happened many times before, just look at DEC (Digital Equipment Corporation), the hugely successful mid-size computer company back in the 80s which basically died overnight; they did not step outside their comfort zone and subsequently disappeared.

Kapow Software’s revolutionary innovation, the Kapow Extraction Browser (which by the way is the only purpose-build integration browser on the market), has proven to help hundreds of IT projects overcome these challenges and make an easy jump from the blue to the red area.

By leveraging the fact that all applications today either have an API or GUI, where data and transactions can be accessed and controlled, means you no longer are dependent on the data and application owners. That’s a big deal. No more rewriting of legacy apps and no more begging the application owner to provide APIs. Just go do it.

That is a new way of thinking and new paradigm. Are you ready for it?

Not only does it work, it’s proven by more than 500 customers worldwide. It’s also delivering incredible business value and business agility among your competitors.

Step out of your box and try Kapow Software yourself!

By: Stefan Andreasen

Cloud Integration is more than just pre-existing connectors and SalesForce integration

As companies move their IT infrastructure and business applications to SaaS and the cloud it creates increasing need “cloud integration”, the ability to integrate data between applications in the hybrid world of internal apps, cloud apps and business partner apps.

Many Cloud Integration companies claim to offer a complete solution for integrating cloud and SaaS apps, but I claim they are all incomplete. Why?

If you check their demos and use-cases you quickly realize they provide a solution that only works if you have access to existing (and documented) web service APIs. And consequently, most of their examples are centered around salesforce.com integration and salesforce.com APIs.

This approach does not apply to the real cloud world – a world that is far from homogenous, but rather a hybrid, fragmented, and distributed world.

A cloud integration solution is only complete if it can integrate all applications in the cloud, whether they have Web Service APIs or not.

Today there are more than 200 million websites/applications on the internet, and many have complex features and data structure behind them. In this more holistic, big picture, only a fraction of these millions of websites are covered by documented APIs, making most traditional cloud integration solutions all but incomplete. The likelihood that your next cloud integration project will not be covered by “standard” connectors is very high, and therefore you need a cloud integration platform that can integrate to any layer in the application stack: database (SQL), web service (SOAP, REST), or through the presentation layer (HTML, AJAX).

Based on the patended Kapow Extraction Browser that leverages any HTML/AJAX application interface as an API if no web services API is present, Kapow Katalyst is the only Cloud Integration platform that has a complete data extraction, data integration, data transformation and data migration solution for your cloud integration challenges.

Proven by more than 500 customers world-wide, Kapow Katalyst is the only cloud integration platform that provides total connectivity and total data delivery in the cloud, in the enterprise and with your business partners.

Are you ready for your next cloud integration project?

By: Stefan Andreasen

“Your old road is rapidly agin’
Please get out of the new one
If you can’t lend your hand
For the times they are a-changin.”
-  Bob Dylan  1963

Fast forward almost 50 years to 2010 and this Dylan verse offers a modern-day perspective on recent developments in the corporate enterprise, as business users set down new demands (expectations) for the access and use of data.

Economic downturns can be painful, but often have a positive effect on companies by forcing clarity of business execution.  Hardened product/service deliverables and timelines suddenly become indisputable for managers and employees.  Expensive IT projects and initiatives previously thought to be valuable to the Company are ‘rudely’ terminated in favor of genuine efficiency demands, with the goal of lowering transactional costs of the Company.  The Lines of Business (LsOB) and IT organizations are given very specific marching orders: ‘Do more with less and do it faster and better than before!’

Are you “Waiting on IT” in an impatient business climate?

‘Clouds’ have gathered in the wake of the 2008 financial crisis and the perfect storm for change is imminent.  An Economic Downturn, Cloud Computing, and Social/Mobile media are driving rapid and transformational change in the way the LsOB will consume and distribute information in the modern enterprise to their partners and customers.

The business users’ insatiable need for access to data in all of its forms and from as many sources has created enormous friction for IT.    As the traditional gate-keeper of corporate data, IT’s role is to provide the means to make data-delivery easier for the business with integrated data being an expectation of the business user.

So why has data integration in the enterprise fallen so far behind in recent years?  In Clay Christiansen’s book “The Innovator’s Dilemma” there is a passage on new product development in the construction industry and the use of older machines prior to a disruptive innovation.  The excavator specifically triggered an analogy for me, as it relates to the current state of data integration in the enterprise.

The excavator is a massive piece of machinery to acquire, operate and maintain.  It requires highly-skilled specialists to run with the primary function of moving massive amounts of earth, in bulk.

As you can see in the graphic, the excavator approach to data integration (purple area) worked well, keeping pace with the distribution of data up until the late 90’s.  From there, the explosion of data from web apps commenced (red triangle) and has continued to accelerate year after year.   The data requirements of the business user changed, and as a result, IT has been overrun with rapidly expanding data demands as new data sources are discovered on what seems like a daily occurrence.  Using traditional, excavator-like machinery for these modern data integration projects has proven to be just too expensive and cumbersome.

In fairness to IT, the use of an excavator to address these dynamic integration projects was (and still is) an unfair disadvantage for IT.   Someone needed to invent the modern data integration equivalent of the BOBCAT, an agile, industrial strength, versatile, data integration platform that is inexpensive to operate and maintain; that can be quickly deployed for on-demand integration needs and used for the continuous progression of enterprise-extensible application lifecycles.

Today, Kapow introduces the Kapow Extraction Browser TM, the first and only web browser purpose built to extract, transform, integrate and migrate data from apps in the enterprise, on the web, and in the cloud.  Data is extracted from any layer in the application stack:  database, app logic and/or presentation layer.

Kapow KatalystTM – Browser-based data integration – a pragmatic new approach to data integration – No APIs required – No dependencies

The emergence of the browser as the defacto standard for data viewing has defined the linkage for business users to their data.  Brisk, mobile adoption and social media amplification are driving new data demands to the forefront of importance for the LsOB as the corporate enterprise ecosystem expands.   IT must begin to anticipate and adapt to these powerful business trends.

Business Users are fast-tracking the extensibility of the corporate enterprise by surfacing new applications for delivery on newer and less encumbered devices.  They will need assistance in dealing with these very real, dynamic and complex data movements. The opportunity for IT to step up and bridge this modern integration chasm for the LsOB has arrived.  Browser Integrated Data is the new road traveled – but get out of the way…

“…If you can’t lend your hand
For the times they are a-changin.”

By: John Yapaola

I recently attended a CIO event where the hot discussion topic was about how, when and why we should move our applications to the cloud and SaaS.

What surprised me was the main concern was less about moving applications and data in to the cloud but more about how do we get out of it?

And that is actually a really good question.

For obvious reasons SaaS vendors have little interest in customers ever moving away from their applications – they want to be as “sticky” as possible and grow their business.

For example, how many times do you hear about customers moving away from Salesforce? I’ve never heard of any, but I have heard many say that SaaS vendors turn up prices over time, so it’s not unlikely that someday soon you will start to hear more and more CIOs asking about how to “exit” the cloud, or at least how to move from one SaaS vendor to another.

The problem with migrating from a SaaS vendor is that you don’t own access to the database. The only way you can access and potentially move your data is using the API supplied by the SaaS vendor.

But because of the SaaS vendor’s desire to create “stickiness” you will likely not find all the features in these APIs for a pain-free migration to another SaaS vendor, or even back inside your firewall.

Fortunately there is a solution; it’s called “Browser-based Data Integration” by Kapow Software.

Browser-based Integration can access all the data and business logic in any SaaS application without any need for APIs which enables you to easily extract your data, transform it to your new desired format, and then load the data into a new SaaS application (also without an API), or load the data back into your database.

VOILA! Now you can very quickly move from one SaaS application to another, or even migrate back to an application on your private cloud. No APIs required.

Come learn more at www.KapowSoftware.com.

By: Stefan Andreasen

Today we announced a partnership with Composite Software to simplify and accelerate the integration of critical web data in large-scale data virtualization environments.

This is exiting!

In today’s world where data is rapidly spreading into data silos inside and outside company boundaries, the days of traditional ETL data consolidation to a central database or data warehouse are starting to fail with more frequency.

This is true for many reasons.  For example, data synchronization and data timeliness issues, and all the expensive database and data warehouse licenses and storage hardware needed for the consolidation.

Today it’s critical to be able to access any data on the fly as efficiently as if the data was consolidated, but access at its source where it lives best and is accurate and fresh.

This is exactly the problem that Composite Software solves. Their smart virtualization and data access algorithms mean that you can access data as fast (or sometimes even faster) than if you consolidate the data, and at the same time get fresh, real-time data directly from the source.

But without Kapow, Composite can only access the data if you have documented SQL or API access, and as I wrote in my previous blog Why Distribution of Data is exploding and what we can do about it, more and more relevant data does not have documented programmatic access.

Now, combining the power of Kapow Web Data Server with Composite Software, you get a complete one-stop solution to access your data at its source – a solution that covers 100% of all relevant data sources for your business applications or analytics/BI solutions.

All this without expensive recoding of existing applications, expensive data warehouse solutions, nor needing documented API or feed access.

This is a valuable new partnership for Kapow and Composite, which our customers, and the market at large, will greatly benefit from!

By:  Stefan Andreasen

We all know the amount as well as the distribution of data and applications is exploding around us. This explosion occurs not only within our company, but also globally on the internet, with government agencies and with business partners.

This causes huge problems for CIOs and IT departments all over the world. They are increasingly challenged to deliver projects that drive increased IT project backlog and budget overrun:

• Automated B2B integrations
• Relevant and timely data for analytics
• Innovative new business applications
• Mobilization of existing applications

What is causing the Distribution of Data to explode?

To understand this we need to look back to the late nineties where two extremely important and closely tied technology innovations occurred:

• The Web Browser: A new one-fits-all interface to access any application, independent of underlying architecture and platform.
• The Internet: An extremely easy way to reach data and applications anywhere at any time.

Today, employees in companies all over the world perform their daily work through a web browser. Jobs in Business Development, Marketing, Sales, Business Intelligence, Market analysis, Financial management, Procurement, Research, HR or just about any other job function can be performed accessing applications and tools through a web browser.

The ease of reach to applications creates an explosion in demand for new data and new ways to work with data and applications, which in turn creates a huge pain for CIO’s and IT departments all over the world.

How to address Distribution of Data complexities?

Looking at this figure we see the explosion in distribution of data (the red area).  Up until the late 1990s, application and data integration required standard interfaces (SQL, ODBC, JDBC, APIs, etc.).  But with the explosion of content and data on the internet and the emergence of SaaS and Cloud computing, IT is faced with an exponential growth of web applications they can’t control and that don’t have standard API interfaces.

The problem is that there is no easy access to documented APIs.

Companies are spending an increasing amount of money to make those applications compatible with the known data integration world by creating APIs or data feeds.

I suggest we look at this challenge from a new perspective. If we could programmatically access the data and applications the same way a web browser does, we’d have access to everything instantly.

Today more than 400 companies globally are doing exactly this using Web Data Services from Kapow Technologies.  It’s a one-stop solution to easily access data and applications with incredible success.

Not only can they now get to all the data their LOB is demanding, but also achieve huge cost savings around:

• Decreased spending on information providers – they simply grab the data they need themselves.
• Increased automation of manual processes such as data-entry and B2B interactions.
• Elimination of the timely and often impossible negotiation and implementation of documented B2B integration protocols with suppliers and retailers.
• Lowered expenses on consultants for legacy re-write projects – instead they simply leverage the existing web interface and meet the demand for new business and mobile apps.

Try it for yourselves, it really works.

By:  Stefan Andreasen

Tune in and join Stefan as a guest on DM Radio’s broadcast, “A Tactical Approach to Enterprise Data Integration“, this Thursday, January 28, at 3PM EST.  Learn more about how Web Data Services delivers timely, trusted, web data for enterprise data integration.

On the front lines of enterprise data integration, practitioners employ a whole host of tactics in order to deliver timely, trusted, valuable data. These days, a wide range of options are available for gathering, cleansing and delivering enterprise data. From Web scraping to federated data management, ETL to ELT, open-source to the traditional heavy lifting done by major vendors – the data integration practice is more mature than ever. Tune into this episode of DM Radio to learn how the modern enterprise can mix and match tactics to suit practically any information need. We’ll talk to industry analyst Philip Russom of TDWI, plus Ilan Sehayek of Jitterbit, Peter Tran of Composite Software, and Stefan Andreasen of Kapow Technologies.

Click over to DM Radio on the Information Management site for more details.

Many of you have asked for a deeper dive on the meaning of “Web Data Services”, so let me answer it here.

First, it’s important to understand the terms “Data Integration” and “Application Integration”.

Data Integration (DI) and (Enterprise) Application Integration (EAI) are not the same, though many vendors often confuse the two. Application Integration focuses on managing transactions or messages between applications while Data Integration focuses on managing the flow of data and providing standardized API’s to access the information. For more details, refer to Mark Madsen’s blog, Key Differences Between Data Integration and App Integration.

There are essentially three different types of Data Integration:

• Consolidation means moving all the data from the original data sources to a new repository, much like an ETL tool.

• Propagation means moving only the necessary data to a local storage for each application consuming the data.

• Federation means leaving the data at the original source and accessing it as needed in real-time.

Web Data Services is in reality all forms of Data Integration as well as Application Integration, with two distinct differences.  With Web Data Services:

• You primarily access data and business logic residing on the web (any application or data source you can access from a Web browser like Internet Explorer, Firefox or Safari). This includes applications inside your own organization and even at your business partners.

• You do not need to recode or have programmatic access to any of the data sources. As long as you have access from a Web Browser, you can access the data with no coding and be up and running in a matter of hours rather than weeks or months.

Web Data Services is the new highly productive way to access almost any of the data you need for Business Intelligence (BI), Data Validation and Acquisition, Enterprise Mashups, Partner Integration, or basically any solution that needs agile access to data or business logic. Web Data Services gives unheard of business agility and competitive advantage compared to traditional Data Integration or Enterprise Application Integration methods.

Try it free with our Kapow Web Data Server Trial Offer

Also check the Wikipedia entry on Web Data Services to read the definition from leading industry analysts.

By:  Stefan Andreasen