Informatica Roadshow Seattle, October

The theme of the roadshow is centered around solving the return on data equation which Informatica defines as the value of data divided by the cost of data:

Return on Data = Value on Data / Cost of Data

Specifically I am talking about how this applies to what is quickly becoming the most important and fastest growing data domain – “Big Interaction Data”. It is your customers’ blogs and tweets, your competitors pricing information and the data on your partners’ portals. It is unstructured, rapidly changing and provides unique insights that can turn your company into the market leader.

From being almost non-existing a few years back, Big Interaction Data is today growing explosively, with most of that happening on the internet and available as “Web Data”. My previous blog Dynamic pricing with automated competitive price monitoring provides a good example of how the use of Interaction Data can create huge differentiation, drive topline growth and turn companies into winners in their industry.

For many, understanding the huge importance of Big Interaction Data is just in its infancy, but I can assure you this is quickly changing. In my presentation, I give a few other examples of how companies leverage Interaction data. One example is about, Fiserv, a large financial services company who had to monitor point-in-time cash position across 300 partner banks for regulatory compliance. They had no other way of getting the data other than manually logging in to 300 portals and retrieving bank balances. Being able to get this type of data in an automated way was game changing for this organization. So as you can see Big Interaction Data is far more than just social media and provides significant value.

The other side of the equation is the “Cost of Data”. If the data is impossible to get, or very expensive to get, or cheap but out-of-date, then the return will be lower.

The most successful companies will be the ones that put data first, the most relevant data, and leverage technologies to get all that data at a low cost – in real-time.

Here is a snippet video of the presentation I gave in New York earlier this month:

On November 7th I will be presenting again in Atlanta, but you can also meet my colleagues at the San Francisco event on the same day. I will also be speaking in Dallas on November 13th if you happen to be in that area.

Come see me and let’s discuss the topic.

Stefan

Traditionally, when we talk about how data relates to applications, we think of data as the bottom of the three layers of the application stack:

But that distinction is evaporating fast, and soon we will end up with only one layer: the application itself.

Modern application frameworks also define the data model and tie it tightly with the application logic and the user interface. This means that even if you could access the data directly in the SQL database, the data won’t make much sense without the application logic that ties it together.

Similary application frameworks are also used to build the user interface, which means that application logic in modern applications is also woven into the the user interface; for example, into the JavaScript of a web application, leveraging one of many AJAJX frameworks to create a vivid user experience.

Now let’s look at BIG DATA applications. These don’t even use standard SQL databases anymore, but specialized NoSQL databases like HBase and Hive, build on the Hadoop processing framework. Now capabilities like search and analytics are tied together in one platform and the data can only be accessed through the BIG DATA application itself. If you need to fresh up on NoSQL technologies, check out this good (albeit a bit old) article, Decline of the Enterprise Data Warehouse by Road to Failure blogger Bradford.

The consequence of all this is that data and applications are becoming one and the same thing.

Similarly, data integration and application integration products are becoming one and the same thing, and anyone who needs to access data, will think of data in an application, not data standing alone.

To deal with this, we need to develop innovative new integration technologies that can access data through the application — a hybrid data integration and application integration platform, so to speak, but much more agile, since the pace of business and the impatience of business users today don’t permit waiting months or years to leverage that data.

The Kapow Innovation is the scalable Kapow Katalyst Enterprise Application Integration Platform that can live both on-premise or in the cloud; a technology that can access the data for the modern world, a world where there are no longer defined application layers.

Do you agree or disagree?

By: Stefan Andreasen

I’m flying back from New York where I presented “The Moneyball Approach to Big Data – Creating an Unfair Advantage” at the Wall Street Technology Association’s Hot Technologies Forum. Big Data is an area technologists are curious about, but I’m concerned there’s a “wait and see” approach. My job is to create value for my customers, and I’d hate for any of you to miss out on this opportunity.

Skepticism or “late adopter” mentality is understandable – if you want to forego a low-risk, high-reward opportunity and let your competition gain the advantage. Everyone is benefitting from Big Data in some form or another – most probably don’t even know it. But believe me, there are 100s of scenarios I could walk you through that could save your company millions of dollars, grow revenue double digit percentages, create more personalized products that delight your customers, automate real-time feedback on your brand, products, and competitor prices, create your own custom research that gives allows you to see trends before your competitors, and overall make you a much more agile business that scales with your new found vigor and growth.

What’s the secret to Big Data rewards? “Relevance”, “Access”, “Intelligence” and “Action”.

The most common definition I’ve seen for Big Data relates to the 3 Vs:

• Volume: it’s Big – terabytes and petabytes
• Variety: it comes in many forms – internal, external, structured and unstructured
• Velocity: it grows fast and changes quickly – making real-time capture and action hugely important

And this is always supported by numbers showing how gynormous Big Data is:

• The New York Stock Exchange creates 1 terabyte of data per day (InformationWeek)
• 10,000 payment card transactions are made every second around the world (American Banker)
• 30 billion pieces of content shared on FB every month (McKinsey)
• Twitter feeds generate 8 terabytes of data per day (InformationWeek)

Before you go out and buy more storage, consider what you want to do with it. If there are 200M tweets a day equaling 8 terabytes of data, but only 1000 of the tweets relate to your product or company, do you need to store and analyze all 8 terabytes every day? Although Big Data is big, don’t get caught up in all the massive numbers. Stick with what’s relevant to your business.

Forrester Analyst Brian Hopkins made a great point in his blog “Big Data will help shape your markets next big winner”, stating that Forrester estimates enterprises use only 5% of their available data. So the playing field is wide open for anyone to quickly take advantage of the 95% they’re currently ignoring.

But slow down there pardner. Sybase published Big Data, Big Opportunity that stated, “for the median fortune 1000 company… a 10% increase in usability of data translates to an increase of $2.01B in total revenue per year, [and that] a 10% increase in accessibility to data translates to an additional $65.67M in net income per year.” So don’t think you have to go from 5% to 100%. You really only need to go from 5% to 5.5%.

The internet plays a huge role in the rapid growth of Big Data, giving individuals the ability to post and upload immense amounts of pictures, text, video, and mobile data, and businesses the channel to offer access to customers and partners through web-based applications (think Oracle, salesforce.com, social media, procurement, logistics, publishers, and so on).

In reviewing other articles about Big Data, despite all the discussion around the massiveness of Big Data, I didn’t find a single article mentioning the difficulty of accessing the data spread throughout all these applications. This is a HUGE POINT to understand because you are SOL if you can’t access the data you need. If I told you I could guarantee any app or data you can see in your web browser (customer data, bank transactions, twitter, blogs, supply chain vendors, government data, competitor prices, etc.) could be automatically accessed and loaded into the app, database, or spreadsheet of your choice, how many game-changing Big Data projects could you think of? Point-in-time cash position understanding of billions of dollars across 300 banks? No problem. Monitor competitor pricing on 50,000 SKUs every day? Simple. Automate a twenty-three step manual invoicing process to get paid millions of dollars 2 days faster? Done. Real-time, automated access to the data you need is the key to success with Big Data. Lest you think this all fantasy, learn how Kapow Katalyst Application Integration Platform provides real-time access to Big Data.

There’s huge difference between “I have terabytes of data – videos, satellite pictures, social media conversations, and research reports” and “I know where Osama Bin Laden is”. It’s Data vs. Intelligence. Data is useless if you can’t extract meaningful intelligence from it. And the quality of your intelligence is most likely much less dependent on the volume than the relevance and ability to access it.

And the whole point of having relevant, accessible, intelligent Big Data is that it is actionable. Otherwise it’s just a recommendation or a strategy without execution What’s incredibly cool about Big Data and the web-based nature of so much of it is that just as easily as you can access anything you can see, you can just as easily transform the data, perform an operation on it, and automate a resulting action for you. Huh? Here’s an example. You know consumers and even your B2B purchasers research prices online and that loyalty to any one vendor has deteriorated as buyers have more pricing knowledge a search and mouse-click away. But you are smarter than your competitors because you’re already doing the extra 10%. So you set up automated monitoring of your competitor’s pricing, and when their price drops below yours your Big Data Integration Platform calculates the difference plus 10%, logs into your ecommerce site and adjusts your prices automatically, all within a few ticks of the clock.

And the beauty is that this can all be set up in hours, if not a few days, and you don’t have to bring in an army of developers or consultants to create custom code to do any of this.

So let the Big Data party begin. Kapow Software is here to help. To learn more about Big Data Solutions or to set up a Big Data Sales Consultation, click either link, because you’ve read this far and deserve it!

By: Rick Kawamura

In 2007, James Governor penned an article, Why Applications Are Like Fish and Data Is Like Wine, depicting how data gets better with age, while apps, like fish, begin to smell over time.

Earlier this year, Chuck Hollis from EMC offered his own wrinkle on the topic, also making a case for keeping the data (wine) and dumping those ‘lumbering apps’ (stale fish).

Both offer interesting reads, but too quickly dismiss the value of apps. Fish and wine go well together and depending on how you pair them, can make or break the experience or value of the meal. The same is true in the modern enterprise – the fish (application) is equally, if not more important than the wine (data), especially when the apps are kept “fresh and simple” with “many varieties to choose from…”

Consider if all the data trapped within applications were easily accessed and made even easier to interact with other applications.

There should be a straightforward way to interact and automate as many ‘fresh’ application sources as possible, in the shortest amount of time. IT organizations can no longer remain ‘comfortably numb’ in their avoidance of these agile Line Of Business (LOB) activities.

So, does the LOB really want a customary IT application integration fix or are they asking for something different? When the LOB says, “I really need access to this data now!” what they are really asking for is to interact with applications (or websites) in an impromptu manner in order to keep pace with emerging business dynamics and acquire key data advantages for the business.

The blistering pace and expansion of social media access, in all its forms (brand management, blog monitoring, anti-piracy, competitive intelligence, analyst research, prospect & customer mapping, consumer behavior analysis, marketing research, partner & reseller communication, on-line videos, risk compliance, legal monitoring, corporate reputation, background checks, R&D innovation research, consumer & customer research, sentiment analysis, predictive analytics, social CRM) makes the LOB integration demand on IT seem like an insurmountable task. The intensification of interactions from sources without API coverage (or even marginal API compliance) can no longer shackle the LOB data integration needs.

The traditional corporate IT blueprint has been to deploy long established application integration methodologies. This old design is already showing the strain of abandonment. I would propose a modern application integration approach consisting of the following components:

APPLICATION INTEGRATION PILLARS:

All four pillars are key to providing real-time integration, but they must also move toward a new form of LOB automation – a self-service component for the LOB. Ubiquitous access with the ability to quickly prototype and test application interactions are the most critical components to this offering. Applications need to be integrated in order for the LOB to interact and drive time-to-value (TTV) in the enterprise.

Adding to the turmoil for IT organizations are the hundreds (often thousands) of in-house applications constructed over the years. The vast majority of these have not been SOA enabled. Those ‘lumbering apps’ are also inclusive of the LOB need. There’s valuable data trapped in those in-house apps that must be freed.

Integrating to applications will require more frequent and varied connections with real-time and on-demand communications. These integrations may be more permanent links to applications or have ‘throw-away’ integration conditions.

The LOB is forcing the application integration challenge to the forefront of the IT stack. Applications are the real-time component of the raging BIG DATA frenzy. The day is coming, when the LOB will self-serve most of their application and website interactions.

The data wine cellars of the enterprise will be cared for by the IT stewards, but the Line Of Business wants FISH.

By: John Yapaola

Do you dare to step out of your comfort zone? The world is changing at a faster and faster pace, especially in the world of IT.

Driven by the explosion of internet connectivity, distribution of data sources, and the number of legacy and networked applications, impatient business demands for IT projects are growing both in numbers and in complexity.

Line of Business needs projects delivered yesterday and with data as fresh as in the original data source. “Rotten” data is no longer acceptable.

In a situation like this, the traditional methods of working often fall short, something that can be very difficult to realize when changes happen gradually. Things don’t get out of control from one day to the next, but over months and years, until it’s too late.

I tried to illustrate the situation in the 3D graph above. The area of control (blue) is gradually turning out of control (red).

• Data demand has moved from batch to real-time
• Complexity has increased with the distribution of applications and data source
• Business urgency requires unrealistic time-to-market goals
• Number of IT projects is growing, but the budget is not

In a changing environment like this, we constantly need to challenge our processes and tools.

So my question to you is: “Are you ready to change the way you do things? Do you dare to step out of your comfort zone?”

If not, you’ve got a problem. You might not see this problem on a daily basis; it’s one of those problems that slowly sneaks up on you until one day it’s too late to fix.

It’s happened many times before, just look at DEC (Digital Equipment Corporation), the hugely successful mid-size computer company back in the 80s which basically died overnight; they did not step outside their comfort zone and subsequently disappeared.

Kapow Software’s revolutionary innovation, the Kapow Extraction Browser (which by the way is the only purpose-build integration browser on the market), has proven to help hundreds of IT projects overcome these challenges and make an easy jump from the blue to the red area.

By leveraging the fact that all applications today either have an API or GUI, where data and transactions can be accessed and controlled, means you no longer are dependent on the data and application owners. That’s a big deal. No more rewriting of legacy apps and no more begging the application owner to provide APIs. Just go do it.

That is a new way of thinking and new paradigm. Are you ready for it?

Not only does it work, it’s proven by more than 500 customers worldwide. It’s also delivering incredible business value and business agility among your competitors.

Step out of your box and try Kapow Software yourself!

By: Stefan Andreasen

Cloud Integration is more than just pre-existing connectors and SalesForce integration

As companies move their IT infrastructure and business applications to SaaS and the cloud it creates increasing need “cloud integration”, the ability to integrate data between applications in the hybrid world of internal apps, cloud apps and business partner apps.

Many Cloud Integration companies claim to offer a complete solution for integrating cloud and SaaS apps, but I claim they are all incomplete. Why?

If you check their demos and use-cases you quickly realize they provide a solution that only works if you have access to existing (and documented) web service APIs. And consequently, most of their examples are centered around salesforce.com integration and salesforce.com APIs.

This approach does not apply to the real cloud world – a world that is far from homogenous, but rather a hybrid, fragmented, and distributed world.

A cloud integration solution is only complete if it can integrate all applications in the cloud, whether they have Web Service APIs or not.

Today there are more than 200 million websites/applications on the internet, and many have complex features and data structure behind them. In this more holistic, big picture, only a fraction of these millions of websites are covered by documented APIs, making most traditional cloud integration solutions all but incomplete. The likelihood that your next cloud integration project will not be covered by “standard” connectors is very high, and therefore you need a cloud integration platform that can integrate to any layer in the application stack: database (SQL), web service (SOAP, REST), or through the presentation layer (HTML, AJAX).

Based on the patended Kapow Extraction Browser that leverages any HTML/AJAX application interface as an API if no web services API is present, Kapow Katalyst is the only Cloud Integration platform that has a complete data extraction, data integration, data transformation and data migration solution for your cloud integration challenges.

Proven by more than 500 customers world-wide, Kapow Katalyst is the only cloud integration platform that provides total connectivity and total data delivery in the cloud, in the enterprise and with your business partners.

Are you ready for your next cloud integration project?

By: Stefan Andreasen

“Your old road is rapidly agin’
Please get out of the new one
If you can’t lend your hand
For the times they are a-changin.”
-  Bob Dylan  1963

Fast forward almost 50 years to 2010 and this Dylan verse offers a modern-day perspective on recent developments in the corporate enterprise, as business users set down new demands (expectations) for the access and use of data.

Economic downturns can be painful, but often have a positive effect on companies by forcing clarity of business execution.  Hardened product/service deliverables and timelines suddenly become indisputable for managers and employees.  Expensive IT projects and initiatives previously thought to be valuable to the Company are ‘rudely’ terminated in favor of genuine efficiency demands, with the goal of lowering transactional costs of the Company.  The Lines of Business (LsOB) and IT organizations are given very specific marching orders: ‘Do more with less and do it faster and better than before!’

Are you “Waiting on IT” in an impatient business climate?

‘Clouds’ have gathered in the wake of the 2008 financial crisis and the perfect storm for change is imminent.  An Economic Downturn, Cloud Computing, and Social/Mobile media are driving rapid and transformational change in the way the LsOB will consume and distribute information in the modern enterprise to their partners and customers.

The business users’ insatiable need for access to data in all of its forms and from as many sources has created enormous friction for IT.    As the traditional gate-keeper of corporate data, IT’s role is to provide the means to make data-delivery easier for the business with integrated data being an expectation of the business user.

So why has data integration in the enterprise fallen so far behind in recent years?  In Clay Christiansen’s book “The Innovator’s Dilemma” there is a passage on new product development in the construction industry and the use of older machines prior to a disruptive innovation.  The excavator specifically triggered an analogy for me, as it relates to the current state of data integration in the enterprise.

The excavator is a massive piece of machinery to acquire, operate and maintain.  It requires highly-skilled specialists to run with the primary function of moving massive amounts of earth, in bulk.

As you can see in the graphic, the excavator approach to data integration (purple area) worked well, keeping pace with the distribution of data up until the late 90’s.  From there, the explosion of data from web apps commenced (red triangle) and has continued to accelerate year after year.   The data requirements of the business user changed, and as a result, IT has been overrun with rapidly expanding data demands as new data sources are discovered on what seems like a daily occurrence.  Using traditional, excavator-like machinery for these modern data integration projects has proven to be just too expensive and cumbersome.

In fairness to IT, the use of an excavator to address these dynamic integration projects was (and still is) an unfair disadvantage for IT.   Someone needed to invent the modern data integration equivalent of the BOBCAT, an agile, industrial strength, versatile, data integration platform that is inexpensive to operate and maintain; that can be quickly deployed for on-demand integration needs and used for the continuous progression of enterprise-extensible application lifecycles.

Today, Kapow introduces the Kapow Extraction Browser TM, the first and only web browser purpose built to extract, transform, integrate and migrate data from apps in the enterprise, on the web, and in the cloud.  Data is extracted from any layer in the application stack:  database, app logic and/or presentation layer.

Kapow KatalystTM – Browser-based data integration – a pragmatic new approach to data integration – No APIs required – No dependencies

The emergence of the browser as the defacto standard for data viewing has defined the linkage for business users to their data.  Brisk, mobile adoption and social media amplification are driving new data demands to the forefront of importance for the LsOB as the corporate enterprise ecosystem expands.   IT must begin to anticipate and adapt to these powerful business trends.

Business Users are fast-tracking the extensibility of the corporate enterprise by surfacing new applications for delivery on newer and less encumbered devices.  They will need assistance in dealing with these very real, dynamic and complex data movements. The opportunity for IT to step up and bridge this modern integration chasm for the LsOB has arrived.  Browser Integrated Data is the new road traveled – but get out of the way…

“…If you can’t lend your hand
For the times they are a-changin.”

By: John Yapaola

I recently attended a CIO event where the hot discussion topic was about how, when and why we should move our applications to the cloud and SaaS.

What surprised me was the main concern was less about moving applications and data in to the cloud but more about how do we get out of it?

And that is actually a really good question.

For obvious reasons SaaS vendors have little interest in customers ever moving away from their applications – they want to be as “sticky” as possible and grow their business.

For example, how many times do you hear about customers moving away from Salesforce? I’ve never heard of any, but I have heard many say that SaaS vendors turn up prices over time, so it’s not unlikely that someday soon you will start to hear more and more CIOs asking about how to “exit” the cloud, or at least how to move from one SaaS vendor to another.

The problem with migrating from a SaaS vendor is that you don’t own access to the database. The only way you can access and potentially move your data is using the API supplied by the SaaS vendor.

But because of the SaaS vendor’s desire to create “stickiness” you will likely not find all the features in these APIs for a pain-free migration to another SaaS vendor, or even back inside your firewall.

Fortunately there is a solution; it’s called “Browser-based Data Integration” by Kapow Software.

Browser-based Integration can access all the data and business logic in any SaaS application without any need for APIs which enables you to easily extract your data, transform it to your new desired format, and then load the data into a new SaaS application (also without an API), or load the data back into your database.

VOILA! Now you can very quickly move from one SaaS application to another, or even migrate back to an application on your private cloud. No APIs required.

Come learn more at www.KapowSoftware.com.

By: Stefan Andreasen

Today we announced a partnership with Composite Software to simplify and accelerate the integration of critical web data in large-scale data virtualization environments.

This is exiting!

In today’s world where data is rapidly spreading into data silos inside and outside company boundaries, the days of traditional ETL data consolidation to a central database or data warehouse are starting to fail with more frequency.

This is true for many reasons.  For example, data synchronization and data timeliness issues, and all the expensive database and data warehouse licenses and storage hardware needed for the consolidation.

Today it’s critical to be able to access any data on the fly as efficiently as if the data was consolidated, but access at its source where it lives best and is accurate and fresh.

This is exactly the problem that Composite Software solves. Their smart virtualization and data access algorithms mean that you can access data as fast (or sometimes even faster) than if you consolidate the data, and at the same time get fresh, real-time data directly from the source.

But without Kapow, Composite can only access the data if you have documented SQL or API access, and as I wrote in my previous blog Why Distribution of Data is exploding and what we can do about it, more and more relevant data does not have documented programmatic access.

Now, combining the power of Kapow Web Data Server with Composite Software, you get a complete one-stop solution to access your data at its source – a solution that covers 100% of all relevant data sources for your business applications or analytics/BI solutions.

All this without expensive recoding of existing applications, expensive data warehouse solutions, nor needing documented API or feed access.

This is a valuable new partnership for Kapow and Composite, which our customers, and the market at large, will greatly benefit from!

By:  Stefan Andreasen

We all know the amount as well as the distribution of data and applications is exploding around us. This explosion occurs not only within our company, but also globally on the internet, with government agencies and with business partners.

This causes huge problems for CIOs and IT departments all over the world. They are increasingly challenged to deliver projects that drive increased IT project backlog and budget overrun:

• Automated B2B integrations
• Relevant and timely data for analytics
• Innovative new business applications
• Mobilization of existing applications

What is causing the Distribution of Data to explode?

To understand this we need to look back to the late nineties where two extremely important and closely tied technology innovations occurred:

• The Web Browser: A new one-fits-all interface to access any application, independent of underlying architecture and platform.
• The Internet: An extremely easy way to reach data and applications anywhere at any time.

Today, employees in companies all over the world perform their daily work through a web browser. Jobs in Business Development, Marketing, Sales, Business Intelligence, Market analysis, Financial management, Procurement, Research, HR or just about any other job function can be performed accessing applications and tools through a web browser.

The ease of reach to applications creates an explosion in demand for new data and new ways to work with data and applications, which in turn creates a huge pain for CIO’s and IT departments all over the world.

How to address Distribution of Data complexities?

Looking at this figure we see the explosion in distribution of data (the red area).  Up until the late 1990s, application and data integration required standard interfaces (SQL, ODBC, JDBC, APIs, etc.).  But with the explosion of content and data on the internet and the emergence of SaaS and Cloud computing, IT is faced with an exponential growth of web applications they can’t control and that don’t have standard API interfaces.

The problem is that there is no easy access to documented APIs.

Companies are spending an increasing amount of money to make those applications compatible with the known data integration world by creating APIs or data feeds.

I suggest we look at this challenge from a new perspective. If we could programmatically access the data and applications the same way a web browser does, we’d have access to everything instantly.

Today more than 400 companies globally are doing exactly this using Web Data Services from Kapow Technologies.  It’s a one-stop solution to easily access data and applications with incredible success.

Not only can they now get to all the data their LOB is demanding, but also achieve huge cost savings around:

• Decreased spending on information providers – they simply grab the data they need themselves.
• Increased automation of manual processes such as data-entry and B2B interactions.
• Elimination of the timely and often impossible negotiation and implementation of documented B2B integration protocols with suppliers and retailers.
• Lowered expenses on consultants for legacy re-write projects – instead they simply leverage the existing web interface and meet the demand for new business and mobile apps.

Try it for yourselves, it really works.

By:  Stefan Andreasen