Jump to content

Nov 18

I’m flying back from New York where I presented “The Moneyball Approach to Big Data – Creating an Unfair Advantage” at the Wall Street Technology Association’s Hot Technologies Forum. Big Data is an area technologists are curious about, but I’m concerned there’s a “wait and see” approach. My job is to create value for my customers, and I’d hate for any of you to miss out on this opportunity.

Skepticism or “late adopter” mentality is understandable – if you want to forego a low-risk, high-reward opportunity and let your competition gain the advantage. Everyone is benefitting from Big Data in some form or another – most probably don’t even know it. But believe me, there are 100s of scenarios I could walk you through that could save your company millions of dollars, grow revenue double digit percentages, create more personalized products that delight your customers, automate real-time feedback on your brand, products, and competitor prices, create your own custom research that gives allows you to see trends before your competitors, and overall make you a much more agile business that scales with your new found vigor and growth.

What’s the secret to Big Data rewards? “Relevance”, “Access”, “Intelligence” and “Action”.

The most common definition I’ve seen for Big Data relates to the 3 Vs:

  • Volume: it’s Big – terabytes and petabytes
  • Variety: it comes in many forms – internal, external, structured and unstructured
  • Velocity: it grows fast and changes quickly – making real-time capture and action hugely important

And this is always supported by numbers showing how gynormous Big Data is:

  • The New York Stock Exchange creates 1 terabyte of data per day (InformationWeek)
  • 10,000 payment card transactions are made every second around the world (American Banker)
  • 30 billion pieces of content shared on FB every month (McKinsey)
  • Twitter feeds generate 8 terabytes of data per day (InformationWeek)

Before you go out and buy more storage, consider what you want to do with it. If there are 200M tweets a day equaling 8 terabytes of data, but only 1000 of the tweets relate to your product or company, do you need to store and analyze all 8 terabytes every day? Although Big Data is big, don’t get caught up in all the massive numbers. Stick with what’s relevant to your business.

Forrester Analyst Brian Hopkins made a great point in his blog “Big Data will help shape your markets next big winner”, stating that Forrester estimates enterprises use only 5% of their available data. So the playing field is wide open for anyone to quickly take advantage of the 95% they’re currently ignoring.

But slow down there pardner. Sybase published Big Data, Big Opportunity that stated, “for the median fortune 1000 company… a 10% increase in usability of data translates to an increase of $2.01B in total revenue per year, [and that] a 10% increase in accessibility to data translates to an additional $65.67M in net income per year.” So don’t think you have to go from 5% to 100%. You really only need to go from 5% to 5.5%.

The internet plays a huge role in the rapid growth of Big Data, giving individuals the ability to post and upload immense amounts of pictures, text, video, and mobile data, and businesses the channel to offer access to customers and partners through web-based applications (think Oracle, salesforce.com, social media, procurement, logistics, publishers, and so on).

In reviewing other articles about Big Data, despite all the discussion around the massiveness of Big Data, I didn’t find a single article mentioning the difficulty of accessing the data spread throughout all these applications. This is a HUGE POINT to understand because you are SOL if you can’t access the data you need. If I told you I could guarantee any app or data you can see in your web browser (customer data, bank transactions, twitter, blogs, supply chain vendors, government data, competitor prices, etc.) could be automatically accessed and loaded into the app, database, or spreadsheet of your choice, how many game-changing Big Data projects could you think of? Point-in-time cash position understanding of billions of dollars across 300 banks? No problem. Monitor competitor pricing on 50,000 SKUs every day? Simple. Automate a twenty-three step manual invoicing process to get paid millions of dollars 2 days faster? Done. Real-time, automated access to the data you need is the key to success with Big Data. Lest you think this all fantasy, learn how Kapow Katalyst Application Integration Platform provides real-time access to Big Data.

There’s huge difference between “I have terabytes of data – videos, satellite pictures, social media conversations, and research reports” and “I know where Osama Bin Laden is”. It’s Data vs. Intelligence. Data is useless if you can’t extract meaningful intelligence from it. And the quality of your intelligence is most likely much less dependent on the volume than the relevance and ability to access it.

And the whole point of having relevant, accessible, intelligent Big Data is that it is actionable. Otherwise it’s just a recommendation or a strategy without execution What’s incredibly cool about Big Data and the web-based nature of so much of it is that just as easily as you can access anything you can see, you can just as easily transform the data, perform an operation on it, and automate a resulting action for you. Huh? Here’s an example. You know consumers and even your B2B purchasers research prices online and that loyalty to any one vendor has deteriorated as buyers have more pricing knowledge a search and mouse-click away. But you are smarter than your competitors because you’re already doing the extra 10%. So you set up automated monitoring of your competitor’s pricing, and when their price drops below yours your Big Data Integration Platform calculates the difference plus 10%, logs into your ecommerce site and adjusts your prices automatically, all within a few ticks of the clock.

And the beauty is that this can all be set up in hours, if not a few days, and you don’t have to bring in an army of developers or consultants to create custom code to do any of this.

So let the Big Data party begin. Kapow Software is here to help. To learn more about Big Data Solutions or to set up a Big Data Sales Consultation, click either link, because you’ve read this far and deserve it!

By: Rick Kawamura Rick Kawamura, Director of Marketing

Tagged with:          
Nov 17

I was just rereading the article, “Choose your cloud with integration in mind” by David Linthicum for Infoworld’s Cloud Computing blog.

David knows this space very well, and in this article he discusses the challenge of integrating cloud-based and on-premise applications with the limits on integration interface instances imposed, understandably, by some cloud service providers.

I agree with David that it’s common sense to integrate your on-premise business applications with your SaaS-based business applications. Any company that has business applications inside the firewall as well as outside must integrate them to realize their full value.

I also agree that if you select a cloud-based integration solution, you need to be sure you have enough interfaces in the cloud to support the number and volume of connections from the cloud to your on-premise apps.

But is this the biggest challenge when doing on-premise to cloud integration?

No, or at least it’s not what I hear most often when discussing integration with industry people. They tell me that IT’s biggest concern is security. They ask, “How do you get your IT department to open up the firewall for integrating cloud apps to on-premise apps? Because I get a blank stare when I ask our IT to do it.”

So how do you deal with this?

You choose an integration platform that can be installed both on-premise and in the cloud. This is a must-have requirement.

With your integration solution installed on-premise, you can integrate applications inside-out rather that outside-in, which will be a lot easier for your IT department to accommodate because there’s less risk.

At the same time, you can have the flexibility and easy maintenance of a cloud integration solution for all those applications where firewall issues are not showstoppers.

To avoid any unpleasant surprises, I recommend looking for these features and benefits in addition to the security of a hybrid solution when choosing an application integration platform:

Does the integration platform you’re considering or using provide these features and benefits? How do you deal with IT’s concerns about security and hybrid integration? I’m interested in your experience and perspective.

By: Stefan Andreasen Stefan Andreasen

Tagged with:       
Nov 08

In 2007, James Governor penned an article, Why Applications Are Like Fish and Data Is Like Wine, depicting how data gets better with age, while apps, like fish, begin to smell over time.

Earlier this year, Chuck Hollis from EMC offered his own wrinkle on the topic, also making a case for keeping the data (wine) and dumping those ‘lumbering apps’ (stale fish).

Both offer interesting reads, but too quickly dismiss the value of apps. Fish and wine go well together and depending on how you pair them, can make or break the experience or value of the meal. The same is true in the modern enterprise – the fish (application) is equally, if not more important than the wine (data), especially when the apps are kept “fresh and simple” with “many varieties to choose from…”

Consider if all the data trapped within applications were easily accessed and made even easier to interact with other applications.

There should be a straightforward way to interact and automate as many ‘fresh’ application sources as possible, in the shortest amount of time. IT organizations can no longer remain ‘comfortably numb’ in their avoidance of these agile Line Of Business (LOB) activities.

So, does the LOB really want a customary IT application integration fix or are they asking for something different? When the LOB says, “I really need access to this data now!” what they are really asking for is to interact with applications (or websites) in an impromptu manner in order to keep pace with emerging business dynamics and acquire key data advantages for the business.

The blistering pace and expansion of social media access, in all its forms (brand management, blog monitoring, anti-piracy, competitive intelligence, analyst research, prospect & customer mapping, consumer behavior analysis, marketing research, partner & reseller communication, on-line videos, risk compliance, legal monitoring, corporate reputation, background checks, R&D innovation research, consumer & customer research, sentiment analysis, predictive analytics, social CRM) makes the LOB integration demand on IT seem like an insurmountable task. The intensification of interactions from sources without API coverage (or even marginal API compliance) can no longer shackle the LOB data integration needs.

The traditional corporate IT blueprint has been to deploy long established application integration methodologies. This old design is already showing the strain of abandonment. I would propose a modern application integration approach consisting of the following components:


All four pillars are key to providing real-time integration, but they must also move toward a new form of LOB automation – a self-service component for the LOB. Ubiquitous access with the ability to quickly prototype and test application interactions are the most critical components to this offering. Applications need to be integrated in order for the LOB to interact and drive time-to-value (TTV) in the enterprise.

Adding to the turmoil for IT organizations are the hundreds (often thousands) of in-house applications constructed over the years. The vast majority of these have not been SOA enabled. Those ‘lumbering apps’ are also inclusive of the LOB need. There’s valuable data trapped in those in-house apps that must be freed.

Integrating to applications will require more frequent and varied connections with real-time and on-demand communications. These integrations may be more permanent links to applications or have ‘throw-away’ integration conditions.

The LOB is forcing the application integration challenge to the forefront of the IT stack. Applications are the real-time component of the raging BIG DATA frenzy. The day is coming, when the LOB will self-serve most of their application and website interactions.

The data wine cellars of the enterprise will be cared for by the IT stewards, but the Line Of Business wants FISH.

By: John Yapaola John Yapaola

Tagged with:          
Nov 07

Do you remember when JavaScript was merely used to add some snappy effects to plain HTML websites; perhaps a drop-down menu that rolled out when you moved your mouse over an object or an image that changed every 5 seconds?

We were a bunch of happy-go-lucky web programmers — script kiddies — back in the mid-90s who could not imagine how much business logic, data transformation, and interactive presentation would be taking place in the user’s browser 15 years later. Sure, data was sometimes dynamically generated, but all that magic took place on our server-side scripts.

When Kapow Software was simply extracting data from the web in the late 90s, we could safely ignore the sprinkles of JavaScript on a website; we could simply go straight to the HTML to get the data. Unfortunately, I still see people trying this approach.

Today even the best script kiddie is going to have an extremely hard time crafting a Perl script that can grab data from a modern website or web app where the information displayed in the user’s browser is the result of executing thousands of lines of JavaScript, pulling data from web services, and transforming that data in sophisticated ways. Even worse: Imagine doing web application integration or business process automation just scripting!

Here at Kapow Software, we quickly realized that disregarding JavaScript when trying to integrate web applications was insufficient. And we made the radical decision to base Katalyst Katalyst on our Integration Engine with Web App Interface that allows you to integrate any modern website or application, including all of the dynamic content generated by JavaScript.

That means that as a Kapow Katalyst user, you can focus on the data or apps you want, rather than having to concern yourself with how the website or web application was originally built.

While it was fun being a script kiddie in the nineties, with today’s dynamic web and the exponential growth in content and sources, scripting has proven extremely unproductive, if not impossible. It’s time to shift the focus away from technical issues and move toward solutions that help you solve your business problems.

By: Anne-Sofie Nielsen Fie

Director of R&D at Kapow Software

Tagged with:    

The Kapow Katalyst Blog is…

... a collection of insights, perspectives, and thought leadership around Application Integration.

Comments, Feedback, Contact Us:

blog at kapowsoftware.com

Get Our RSS Feed