GigaOm guru Om Malik goes so far as to call data without proper context “dirt” in a recent blog post, and truly some of the large piles of data gathered by companies today are nothing more than a waste of storage space. It’s a tough job for a BI system – and the analyst working with it – to make sense of data that has lost its accompanying metadata. Especially with huge amounts of data, it is critical that it can be processed in an automated fashion without requiring human intervention or interpretation for every single datum.

There are many products emerging in the big data space where the focus is on driving huge volumes of data into your (local or cloud-based) storage systems. Much of this data is erroneous (due to “smart” algorithms that often turn out not to be less smart than advertised) and isn’t annotated with the relevant metadata that could make the difference between relevant information and garbage data.

Along the same lines, Paul Michaud of Nebility discusses the Garbage In = Garbage Out (GIGO) principle. Analysis becomes meaningless if it is done on top of inconsistent or incorrect data. Don’t get lost in the “big data” frenzy; data isn’t just data. It has to be meaningful, annotated, and consistent.

This requires a powerful ETL tool such as Kapow Katalyst that has rich transformation capabilities and can acquire the proper metadata as the data is extracted, or even add new relevant metadata based on complex business rules. And rather than just pulling random petabytes of data in, you should carefully consider what you wish to accomplish with the data and what requirements that poses on the quality of the data, the number of data sources that must be combined, and the current degree of consistency between these data sources.

It’s tempting to measure your big data strategy by the volume of data consumed, but that would be about as meaningful as judging gourmet restaurants by calories per serving. Focus on accessing, transforming, and delivering the highest quality of data for superior decision-making and competitive advantage.

By: Anne-Sofie Nielsen

Web data – external, unstructured, unwieldy, difficult to access and integrate, but oh-so-valuable – is a large part of what makes Big Data a goldmine. Today marks a major milestone for Kapow Software with the announcement of our new partnership with Informatica and the introduction of Informatica PowerExchange for Kapow Katalyst at InformaticaWorld, so that any data, anywhere on the web, is now available to Informatica customers to build their social media and Big Data strategies, as part of the Informatica 9.5 launch.

This further validates our unique competency to get critical and extremely strategic data from the web that no one else can – a capability that has been recognized and leveraged by over 500 customers worldwide – and now extends Informatica’s reach beyond the usual suspects; Facebook, LinkedIn, and Twitter.

When you think about it, wouldn’t it be great if you could access any data on any web app, even password protected apps, just as easily as accessing data in your own database? What if you could do this without writing a single line of SQL or taxing your already overburdened IT resources?

With Kapow Katalyst there is a rule of thumb: If you can see the data in your web browser then you are only a few clicks away from accessing and putting it to critical use in real-time. It’s that simple – and it’s scalable, secure and robust – used already by hundreds of companies around the world.

The internet, with 40-thousand new websites added every day, is the fastest growing source of information. It’s Big Data at it’s extreme and a discussion topic my colleagues and I have written about in the past.

Accessing the entire web as a database is very much about the 4th dimension of Big Data – the spread of data – and reflects one of the biggest challenges in Big Data – how to deal with accessing data from an explosion of different sources.

That problem is now solved.

With Informatica PowerExchange for Kapow Katalyst, Informatica customers can now access and extract all this data, noise-free and fully structured, to be used in a variety of Informatica products and Hadoop clusters.

I am very excited about this, and cannot wait to hear more about all the great projects Informatica customers will use this for – projects that I guarantee will drive significant value, driven by innovative ideas for new ways to delight customers, outwit the competition or simply operate more efficiently.

You would never gain this newfound value from not having direct access to the data, and certainly would not get it by buying the data from a data provider.

If you happen to be down in Vegas this week, come visit us at InformaticaWorld booth #S5 – we would love to show you how Kapow Katalyst can drive growth for your organization.

By: Stefan Andreasen

In a recent whitepaper for CIOs, Forrester analysts Ted Schadler and John C. McCarthy define mobile as the manifestation of a broader shift to new “systems of engagement”. These systems according to Forrester “help firms empower their customers, partners, and employees with context-aware apps and smart products to help them decide and act immediately in their moments of need”. This type of connectivity shifts more power to individuals, who can now serve themselves to check account status, make a purchase, view information or conduct a transaction – all via mobile devices.

There are multiple examples across many industries where mobility has taken the engagement to different levels, delivering business value and securing unique competitive advantages. Audi, who was recently honored the 2012 Connected Car of the Year awards, is one of these examples. The award spotlights vehicles with technology that strikes the right balance between safety, convenience, and infotainment. Audi A8 was selected for its unmatched Audi connect™ infotainment system. The leading automobile manufacturer wanted to provide real-time answers to location-based travel queries from its A8’s in-dash mobile navigation system. Audi recognized the power of engagement and leveraged mobility to innovate with a unique smart product that received recognition from industry experts and most importantly high-end customers. Audi delivers real-time data feeds from multiple sources about weather, restaurants, gas prices, traffic, construction and road updates directly to the Multi Media Interface in-car computer system. Audi is using mobility to redefine the driver’s everyday experience through access and connectivity to real-time information that is accurate, relevant and timely.

It is always a pleasure to help your customers introduce compelling new products, become leaders in their markets and receive industry-wide recognition for it. Kudos to Audi for their innovative business approach!

By: Hila Segal, Director, Product Marketing

Traditionally, when we talk about how data relates to applications, we think of data as the bottom of the three layers of the application stack:

But that distinction is evaporating fast, and soon we will end up with only one layer: the application itself.

Modern application frameworks also define the data model and tie it tightly with the application logic and the user interface. This means that even if you could access the data directly in the SQL database, the data won’t make much sense without the application logic that ties it together.

Similary application frameworks are also used to build the user interface, which means that application logic in modern applications is also woven into the the user interface; for example, into the JavaScript of a web application, leveraging one of many AJAJX frameworks to create a vivid user experience.

Now let’s look at BIG DATA applications. These don’t even use standard SQL databases anymore, but specialized NoSQL databases like HBase and Hive, build on the Hadoop processing framework. Now capabilities like search and analytics are tied together in one platform and the data can only be accessed through the BIG DATA application itself. If you need to fresh up on NoSQL technologies, check out this good (albeit a bit old) article, Decline of the Enterprise Data Warehouse by Road to Failure blogger Bradford.

The consequence of all this is that data and applications are becoming one and the same thing.

Similarly, data integration and application integration products are becoming one and the same thing, and anyone who needs to access data, will think of data in an application, not data standing alone.

To deal with this, we need to develop innovative new integration technologies that can access data through the application — a hybrid data integration and application integration platform, so to speak, but much more agile, since the pace of business and the impatience of business users today don’t permit waiting months or years to leverage that data.

The Kapow Innovation is the scalable Kapow Katalyst Enterprise Application Integration Platform that can live both on-premise or in the cloud; a technology that can access the data for the modern world, a world where there are no longer defined application layers.

Do you agree or disagree?

By: Stefan Andreasen