AstraZeneca, one of the world's leading pharmaceutical companies, uses Kapow to automate web intelligence for executive reports and dashboards.
Web Intelligence Solution:
Web Data Extraction
- Home
- Solutions
- Web Intelligence
- Web Data Extraction
Automated real-time data extraction provides higher quality, comprehensive data
Any data you see in your web browser, from government and e-commerce to partner sites, social media and tons of other public websites, they all contain data that can add tremendous value to your analytics, intelligence, and information services. This includes local, state and federal statistics and compliance information, job listings, product catalogues, customer sentiment, competitive pricing, etc. The variety and volume of information is overwhelming and growing every day. Kapow automates Web Data Extraction with powerful transformation capabilities to ensure you get the right data at the right time and in the right format.
Featured Customer Success Stories for Web Data Extraction:
Challenge
- To meet U.S. compliance requirements, pharmaceutical companies are required to maintain a Health Care Professionals (HCP) Exclusion List, showing HCPs that should be excluded from existing or new contracts.
- AstraZeneca wanted to automate existing manual processes for developing and maintaining the HCP Exclusion List.
Solution
- They used Kapow to create a master listing from information on external websites.
- The listing is automatically updated with specific regulated information.
- The process is integrated from delivery to a spreadsheet to a downstream compliance system.
- Data is integrated into an internal data mart built on Business Objects to create executive reports and dashboards.
- This is a major reporting system to the FDA, including all money spent for contracting outside HCPs and research organizations.
Results
- After implementing the Kapow automated master web intelligence and matching process, the time to develop a list of possible suspects for non-compliant HCPs was reduced from 6 months to 2 weeks.
- AstraZeneca estimates that it could save millions of dollars in FDA fine avoidance with the improved compliance monitoring.
AGCO, the largest pure play, full-line agricultural equipment manufacturer, uses Kapow to automate web data extraction on hundreds of thousands of parts.
Challenge
- AGCO's parts division needed to pull data from 10 websites, which sell AGCO parts as well as competitors' products.
- Their existing data feed was inflexible. They got infrequent data dumps, and AGCO had to do a lot of manual work on the product description lists.
- Overall not a very good or agile solution, given that they were extracting hundreds of thousands of parts in a very competitive industry.
Solution
- Kapow provides a fully automated data feed solution, including robots to pull from the 10 key websites and a hosted installation of Kapow running on Amazon EC2.
- Kapow also provides ongoing robot maintenance and support.
- An in-house deployment is planned for the next phase.
Results
- Kapow eliminated manual work associated with web data acquisition.
- Competitive information is up-to-date.
This innovative travel service company uses Kapow to comb 500 sites for the best deals.
Challenge
- Aggregate complete and unfiltered travel information from hundreds of travel sites in real time.
- Provide service at no cost to users; rely on sponsored links and ads for revenue.
Solution
- Kapow automates web intelligence with 600+ robots.
- Batch collection runs every few hours. Robot runs are based on frequency of requested routes.
- Provides real-time updates on-demand to confirm current prices.
- User requests are routed to airline sites to book travel.
Results
- Allows users to search almost 500 travel sites and compare results in one legible display. Unlike many aggregator sites, they include major booking engines, national carriers and most low-cost airlines.
- Separates price search from booking process, enabling much more rapid search results.
- Automated web harvesting makes the company's innovative business model economically viable.
Automating web data extraction was key to expanding business for PPG Industries, the world's leading provider of coatings and specialty products and services.
Challenge
- An insurance company has outsourced its claims department to PPG. PPG handles claims requests from car owners through a call center application.
- PPG needs access to each insurance policy data, which typically resides on a mainframe.
- They are now encountering clients whose data resides in web applications. They need to automate web data extraction and integrate the data with internal systems to replace time consuming manual data processing by call center staff.
Solution
- Kapow robots perform web data extraction automatically.
- Robots are wrapped in web services, which make them easy to integrate into the existing solution.
Results
- Automating web data extraction and integration will result in an estimated 50,000 additional cases from the insurance company.
Attivio, a leader in unified information access, uses Kapow to automate web intelligence and integration with regulatory management systems.
Challenge
- Attivio needed to capture regulatory information from public websites, including content in JavaScript and PDFs, for their end-customer, a major financial institution.
Solution
- With Kapow, web data – including content in JavaScript and PDFs – is automatically fed into Attivio's Regulatory Active Management System, which runs on the Attivio Active Intelligence Engine.
- Attivio built a connector to their software so data output from Kapow robots is automatically transformed into the correct format.
Results
- Web data is collected automatically to support regulatory compliance.
- Based on their success with the financial company project, Attivio will be using Kapow robots to automate web data extraction on 200 additional sites.
They built the large job repository and keep it up to date with automatic web intelligence by Kapow.
Challenge
- Build the largest job repository
- Harvest millions of job listings from thousands of sites.
Solution
- Kapow automates aggregation of data from 2,000 sites and 8 million job listings.
- 3 developers built and maintain 1,650 Kapow robots to handle content aggregation.
Results
- Initial time to market was 3 months for 1 million listings.
- The new process is 10X faster than the company's previous “do-it-yourself” approach.
Viking relies on Kapow for secure, noise-free, real-time web intelligence.
Challenge
- Hedge funds depend on “asymmetrical intelligence” for competitive advantage. When information is commonly known, it is just noise. They needed high quality, noise-free data.
- Security is critical for hedge funds. Viking needed a solution they could build, maintain and host onsite.
- Much of their analysis is based on data over time. It can include seasonality and other factors. Therefore, robots accumulate and persist. They needed an industrial strength solution.
Solution
- They now use Kapow to automate “noise-free” web intelligence.
- Robots are developed and maintained onsite.
- Kapow's debugger accelerates development and improves data quality.
Results
- Kapow robots are 75% faster to develop and maintain than the custom-built robots they were using before.
- Data is secure and noise-free.
With Kapow, analysts and traders get real-time information on factors that affect trading price.
Challenge
- In 2007, the EU introduced requirements for greater transparency in the utilities market.
- Organizations affected by the new regulations had to deliver hourly, daily and weekly reports online.
- Deutsche Börse wanted to capture this information in real time to inform trading decisions.
Solution
- With Kapow, they were able to extract and transform data in real time, providing analysts and traders with critical information affecting trading price.
- The solution retrieves information from 150 websites.
Results
- Kapow was the only vendor who could fulfill the requirements in their timeframe.
- “With Kapow, we built powerful fundamental data models. Analysts and traders get critical factors that affect the price of a trade in real-time.” - Mario Schultz, Director, Market Data & Analytics, Deutsche Börse
With Kapow, Jobs2Web can extract all the web data they need, even on sites using JavaScript and Ajax.
Challenge
- Jobs2Web is the leading provider of interactive recruiting solutions.
- They were using Fetch to allow employers to track job applicants on Applicant Tracking System (ATS) websites, but they needed a solution that could extract information from sites using JavaScript and Ajax.
Solution
- Kapow was able to build robots to extract information from several of the problem sites in minutes.
- The company now uses Kapow to automate web data extraction and integration.
Results
- "With Kapow Technologies we are able to instantly pinpoint relevant matches among literally thousands that are buried in online job boards and automatically integrate this information into our system so our team of HR professionals can find the perfect job for Jobs2Web's clients." Ken Holec, CEO, Job2Web
This professional staffing services company automates business processes with Kapow robots, freeing up IT staff and resources.
Challenge
- Information about job opportunities is continuously updated.
- IT was inundated with requests for special reports to be run and delivered, detracting from everyday responsibilities.
- They needed to automate data extraction and interface with internal applications.
Solution
- Kapow robots dynamically take requests from the business as they come in via a web services infrastructure.
- The robots make use of the web interface of various IT systems and business applications to automate business operations and processes.
Results
- With Kapow, the company can scale to meet reporting demands from their business clients.
- Kapow removes the time required for IT staff to manually generate and deliver the requested reports.
- With Kapow, the IT group is more responsive to the business while freeing up their time to execute other responsibilities.
Plastic Jungle enables customers to sell, buy, exchange or donate gift cards. With Kapow, they can automatically log into gift card sites to validate gift card balance.
Challenge
- Plastic Jungle needed an efficient way to verify gift card balance.
- Manual processes were time consuming, expensive and error-prone.
Solution
- They use Kapow robots to automate the process of logging into gift card sites
- The robots are able to verify the balance on electronic gift cards that owners want to sell to Plastic Jungle.
Results
- The Kapow solution automates web intelligence and integrates seamlessly with their web-hosted application.
- By eliminating manual verification processes, the accuracy and volume of day-to-day transactions significantly improved.
With “directory assistance” from Kapow, this top 5 U.S. bank increases conference call security and engagement.
Challenge
- The bank had no visibility into who was joining and dropping out of conference calls.
- They wanted to maximize conference call participation and productivity while preventing unauthorized access.
Solution
- With Kapow, they were able to aggregate internal and external caller information from internal HR phone registries and public white pages.
- A custom web interface enables real-time call monitoring. They can now monitor all joins, drops and time spent on the call.
Results
- Eliminated unauthorized call access and security breaches.
- Demonstrated viable use of aggregated data for “long-tail” solutions.
- Required development of fewer than ten robots and a simple web interface to integrate data feeds.
With Kapow, the energy division of IHS can monitor thousands of sites for factors affecting supply and demand, without relying on offshore resources for web intelligence.
Challenge
- The energy division of IHS needed to monitor thousands of government regulatory product vendor websites for factors that influence oil and gas supply and demand.
- They were relying on off-shore resources to access web data, reducing their control of what is a strategic aspect of their business.
Solution
- With Kapow's automated web data acquisition, they can gather and aggregate critical web information in real time without outsourcing.
- No coding or manual data extraction is required.
Results
- Increased efficiency with automated data extraction.
- Enhanced and improved quality of master data.
- Enhanced product offerings based on better web data.
Innovative online lender automates web data extraction from partner sites with Kapow.
Challenge
- The lender's website allows customers to search through a database of thousands of entries, but each required manual web intelligence of photos, descriptions, prices and locations.
- Manual data entry was error prone and costly.
- When source data changed, there was no way to automatically update the information on the lender site.
Solution
- Kapow automates web data extraction from partner sites and integration to lender site.
- Data is automatically updated as changes occur, based on ongoing monitoring of partner sites.
Results
- Significant savings by eliminating manual data entry.
- Data on the lender site is up-to-date and accurate.
With Kapow, Live Matrix can extract data from any site to feed their innovative real-time events portal – even content from sites using JavaScript and Ajax.
Challenge
- The Live Matrix site is a real-time events portal, collecting and displaying data on all types of live events, but they had problems extracting content from sites using JavaScript and Ajax.
- They needed to scale quickly.
Solution
- Kapow enables them to extract data from sites using JavaScript and Ajax.
- Kapow's visual scripting environment enabled the Live Matrix developers to scale quickly.
Results
- Based on a successful POC involving 20 challenging sites, Live Matrix decided to move to Kapow for web intelligence.
- With Kapow's state of the art software, Live Matrix met their launch date and became the first TV Guide for the web.
Automating integration of new account information was key to business growth for XING, a leading social networking site for business professionals in Europe.
Challenge
- XING competes with LinkedIn and other social networking sites aimed at business professionals.
- To encourage adoption of their service, they wanted to automate integration of customer account and contact information from LinkedIn, FaceBook, Hotmail and Yahoo Mail to XING's profile and contact databases.
Solution
- With Kapow, a new XING user can enter his third party network credentials into XING's registration form and robots will log into the third party sites and collect all additional profile and contact information to enrich the new XING account.
- Collected contacts that are not currently registered XING users receive an invitation to join XING.
Results
- Customers save time by having all their contact and profile information entered automatically, making the new service instantly useful.
- XING gains valuable marketing information at no cost.