Big Data is growing fast. By 2015 the digital universe, as forecast by IDC, will hit 8 zettabytes (ZB). (1ZB = 1021 bytes, one sextillion bytes). Adding to the sheer volume is the remarkable velocity at which data is created. Every minute 600 new blog posts are published and 34,000 Twitter tweets are sent. If some of that data is about your organization, brand, products, customers, competitors, or employees wouldn’t you want to know?

Big data involves both structured and unstructured data. Traditional systems contain predominantly structured data. Unstructured data comes from general files; from smart phones and mobile devices; from social media like Twitter, Facebook, and others; from RFID tags and other sensors and meters; and even from video cameras. All can be valuable to organizations in particular contexts.

Large organizations, of course, can benefit from Big Data, but midsize and small businesses can too. A small chain of pizza shops needs to know the consumer buzz about their pizza as much as Domino’s.

IBM describes a 4-step process for tapping the value of Big Data: align, anticipate, act, and learn. The goal is to make the right decision at the point of maximum impact. That might be when the customer is on the phone with a sales agent or when the CFO is about to negotiate the details of an acquisition.

Align addresses the need to identify your data sources and plan how you are going to collect and organize the data. It will involve your structured databases as well as the wide range of enterprise content from unstructured sources. Anticipate addresses data analytics and business intelligence with the goal of predicting and shaping outcome. It focuses on identifying and analyzing trends, making hypotheses, and testing predictions. Act is the part where you put the data into action, whether it is making the best decision or taking advantage of a new pattern you have uncovered. But it doesn’t stop there. Another payoff from Big Data comes from the ability to learn, for the purpose of refining your analytics and identifying new patterns based on subsequent data.

Big Data needs to be accompanied by appropriate tools and technology. Earlier this month, IBM introduced three task-specific Smarter Analytics Signature Solutions. The first addresses anti-fraud, waste, and abuse by using sophisticated analytics to recommend the most effective remedy for each case. For example it might recommend a different letter requesting payment in one case but suggest a full criminal investigation in another.

The second Signature Solution focuses on next-best-action. This looks at the various data uses real-time analytics to predict customer behavior and preferences and recommend the next best action to take with regard to a customer, such as to reduce churn or up-sell.

The third Signature Solution, dubbed CFO Performance Insight, works on a collection of complex and cross-referenced internal and external data sets using predictive analytics to deliver increased visibility and control of financial performance along with predictive insights and root-cause analyses. These are delivered via an executive-style dashboard.

IBM isn’t the only vendorr to jump on the Big Data bandwagon. EMC has put a stake into this market. Oracle, which has been stalking IBM for years, also latched onto Big Data through Exalytics, its in-memory analytics product similar to IBM’s Netezza. Of course, small players like Cloudera, which early on staked out Hadoop, the key open source component of Big Data, also offer related products and services.

Big Data analytics will continue as an important issue for some years to come. This blog will return to it time and again.

http://bottomlineit.wordpress.com/2012/03/26/big-data-analytics-defines-top-performers/

A nice, short, 2 minute video from edCetra Training with some good facts about big data and data analysis.

The digital universe is 10 times the size it was in 2005
Greater literacy and cloud computing are helping fuel big data
80% of companies data is unstructured – difficult to analyze
Employees spend 2 hours per day searching for the right information

Visualization opens up the channel of communication between the technologists who create the data and the business people who act upon it.  Data visualization tools, such as mashups, executive dashboards, KPI and performance scorecards and other data visualization technology, are becoming more popular and necessary to deal with mind numbing charts and exponential data growth.

However, the C-Suite has heard about the promise of dashboards and interactive scorecard for a few decades now and is typically dissatisfied with what they get from IT and the speed at which they get it. The big difference is that visualization technologies have finally advanced to a level where they can give actionable intelligence to the right people at the right time at the right place.

Lets take for instance an a mobile BI solution using a tool such as an Apple iPad. This gives the business executive the ability to manipulate the data with the ease of reading an e-book.  The visualization library that you can draw upon to create an interactive experience on the iPad includes:

Bi24 data visualisation

There are three critical business requirements addressed by such a solution. These are:

1) Ease of Use: – Pictures and graphs are easier to read than numbers and more insightful than words on a page. The picture of the water levels from the recent tsunami shown in this posting can communicate more information than any seismic data chart.

2) Interactivity – Business users want the ability to change parameters and do their own queries but are loath to get on a keyboard to do write a query and would rather not go through discussion cycle an IT manager to look at data in a different way.

3) Mobility – Any time any place access is a fundamental need of the modern business environment. The wide-scale adoption smartphones and tablets provides an ideal platform to carry out this requirement.

We predict that mobile BI using visualized data which is easy to manipulate will be a giant step in aligning IT with business in 2011. A picture transcends language barriers and communication challenges across organizational silos more effectively than any form of numerical data.

There are those who argue that the emphasis on visualization (simplicity) minimizes the advanced technology that is a bulk of the work in information management. All the work that goes on in processing data is of no value if it cannot support a better business decision. The visualized data is all that matters. The tip of the iceberg is the iceberg.

Thanks to Shirish Netke

Related articles

• Data Visualizations Are More Than Just Pictures (forbes.com)
• Data Visualization Principles: Lessons from Tufte (seomoz.org)
• The Beauty of Infographics – David McCandless Explores the Power of Data Visualization (TrendHunter.com) (trendhunter.com)

Target  was founded in 1902 and is headquartered in Minneapolis, Minnesota. Target operates over 1,750 stores in 49 states under Target and SuperTarget names. It offers general merchandise products through its Website, Target.com. The company distributes its merchandise through a network of distribution centers, as well as third parties and direct shipping. Additionally, it offers credit to guests through its branded proprietary credit cards.

Data and Predicting Pregnant Customers

Andrew Pole had just started working as a statistician for Target in 2002, when two colleagues from the marketing department stopped by his desk to ask an odd question: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that? ”

As the marketers explained to Pole  new parents are a retailer’s holy grail. Most shoppers don’t buy everything they need at one store. Instead, they buy groceries at the grocery store and toys at the toy store, and they visit Target only when they need certain items they associate with Target — cleaning supplies, say, or new socks or a six-month supply of toilet paper. But Target sells everything from milk to stuffed animals to lawn furniture to electronics, so one of the company’s primary goals is convincing customers that the only store they need is Target. But it’s a tough message to get across, even with the most ingenious ad campaigns, because once consumers’ shopping habits are ingrained, it’s incredibly difficult to change them.

There are some periods in a person’s life when old routines fall apart and buying habits are suddenly in flux. One of those moments is right around the birth of a child, when parents are exhausted and overwhelmed and their shopping patterns and brand loyalties are up for grabs. But as Target’s marketers explained to Pole, timing is everything. Because birth records are usually public, the moment a couple have a new baby, they are almost instantaneously barraged with offers and incentives and advertisements from all sorts of companies. Which means that the key is to reach them earlier, before any other retailers know a baby is on the way. Specifically, the marketers said they wanted to send specially designed ads to women in their second trimester, which is when most expectant mothers begin buying all sorts of new things, like prenatal vitamins and maternity clothing. “Can you give us a list?” the marketers asked.

“We knew that if we could identify them in their second trimester, there’s a good chance we could capture them for years,” Pole told me. “As soon as we get them buying diapers from us, they’re going to start buying everything else too. If you’re rushing through the store, looking for bottles, and you pass orange juice, you’ll grab a carton. Oh, and there’s that new DVD I want. Soon, you’ll be buying cereal and paper towels from us, and keep coming back.”

The desire to collect information on customers is not new for Target or any other large retailer, of course. For decades, Target has collected vast amounts of data on every person who regularly walks into one of its stores. Whenever possible, Target assigns each shopper a unique code — known internally as the Guest ID number — that keeps tabs on everything they buy. “If you use a credit card or a coupon, or fill out a survey, or mail in a refund, or call the customer help line, or open an e-mail we’ve sent you or visit our Web site, we’ll record it and link it to your Guest ID,” Pole said. “We want to know everything we can.”

Also linked to your Guest ID is demographic information like your age, whether you are married and have kids, which part of town you live in, how long it takes you to drive to the store, your estimated salary, whether you’ve moved recently, what credit cards you carry in your wallet and what Web sites you visit. Target can buy data about your ethnicity, job history, the magazines you read, if you’ve ever declared bankruptcy or got divorced, the year you bought (or lost) your house, where you went to college, what kinds of topics you talk about online, whether you prefer certain brands of coffee, paper towels, cereal or applesauce, your political leanings, reading habits, charitable giving and the number of cars you own.  

All that information is meaningless, however, without someone to analyze and make sense of it. That’s where Andrew Pole and the dozens of other members of Target’s Guest Marketing Analytics department come in.

Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them. “But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”

Mechanics of Identifying Pregnant Shoppers

And among life events, none are more important than the arrival of a baby. At that moment, new parents’ habits are more flexible than at almost any other time in their adult lives. If companies can identify pregnant shoppers, they can earn millions.

The only problem is that identifying pregnant customers is harder than it sounds. Target has a baby-shower registry, and Pole started there, observing how shopping habits changed as a woman approached her due date, which women on the registry had willingly disclosed. He ran test after test, analyzing the data, and before long some useful patterns emerged. Lotions, for example. Lots of people buy lotion, but one of Pole’s colleagues noticed that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths, it signals they could be getting close to their delivery date.

As Pole’s computers crawled through the data, he was able to identify about 25 products that, when analyzed together, allowed him to assign each shopper a “pregnancy prediction” score. More important, he could also estimate her due date to within a small window, so Target could send coupons timed to very specific stages of her pregnancy.

One Target employee I spoke to provided a hypothetical example. Take a fictional Target shopper named Jenny Ward, who is 23, lives in Atlanta and in March bought cocoa-butter lotion, a purse large enough to double as a diaper bag, zinc and magnesium supplements and a bright blue rug. There’s, say, an 87 percent chance that she’s pregnant and that her delivery date is sometime in late August. What’s more, because of the data attached to her Guest ID number, Target knows how to trigger Jenny’s habits. They know that if she receives a coupon via e-mail, it will most likely cue her to buy online. They know that if she receives an ad in the mail on Friday, she frequently uses it on a weekend trip to the store. And they know that if they reward her with a printed receipt that entitles her to a free cup of Starbucks coffee, she’ll use it when she comes back again.

In the past, that knowledge had limited value. After all, Jenny purchased only cleaning supplies at Target, and there were only so many psychological buttons the company could push. But now that she is pregnant, everything is up for grabs. In addition to triggering Jenny’s habits to buy more cleaning products, they can also start including offers for an array of products, some more obvious than others, that a woman at her stage of pregnancy might need.

Pole applied his program to every regular female shopper in Target’s national database and soon had a list of tens of thousands of women who were most likely pregnant. If they could entice those women or their husbands to visit Target and buy baby-related products, the company’s cue-routine-reward calculators could kick in and start pushing them to buy groceries, bathing suits, toys and clothing, as well.

Notes and References

1. Source: Charles Duhigg,  How companies learn your secrets, New York Times, Feb 16 2012
2. Charles Duhigg ”The Power of Habit: Why We Do What We Do in Life and Business.”
3. Stephen Colbert has a funny segment on this article: http://www.colbertnation.com/the-colbert-report-videos/408981/february-22-2012/the-word—surrender-to-a-buyer-power
4. Target  operates general merchandise stores. The company offers household essentials, including pharmacy, beauty, personal care, baby care, cleaning, and paper products; hardlines consisting of electronics, video game hardware and software, music, movies, books, computer software, sporting goods, and toys; apparel and accessories comprising apparel for women, men, boys, girls, toddlers, infants, newborns, intimate apparel, jewelry, accessories, and shoes; and home furnishings and decor products, such as furniture, lighting, kitchenware, small appliances, home decor, bed and bath, home improvement, and automotive products, as well as seasonal merchandise, including patio furniture, and holiday decor. It also provides food and pet supplies consisting of dry grocery, dairy, frozen food, beverages, candy, snacks, deli, bakery, meat, produce, and pet supplies. Target Corporation sells its merchandise products under private-label, and exclusive licensed and designer brands. In addition, it provides in-store amenities.

Thanks to practicalanalytics.com

Standards and enterprise-grade platforms will continue to be important, but individual BI tools with the functionality to show a single version of the truthwill trump standards.

Enterprises will learn to live with multiple BI tools.

Business Intelligencewill continue to move away from the IT team and into the hands of the business user.

Self Service BI will bring decision makingto the desktop and will become the name of the game.

Mobile BIwill go mainstream. Tomorrows decisions will need to be made when and where they need to be made, not “when I get back to the office”.

Cloud BIwill continue to chip away at on-premises BI, but it’s still a long road ahead.

BI-specific DBMSes(in-memory, others) will go mainstream.

Big datawill start to move out of silos and into enterprise IT. IT will start to learn how to live with it.

BI users will start demanding and vendors will start delivering Bi tools that integrate with email and collaboration platforms. Just integrating BI with Excel is no longer enough.

www.c24.co.uk

Why? Poor design (obviously) and the fact that my waist is not quite what it used to be. Like it or not, I’m gradually expanding…

I read an interesting article today that was titled “Getting Rid of Data — Why is it so hard” and for some reason found some parallel in what’s happened with my jeans…

The article is about Information Governance and how important it is to have good processes and policies in place so that you know with confidence what data needs to be retained and what data can be disposed of — stopping the burgeoning, un-wielding growth of data (or my waist as I continue to try to draw some sort of parallel…)

Some of my previous blogs have been talking about the Tsunami of data from social media — Supersized data that is both difficult to make sense of and expensive to retain.

It’s easy to understand why doing a bit of data spring clearing can clear out some space and enable you to carry on cramming stuff into what ever storage is currently available (my jeans again) just in case you need it some time in the future.

However… More than ever, technologies are changing and evolving as data continues to grow and grow.

Disks are cheap and with the adoption of “big data” solutions such as Hadoop and our own search engine based technology CXAIR, size is less and less of an issue.

When I started in IT back in 1987, I was working on mainframe systems written in Cobol and IDMSX. I was working on hospital Patient Administration Systems where due to limitations in hardware we were always extremely careful of data storage and retention. We used to hold current data and historic data for about 3 months before it was archived off to tape and microfiche.

These days we can slap it on disk and keep it there virtually indefinitely. With all of the laws around compliance and transparency, keeping your data available is a good and comforting thing.

The only issue is when you do need access to some old data, how do you find it?

If it’s stored in an old legacy database, then expect to have to get IT on the case and wait for hours for the data to come back.

If it’s stored in something like a search engine then users can search around perform ad—hoc queries and analysis across the data themselves and get results back in seconds.

Times are changing and the way we store access and retain data is going through a technical revolution at the moment.

Data Governance is important but so is the ability to keep hold of data for historical analysis and that “just in case, ad-hoc” requirement that bites you on the backside as soon as you’ve deleted or made the data less accessible.

So what choices do I have?

Get rid of some of the excess (go on a diet) or buy some bigger (more scalable) and more appropriately designed (latest technology) jeans.

Time to get the credit card out…

Thu, 17/11/2011 – 11:12 — Richard Lewis

CenturyLink has released the following infographic that highlights the exponential increase in data, and the implications of Big Data on enterprises. It is predicted that video and mobile devices will be major contributors in driving the creation of 7.9 zetabytes of data in 2015. Big Data will entail 1.5 times more IT professionals managing 75 times more data moving through enterprise data centers. Image via CenturyLink Find the original post and image here. Related articles Infographic: Data Deluge – 8 Zettabytes of Data by 2015 (readwriteweb.com) Big Data and Little Data (forbes.com)

spotted on a little bit of this, a little bit of that