According to sources, Microsoft is close to striking a nonexclusive data-mining deal with Twitter to integrate the microblogging service’s full feed into the results of its Bing search service.

News of the deal, which was still being worked on by engineers and execs at both the software giant and the start-up as late as yesterday, could even come as early as today at the Web 2.0 Summit conference in San Francisco.

Qi Lu, the head of Microsoft’s online services division, is being interviewed at the event at 11:30 am PDT.

Note: Lu is scheduled to appear onstage after Yahoo (YHOO) CEO Carol Bartz, who is now too sick to attend, the company said. She also missed Yahoo’s third-quarter conference call due to an unspecified illness.

Thus, Lu is the main event of Web 2.0 tomorrow morning and it would be a good place to make a big announcement, if the talks are successful.

But sources close to the situation caution that the deal could still run into a snag and was not yet complete, although it seems more likely than not that a deal will soon be struck with Microsoft (MSFT) first and then Google (GOOG), which is the other company Twitter has been negotiating with.

BoomTown had previously reported that Twitter was in advanced talkswith both the search rivals about just such a real-time search arrangement.

There could be other possibilities on the table, said sources, such as one of the big companies advertising on Twitter or any number of other Twitter partnerships with either one.

When asked about the talks onstage at Web 2.0 yesterday, Twitter CEO Evan Williams turned coy, according to numerous reports, joking “Whose deals?”

But, in fact, Twitter, which is based in San Francisco, is very much engaged in dealmaking in this regard.

Here is what I wrote previously, as to what and why:

Sources said a number of scenarios are being discussed to compensate Twitter for its huge and potentially valuable trove of real-time and content-sharing information, generated from the data stream of billions of tweets from its 54 million monthly users.

These include a number of structures, including a payment of several million dollars to Twitter, along with various revenue-sharing proposals that would give Twitter a piece of the revenue made from search results.

The deals, stressed sources close to the situation, are nonexclusive, especially because Twitter’s management is keen to remain independent and also nonpartisan in the growing search battle between Google and Microsoft.

This means Yahoo–which recently struck a search-technology and online-advertising partnership with Microsoft–could also license Twitter’s feed to make its search results even more robust, although Boomtown could not determine if the company is in talks with the San Francisco start-up.

Sources said it is also possible that no agreement would be reached with either company.

And execs at Twitter, Microsoft and Google had no comment when asked about talks.

But doing these kinds of data deals with big search players does make a lot of sense, since it would be hard for Twitter to turbocharge its own search engine without running into the big cash-laden guns at both Google and Microsoft, which recently launched its new Bing search service.

Twitter is, instead, seeking to create a large open platform, which many could plug into, from search engines to marketers to publishers to developers.

Twitter has also been considering offering premium services to these groups and is contemplating some form of advertising offering.

But, most of all, Silicon Valley’s hot start-up is focusing now on spurring growth and engagement, along with fine-tuning its product offering.

Being deeply integrated into big search services would give Twitter a huge footprint.

Microsoft had already done a small experiment this past summer integrating Twitter data into search results, starting with tweets of bloggers like me.

How much indexing of its data Twitter will allow is unclear, but the company has certainly bought itself time to think carefully about all its options, given that it now has a lot of money in the bank.

Late last month, Twitter raised another $100 million in new funding, after already having raised $55 million.

This has given it a $1 billion valuation, despite negligible revenue.

The valuation also effectively stated that the innovative company was pretty much putting itself out of play to be acquired and is very interested in forging its own destiny.

Both Google and Microsoft execs have contemplated the idea of buying Twitter in the past, although no serious talks ever moved forward.

If they both strike data deals with Twitter, they will get the next best thing–an ability to offer all the information disseminated on Twitter in search results.

Popularity: 2% [?]

       

Every year, the market-research firm Millward Brown conducts a survey to determine the economic worth of the world’s brands — in other words, to put a dollar value on the many corporate logos that dominate our lives. Lately the firm’s results have been stuck on repeat: Google has claimed the top spot for the past three years. The most recent report values Google’s brand — those six happy letters that herald so many of our jaunts down the Web’s rabbit hole — at more than $100 billion.

What’s astonishing about this stat is how effortlessly Google seems to have earned the public’s affection. Other companies — Microsoft, Coke, IBM, McDonald’s — spend enormous sums to stay in the consciousness. Google, which makes most of its money from ads, rarely advertises itself. Telling the world how well it does what it does just isn’t Google’s way.(See pictures of work and life at Google.)

But Google’s humility is being tested as never before. The firm’s headquarters in Mountain View, Calif., seem besieged by competitors gaining new momentum. Even nominal allies are questioning the company’s motives and long-term plans. In July, Google’s largest competitors, Microsoft and Yahoo!, agreed to work together in an attempt to dethrone it as the world’s dominant search engine. The deal, which awaits government approval, would create a first: a tenacious, well-financed search rival.

Conflicts are beginning to take place in other areas where Google has ventured. That includes e-mail and office programs (Gmail, Google Docs), a cell-phone operating system (Android) and a Web browser (Chrome). Google scans and sells books, runs a phone system and is even working on a desktop operating system to rival Windows. CEO Eric Schmidt recently stepped down from Apple’s board of directors because the two companies now compete in so many areas. The U.S. Justice Department is investigating a legal settlement between Google and the publishing industry over the company’s book-scanning service, and Christine Varney, Justice’s antitrust chief, said she sees Google as a “problem.”

At the moment, Google’s most pressing problem is Microsoft. The software giant is spending $100 million to market its new search engine, Bing — and in the process, to get us all bummed about Google. Bing’s slick ads are unavoidable and blistering. They suggest that Google is broken, that it rarely leads us to what we’re looking for and turns us all into blathering zombies who spew out search keywords in casual conversation.(See the top 10 TV ads of 2008.)

Microsoft claims Bing isn’t even a search engine — it’s a “decision engine.” What that means isn’t exactly clear. Bing seems to work the same way Google does: type in some keywords, it gives you some Web results. But the marketing shows signs of gaining traction. According to the media-metrics firm comScore, Bing captured 8.9% of the search-engine queries in July, a tiny increase from 8.4% in June. “All of us in the search industry were surprised by Bing,” says Anna Patterson, a former Google engineer who has since gone on to found Cuil (pronounced Cool), one of the many smaller search start-ups in Google’s shadow. “It’s the first time you have someone with deep pockets that’s willing to lose money in order to compete with Google, and they’re willing to stick with it over the long term.”(See the top 10 Microsoft moments.)

Google says it isn’t worried, and publicly at least, the company is pretending not to notice Bing. The search engine is Google’s cash cow, and the firm constantly pours resources into improving it — hiring the industry’s brightest and most experienced engineers, paying them handsomely and letting them work on what is effectively the world’s largest data-mining project. Just this month, Google unveiled a project it calls Caffeine, a massive overhaul of its back-end infrastructure that promises to create a faster, more accurate and more comprehensive search engine. “We aren’t resting,” says Gabriel Stricker, a Google spokesman. “We’re continuing to innovate — I’M FEELING LUCKY is getting luckier all the time.”

This sort of constant improvement pays off: two-thirds of all searches in the U.S. are now conducted through Google — about 7 billion a month. Yahoo! has less than 20% of the market, and Microsoft less than 10%. Despite Microsoft’s claims, most people think Google works pretty well as it is.

Microsoft argues that the Yahoo! deal will help change that perception. If the partnership is approved, Microsoft will take over Yahoo!’s search engine — type in “Britney Spears” at Yahoo! and you’ll get results provided by Bing. Microsoft points out that search engines get smarter as more people use them; if a search engine notices lots of clicks on Spears’ music videos after searching for the pop star, it can begin to highlight those videos in future searches. That’s how the Yahoo! deal will help Bing beat Google, Microsoft says. By massively expanding its market share to a potential 26%, Microsoft will get access to a much broader pool of user data, which will in turn make it better at predicting what you want when you search.

Google pooh-poohs this claim. Hal Varian, the company’s chief economist, has pointed out that most search engines look at only a small sample of their data in order to improve their results. In other words, Microsoft already has enough data to learn from its users. “It’s not the quantity or quality of the ingredients that make a difference. It’s the recipes,” Varian told CNET. The recipes are Google’s proprietary algorithms, which it has slaved over for more than a decade. They’re Google’s ultimate competitive advantage, and Google believes they’ll help it weather the coming assault.(See the best social networking applications.)

Privately, Googlers will tell you that the Bing ads rankle. They describe them as misleading and unfair, painting a picture of Google that doesn’t match reality. Maybe, but Microsoft — a company not previously known for its marketing savvy — is taking a page out of a 1960s Procter & Gamble playbook: create a problem consumers don’t know they have, then solve it. Bing!

Can Google play defense if Bing starts to move the needle? Google’s first instinct has always been to innovate its way out of trouble. But there are a number of features in Google’s search engine that most of the public is unaware of. Like how it can give you the local weather and movie times and perform currency conversions with a single search query. It’s not in Google’s DNA to run confrontational ads, but it’s easy to imagine a campaign that shows off all the amazing things your friendly search giant can do.(See 10 ways Twitter will change American business.)

At the moment, Google derives about 97% of its revenue from advertising. Barry Schwartz, CEO of the Web consulting firm RustyBrick and an editor at Search Engine Land, says that some at Google have to be getting a little jittery that the company’s entire revenue stream rests on a single product. “They keep downplaying that they’re competing with other companies — whenever they pitch something like Android or their new Chrome OS, they say it’s just an attempt to get people to use the Web more,” Schwartz says. But here’s the irony: Google faces a problem very similar to the one plaguing Microsoft, which itself makes the bulk of its money from just two products — Windows and Office. Each company sees the other’s business as its own path forward. The rest of us, we’re just bystanders.

Popularity: 2% [?]

       

With Bing, Microsoft is trying to reinvigorate its role in the search business. It has also inadvertently brought renewed attention to the problem of illicit pharmacies operating on the Internet. The attention on Bing came earlier this month with the results of a study that examined Internet pharmacy ads (PDF) on Microsoft’s revamped search engine. The study, conducted by LegitScript, an online pharmacy verification service, and KnujOn, an Internet compliance company, found that 90 percent of the reviewed Internet pharmacy advertisements were from fake or illegal Internet pharmacies. It also found that most of the Internet pharmacies reached through sponsored ads on Bing did not require a valid prescription. Sponsored ads are links, paid for by companies hawking products and services, that turn up at the top of search results pages alongside noncommercial links.

“We were able to purchase potentially addictive drugs without a prescription or any age verification via Bing.com ads,” LegitScript President John Horton told CNET News. “We also received counterfeit medication. Microsoft profits from these illegal ads, which put Internet users at risk.” But the problem isn’t confined to Bing. For all the buzz generated by Bing–which debuted in June, replacing Microsoft’s Live Search–it’s still only the third most-used search tool, dwarfed by first-place Google and also well behind Yahoo. And those search engines themselves are no strangers to ads for illicit pharmacies. The problem has also been around since consumers began flocking to the Internet more than a decade ago. In 2003, for instance, Yahoo’s Overture unit bowed to pressure from pharmacy groups and stopped selling search-related advertising to unlicensed online pharmacies. That also spelled an end to the troublesome ads on Microsoft’s MSN portal, at that time a significant partner of Overture. Over the last decade, the situation has evolved to bring new challenges. “In the early years of the Internet, it was a case of entrepreneurs not understanding the legal requirements for the dispensing of drugs. Later, it was the push by senior citizens and public officials to obtain drugs that were cheaper than medications available in the U.S.,” said Carmen Catizone, executive director of the trade groupNational Association of Boards of Pharmacies. “At the present time,” said Catizone, who vouched for the research by LegitScript, “the Internet has become a haven for drug seekers and abusers, particularly (regarding) controlled substances. It is a much more serious and dangerous phase of the Internet.” Rogue online pharmacies sell a wide range of medications, from the sleep aid Ambien to the muscle relaxant Soma and the erectile dysfunction treatments Viagra and Cialis. The NABP lists only 18 certified and recommended online drugstores at its Web site, while more than 3,800 are non-compliant and not recommended The response from Redmond Microsoft disputes LegitScript’s claim that 90 percent of the sponsored Internet pharmacy ads on Bing are fake or illegal, adding that it is working to weed out the rogue advertisers that do slip through. The company uses an Internet pharmacy verification service called PharmacyChecker–a competitor of LegitScript–to ensure that its sponsored prescription drug advertisements are legitimate.

“Our editorial system used PharmacyChecker’s list of approved advertisers as our guideline in this case,” Microsoft said in a statement to CNET News. “During a quality analysis of the sponsored results for the most frequently used Pharma terms (a more exhaustive set than was used in the study), we found that the actual rate of violation was closer to 15 (percent).” The Bing study is supposed to be the first in a series of reports from LegitScript, but it’s unclear who’ll be next on the list, or when such a report might come out. Asked if LegitScript had tracked or plans to track ads on Google or Yahoo, Horton said he couldn’t comment. (Editors’ note, 5:47 a.m. PDT: On Tuesday, LegitScript and KnujOn released their report on Yahoo (PDF), charging that more than 80 percent of the pharmacy ads that turn up in Yahoo searches violated state or federal laws.) The NABP’s own analysis of search results from Google and Yahoo turned up many drug ads from sketchy purveyors. In an April 2008 study of 558 Internet drug outlets, the NABP discovered that nearly half were selling prescription drugs illegally or unprofessionally. Out of 258 rogue pharmacies, 191 did not require a valid prescription, 118 offered foreign or non-FDA approved drugs, and 91 were located outside the U.S. but offered to ship to U.S. customers, all of which is illegal. The NABP says that when informed of the study, Google responded that it would start using the association’s list of certified Internet pharmacies to filter out rogue sites, while Yahoo said it was relying on PharmacyChecker to help screen out illegal drug vendors. Washington jumps in It’s not just the private sector that’s targeting rogue pharmacies. Congress last year passed the Ryan Haight Online Pharmacy Consumer Protection Act, which went into effect this past April. Named for an 18-year-old who died from an accidental overdose of drugs he bought online, the legislation bans the sale of prescription drugsover the Internet without a valid prescription. The Ryan Haight act should be a significant step in the right direction of trying to control this open channel of distribution, according to Susan Foster, director of policy research and analysis for Columbia University’s National Center for Addiction and Substance Abuse (CASA). “It will clarify the law,” she said. “The problem was that the Controlled Substances Act (passed in 1970) was written prior to the Internet, so there were questions about the online sale of drugs and what was a legitimate doctor/patient relationship.” Foster is familiar with the LegitScript study and said that its results were consistent with her organization’s own findings. CASA’s July 2008 report was its fifth annual study examining the online availability of prescription drugs, and it focused on substances including Valium, Xanax, and Ritalin. CASA found that the number of Web sites selling prescription drugs rose each year from 2004 to 2007, then dipped somewhat in 2008. In 2008, the number of online drugstores CASA found that didn’t require a prescription was around 85 percent. Out of 365 sites discovered advertising or selling prescription drugs, only two were certified by the NABP as legitimate. The group found sponsored ads for rogue pharmacies prominently displayed on both Google and Yahoo, and also spotted similar ads on MSN/Live Search. CASA was unable to issue another study this year due to lack of funding. When asked by CNET News to discuss this issue, Google declined to comment. The company’s Pharmacy Qualification requirements are available online. Yahoo did not respond to a request for comment. Back to Bing Overseas pharmacy sites are a stubborn problem when it comes to online drug ads, as is the involvement of criminal networks responsible for a significant portion of the world’s spam, fake drugs, and cybercrime. (The Ryan Haight act regulates only online drugstores in the U.S.)

“These Bing.com ads aren’t real pharmacies,” said Garth Bruen, president of KnujOn, which tracks spam and other online and e-mail-based threats and conducted the Bing study along with LegitScript. “These types of sites are usually the product of organized crime and vast illicit drug networks, many of them based in Russia and Eastern Europe, that deceive, defraud, and poison Internet users.” In the LegitScript study, researchers found a total of 69 pharmacies by running random searches on Bing for prescription drugs, using terms such as “online pharmacy” and “buy Viagra” to find sponsored ads. Of the 69 drug vendors uncovered, only seven were certified as legitimate by LegitScript. The rest were considered to be operating illegally in one way or another. The authors took a closer look at 10 of the 69 online drugstores. None of the 10 required a valid prescription. Orders were placed with two of them. Of the two drugs received, both were tested and one was found to be counterfeit. “It is important to emphasize that the ten advertisers analyzed are not engaged in minor violations of pharmacy law,” the report says. “Rather, they are wholly fraudulent websites run, in most cases, by criminal networks. They sell unapproved or counterfeit drugs, including addictive medications, without any requirement of a prescription. The drugs come from places like Calcutta, India, which is a violation of US drug safety regulations. In several cases, the websites are operated by individuals in Russia or Eastern Europe, not US-based pharmacists. In short, these ‘Internet pharmacies’ are neither pharmacies at all, nor run by pharmacists: they are simply online street corners run by drug dealers.” Horton said that LegitScript has tracked more than 41,000 Internet pharmacies, and at best only about 2 percent to 3 percent of those are legitimate. The authors said they also uncovered security holes in Microsoft’s online advertising program. A rogue Internet pharmacy called store.k2med.com was able to advertise under the name of a U.S.-licensed pharmacy but redirect traffic to the fake Web site. This same security flaw was found in other cases. In response to the LegitScript/KnujOn study, an August 7 blog post written by Microsoft AdCenter Community blogger Carolyn Miller said the company believes “the advertisers noted in the report found a way to work around the PharmacyChecker.com verification process after being validated to advertise on Bing. These rogue advertisers manipulated the system by ‘hijacking’ and/or misusing landing pages. Our internal teams are continuing to investigate how these advertisers sidestepped the policy.” As immediate fixes, the blog noted that Microsoft has reviewed all pharma-related keywords to filter out any advertisers in violation of company policy. Microsoft also said its editorial team is validating the claims in the LegitScript report to investigate the hijacking and misuse of landing pages. For the longer term, the company said it is reviewing its processes to document how these advertisers got onto the system. Microsoft says that its guidelines clearly require online pharmacies that advertise on Bing to adhere to U.S. laws. And in a statement this week to CNET News, Microsoft said: “At this point, we believe that PharmacyChecker’s system worked as designed. It is important to note, however, that PharmacyChecker’s list of validated pharmacies is only one part of a complex system of editorial processes that enforce policy. We do expect this experience will drive system-wide improvements.”

Popularity: 2% [?]