a bit like ifttt

"Crossfilter is a JavaScript library for exploring large multivariate datasets in the browser. Crossfilter supports extremely fast (<30ms) interaction with coordinated views, even with datasets containing a million or more records; we built it to power analytics for Square Register, allowing merchants to slice and dice their payment history fluidly."

"Bring meaning to your metrics and stories to your dates with our API of interesting number facts."

clever! "Super-quick dummy JSON
Ever needed lots and lots of bullshit data? Now you can get it whenever you need it."

"GitHub Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis.

Privacy patterns are design solutions to common privacy problems — a way to translate "privacy-by-design" into practical advice for software engineering. We believe design patterns can help document common practices and standardize terminology and while we're starting with a set of patterns for location-based services, we hope to build a living, community space where all can contribute their own patterns.

"In my lab, a team working with Konica Minolta has developed a system to automatically detect learning. In one study we used the Kinect to gather nonverbal data during one-on-one, student-teacher interactions. Later, we used that data to predict the students’ test scores. The early results are encouraging, though preliminary, with around 10 movements (those relating to the shoulder and elbow, for instance) being the most predictive. What makes this type of experiment so powerful is the “bottom up” nature of the research. Instead of looking for specific known gestures, like nodding or pointing, we can mathematically uncover subtle movement patterns, many of which would not be noticed by the human eye. Just imagine if teachers, based on a small sample of their students’ nonverbal behavior, could instantly detect which students needed extra attention or specialized assignments."

"Precog is a big data analytics platform that helps you build apps that are intelligent and insightful."

""

via @rem: Drag and drop images and get a data url: http://t.co/mrxXLwge (includes trimming of transparent space) no server required :)

"Today we're introducing Google Fusion Tables on Labs, an experimental system for data management in the cloud. It draws on the expertise of folks within Google Research who have been studying collaboration, data integration, and user requirements from a variety of domains. Fusion Tables is not a traditional database system focusing on complicated SQL queries and transaction processing. Instead, the focus is on fusing data management and collaboration: merging multiple data sources, discussion of the data, querying, visualization, and Web publishing. We plan to iteratively add new features to the systems as we get feedback from users."

"The data preparation phase is often known as ETL (Extract Transform Load) or the data factory. "Factory" is a good analogy because it captures the essence of what is being done in this stage: Just as a physical factory brings in raw materials and outputs products ready for consumers, so a data factory brings in raw data and produces data sets ready for data users to consume. Raw data is loaded in, cleaned up, conformed to the selected data model, joined with other data sources, and so on. Users in this phase are generally engineers, data specialists, or researchers.

The data presentation phase is usually referred to as the data warehouse. A warehouse stores products ready for consumers; they need only come and select the proper products off of the shelves. In this phase, users may be engineers using the data for their systems, analysts, or decisionmakers.

Given the different workloads and different users for each phase, we have found that different tools work best in each phase. Pig (combined with a workflow system such as Oozie) is best suited for the data factory, and Hive for the data warehouse."

"OTS creates a middle ground between a browser app fetching Web documents from any server and an app that connects to a dedicated API server to fetch data objects.

In OTS you just have a single browser app that fetches those data objects from any server, instead of documents, and instead of connecting to any particular API server.

These OTS objects are simple, open, linked, data structures representing mobile stuff in pre-defined formats: people, places, dates, photos, messages, feeds, lists, galleries, maps, calendars, etc. They are linked together and linked into the Web.

Then the OTS browser app can show these Web objects and allow the user to interact with them. You only have one OTS browser app, but it's a simple, mobile-dedicated browser that can access an entire Web-full of data in OTS objects."

"The story of the Internet has been deflationary from Amazon to Craigslist to iTunes.  And so too will be Factual.  They have built algorithms that automatically crawl the web for the world’s best structured data and use heuristic techniques to ensure the quality of the data.  They have built tools to store the data but also to allow 3rd-party developers to rapidly consume or even write data to their tables.<br />
<br />
"

"links data across APIs so that you can build applications using the best data sources on the Web and put your own data into context."

OStatus is an open standard for distributed status updates. Our goal is a specification that allows different messaging hubs to route status updates between users in near-real-time

"In Google Refine, clustering refers to the operation of "finding groups of different values that might be alternative representations of the same thing". For example, the two strings "New York" and "new york" are very likely to refer to the same concept and just have capitalization differences, and "Gödel" and "Godel" probably refer to the same person.<br />
<br />
It is worth noting that clustering in Google Refine works only at the syntactic level (the character composition of the cell value) and while very useful to spot errors, typos and inconsistencies it's by no means enough to perform effective semantically-aware reconciliation. This is why Google Refine uses external semantically-aware reconciliation services (such as Freebase's) to compensate the deficiencies of syntax-level clustering alone."

see the "Download Your Information" section of your profile page

"A friend (from Google) recently trolled me, asking, “What’s up with the data lock-in at Flickr?”. Got me thinking about standards. I wrote back a rant to a mailing list of fellow senior hacker, and coders types. Below I’ve included that rant, largely verbatim. I’d been meaning to turn it into a more reasoned blog post, maybe something suitable for posting on a more official outlet, but life is short, and Rod’s post about Quora reminded me to get on it. As software engineers, as social software engineers, it’s important to have standards. You can debate the how much of what we do can be called engineering, even charitably, but the code we write determines the rules that govern the spaces more and more people spend time in, and while “First, do no harm” might be reaching, a few standards that you should be embarrassed to not meet seem appropriate. One of those is around data access, data ownership, and sharecropping. This is something Flickr takes very seriously."

From the File tab (2010) or the Office Button menu (2007), select Save As. To save a copy of your original document, enter a name in the "File name" field. Note: It's a good idea to use the Document Inspector on a copy of your original document, because it's not always possible to restore the data that the Document Inspector removes. In your original document, in 2010, from the File tab, click Info, then Check for Issues, and then Inspect Document. In 2007, from the Office Button menu, select Prepare, and then click Inspect Document. In the Document Inspector dialog box, check the boxes to choose the types of hidden content that you want to inspect. For more about the individual Inspectors, see Remove hidden data and personal information from Office documents. Click Inspect. Review the results of the inspection in the Document Inspector dialog box. Click Remove All next to the inspection results for the types of hidden content that you want to remove from your document.

"Some people interpreted the Geotaggers' World Atlas maps to be maps of tourism. This set is an attempt to figure out if that is really true. Some cities (for example Las Vegas and Venice) do seem to be photographed almost entirely by tourists. Others seem to have many pictures taken in piaces that tourists don't visit. Blue points on the map are pictures taken by locals (people who have taken pictures in this city dated over a range of a month or more). Red points are pictures taken by tourists (people who seem to be a local of a different city and who took pictures in this city for less than a month). Yellow points are pictures where it can't be determined whether or not the photographer was a tourist (because they haven't taken pictures anywhere for over a month). They are probably tourists but might just not post many pictures at all. The maps are ordered by the number of pictures taken by locals."

"Your Portability Policy is a plain-language document that tells your visitors what they can easily bring in and take out. Steve Greenberg, chair of the working group that developed the idea, describes it like this: “In the same way that your Privacy Policy tells visitors what you can do with information they provide, your product’s Portability Policy tells visitors what they can do with it. It should be clear enough that an average user can understand, and short enough that people can actually read it.” The new site, PortabilityPolicy.org, contains a series of questions that will help guide you in creating a Portability Policy as well as several samples."

"Sometimes you just need to explore. With this tool you can play with the Data Export API by building queries to get data from your Google Analytics profiles. You can use these queries in any of the client libraries to build your own tools. If you want to learn more about the how the API works, checkout the interactive examples."

Interesting info avail to dig into about Twitter "A collection of various datasets about the online phenomenon Twitter."

Interesting Match API, offers to match place to location

"With all the new projects this year, it was hard to filter down to the best, but here they are: two honorable mentions and the five best data visualization projects of 2009. Visualizations were chosen based on analysis, aesthetics, and most importantly, how well they told their story (or how well they let you tell yours)."

" after answering the first question, you learn that almost everything on your profile, even if you use privacy settings to limit access, is available to the quiz."

"Service data. Service data is the data you need to give to a social networking site in order to use it. It might include your legal name, your age, and your credit card number. Disclosed data. This is what you post on your own pages: blog entries, photographs, messages, comments, and so on. Entrusted data. This is what you post on other people's pages. It's basically the same stuff as disclosed data, but the difference is that you don't have control over the data -- someone else does. Incidental data. Incidental data is data the other people post about you. Again, it's basically same same stuff as disclosed data, but the difference is that 1) you don't have control over it, and 2) you didn't create it in the first place. Behavioral data. This is data that the site collects about your habits by recording what you do and who you do it with."

"services include a reverse geocoder; deep data about 16 million businesses and tens of thousands of points of interest; a writable layer for developers to annotate the world and do complex geo-queries; and location-enabled media layers (e.g., Twitter and Flickr). We've also recently added an iPhone SDK to speed up mobile development."

"Google has released an API for Google Sites that lets you create or edit pages, upload or download attachments, monitor the activity of a site programmatically. The API could be use to create a new interface for Google Sites, to upload files from other sources or to migrate your data. Google's Data Liberation team built a Java application for importing and exporting Google Sites. The application lets you export the pages from a site and all their attachments to a folder. "The folder structure of an exported site is meant to mimic the Sites UI as closely as possible. Thus if exporting to a directory "rootdirectory," a top-level page normally located at webspace/pagename, would be in a file named index.html, located in rootdirectory/pagename. A subpage of that page, normally located at webspace/pagename/subpage, would be in a file named index.html in rootdirectory/pagename/subpage. Attachments are downloaded to the same directory as the index.html page to which they belong."

persistent storage options for commands

UPDATE:analysis is flawed, see http://www.jgc.org/blog/2009/06/scaccobeber-analysis-of-iranian.html "Not so in the data from Iran: Only 62 percent of the pairs contain non-adjacent digits. This may not sound so different from 70 percent, but the probability that a fair election would produce a difference this large is less than 4.2 percent. And while our first test -- variation in last-digit frequencies -- suggests that Rezai's vote counts are the most irregular, the lack of non-adjacent digits is most striking in the results reported for Ahmadinejad. Each of these two tests provides strong evidence that the numbers released by Iran's Ministry of the Interior were manipulated. But taken together, they leave very little room for reasonable doubt. The probability that a fair election would produce both too few non-adjacent digits and the suspicious deviations in last-digit frequencies described earlier is less than .005. "

GOOG's examples of vcard markup

"Behind the counter of an abandoned McDonalds lie 48,000 lbs of 70mm tape the only copy of extremely high-resolution images of the moon. Forty years ago, unmanned lunar orbiters circled the moon taking extremely high-res photos of the surface to plan landing spots for Apollo 11 onward... In this McDonalds, the only copy of that data is about to be resurrected. These tapes were recorded 40 years ago as part of the Apollo program to map the lunar surface to plan landing spots for Apollo 11 onward. They have never been seen by the public because at the time, they were classified as they reveal the extreme precision of our spy satellites. Instead, all we have ever seen are the grainy photo-of-a-photo images that were released to the public. "

"About a year ago the United Nations announced UNdata, a way to disseminate data stretched out across 22 United Nations databases through one central application. While UNdata houses 66 million records, it's tough to get a sense of what's going on without a visual representation. Progress is an effort to make this world data visible. More than anything though, it was a chance for me to mess around with some data."

aims to be ping for the social web, meaning push, not pull (gnip = ping backwards)

"Three of the circumvention tools — DynaWeb FreeGate, GPass, and FirePhoenix — used most widely to get around China’s Great Firewall are tracking and selling the individual web browsing histories of their users. Data about aggregate usage of users of the tools is published freely. You can see, for example, that the three sites most visited by users of these circumvention tools are live.com, google.com, and secretchina.com. Aggregate data like this is a terrific resource for those of us interested in researching circumvention tool usage, and not much of a privacy risk for the circumventing users if it is only stored (as well as displayed) in the aggregate."

nice visualization of US oil imports and dollars spent over time

very pretty visualization of 2008 edits to the OpenStreetMap

nice example of how tools like DabbleDB are a glimpse of the future of analyzing data on the web.

"If the next Presidential administration really wants to embrace the potential of Internet-enabled government transparency, it should follow a counter-intuitive but ultimately compelling strategy: reduce the federal role in presenting important government information to citizens. Today, government bodies consider their own websites to be a higher priority than technical infrastructures that open up their data for others to use. We argue that this understanding is a mistake. It would be preferable for government to understand providing reusable data, rather than providing websites, as the core of its online publishing responsibility. "

“Making Visible the Invisible” is a commission for the Seattle Central Library, situated in the Mixing Chamber, a large open 19,500 sq ft space dedicated to information retrieval and public accessible computer research. The installation consists of 6 large LCD screens located on a glass wall horizontally behind the librarians’ main information desk. The screens feature real-time calculated animation visualizations generated by custom designed statistical and algorithmic software using data received each hour. This data consists of a list of checked-out items organized in chronological order. The item may be a book, a DVD, a CD, a VHS tape, etc. and from the list we can collect and aggregate titles, checkout time, catalog descriptors such as keywords, Dewey classification code if they are non-fiction items. There are approximately 22000 items circulating per day. Items with Dewey Decimal System labels provide for a way to get a perspective on what subject matters are of current interes

a program that allows you to view and manipulate data on many CDMA phones from LG, Samsung, Sanyo and other manufacturers. This includes the PhoneBook, Calendar, WallPapers, RingTones (functionality varies by phone) and the Filesystem for most Qualcomm CDMA chipset based phones.

supposedly fairly user friendly. oh, and free.

Imagine if kids had peppered the town with [NFC ]tags. They could then leave each other messages in different areas, messages that only those that were 'in', owning the right tag readers, could see. This would be like hiding secrets in plain sight.

some web-based tools for text-wrangling: Filter Lines; Sort Lines; Reverse Lines; Shuffle Lines; Delete Duplicate Lines; Add Lines Numbers; etc.

from selectron tubes to C64 cassettes

the new monitoring, known as "deep-packet inspection," enables a far wider view -- every Web page visited, every e-mail sent and every search entered

* Log into your network as the Network Creator or as an Administrator * Click on the “Manage” tab * Click on the “Members” link * Scroll to the bottom of the member list and click on the export link at the bottom right corner of the list

Interview with founder of EveryBlock

recovering data from corrupted media

The point of the research was to demonstrate how little information is required to de-anonymize information in the Netflix dataset.

Enter your zip code to see US Census data and comparisons with neighboring zips.

This basic right warrants in this respect the capacity of the individual to determine in principle the disclosure and use of his/her personal data. Limitations to this informational self-determination are allowed only in case of overriding public interest

Someday some of the most important information on earth will be information that is NOT connected to the global internet.

a Firefox extension that turns your browser into a mashup platform, by allowing you to extract data from different web sites and mix them together.

Your images, writing, tags, profile, blog entries, comments, testimonials, video, and music should be yours to download and move anyplace you want.

an open source data integration application to help join your desktop data with the web. Apatar effectively integrates data and applications, and provides visual job designer and mapping, joins, filtering, data cleansing and validation capabilities.

Swivel lets you explore data and share your insights with others. Swivel has data about politics, economics, weather, sports, business and more.

a new feature of Google Image Search that allows you to label random images to help improve the quality of Google's image search results.

Clever use of human computation to solve ai problems

If you do not want to require a password to open or change a document but you do want to suggest that others open it only to read and not modify it, you can select the Read-only recommended check box on the Security tab. Tools -> Options -> Security

displays some examples of the Best and Worst of Statistical Graphics, with the view that the contrast may be useful, inform current practice, and provide some pointers to both historical and current work

Many Eyes is a bet on the power of human visual intelligence to find patterns. Our goal is to "democratize" visualization and to enable a new social kind of data analysis.

Google TechTalks