Saturday, July 13, 2013

Data Managers

Over the years, a number of systems have been created to help users manage their data. I call these systems "Data Managers". There are two types - primary data managers and secondary data managers.

Primary data managers are very general-purpose in nature and are widely adopted in the computing world. File systems, databases, and web servers fall into this category. More recent members of this category include distributed file systems like Hadoop and cloud offerings like Amazon S3. These newer systems are gaining greater acceptance as "Big Data" becomes more pervasive and as users demand more mobile access to all their data.

Secondary data managers are generally more specialized in the types of data they manage. They almost always utilize the services of a primary data manager to store their underlying data. Examples of these kinds of data managers include Apple's iTunes for managing music or Google's Picassa for managing photos. They typically keep most of their unstructured data as files in a file system and create a proprietary database for storing extra metadata. These data managers may also integrate with cloud services to give the user a virtual view of their data even when it may be spread across several systems. Unfortunately, these secondary data managers are nearly always in danger of interference from other programs and must rely upon the security measures offered by the primary data manager. If another application deletes, moves, or renames one or more of the files it manages, the secondary data manager can often have trouble reconciling those changes. If another program deletes one or more of its core metadata files (i.e. its database) then the secondary data manager can fail completely.

The Didget Management System is a primary data manager. It not only provides new functionality that previous data managers lack, but it has also been designed to supplant them. This is very different from other primary data managers like databases for example, which were designed to manage structured data in ways that file systems never could, but were never designed to handle unstructured data well enough to make file systems unnecessary. A consequence of that strategy is that as each of the other primary data managers entered the market, we ended up with yet another "silo" into which a portion of our data can be put.

That is why I designed the Didget Management System to manage both structured and unstructured data well. It is designed to manage that data in both simple configurations and in distributed cluster environments. When the amount of data grows from a few thousand pieces of information to billions of pieces utilizing petabytes of storage, there will not be a costly transition point where all the existing data must be migrated to an entirely new system. If we are successful, new data will not only be created as Didgets instead of as files or traditional database tables, but all the old data will be converted to Didgets as well. Our goal is to replace those other primary data managers completely.

In order to realize that goal, the Didget Management System has to do all the critical data management functions of the system it is replacing in addition to its new feature set. It cannot just be 5%, 10%, or even 50% better either. It has to be at least TWICE as good as the old system. When I designed it, that was my minimum threshold. If I couldn't make it dramatically better, it would not gain widespread adoption and would likely fall into a very narrow niche product and not be worth the effort.

Fortunately, the design has proven to work so well that I not only think we have met that 2x threshold, I think it has greatly exceeded it. I would not be surprised if once all the features are fully implemented, that we will have a system that is 10x better than those other systems. That does not mean that we will do everything 10x better than every feature found in those other systems (for example we will not be able to read a Didget ten times faster from disk than a file system can read a file), but rather that overall it will be that much better when all the factors of performance, feature set, ease of use, security, and flexibility are considered.

Tuesday, April 30, 2013

Another Piece to the Puzzle

Didgets provide new and innovative ways to store, search and organize unstructured data that would normally be stored in files. They have also proven useful for storing structured data that is well suited for entry into a NoSQL database. A missing piece was to use them to store structured data that has been traditionally stored within tables in a regular Relational DataBase Management System (RDBMS) and accessed via a Structured Query Language (SQL).

Since our tags had been effective in implementing NoSQL columns in a sparse table, we decided to use them to try and implement a regular relational table. While I had little hope that it could match the performance of a finely tuned RDBMS like MySQL, I at least wanted to implement something that would be acceptable and maybe provide a few unique features or an easier way of managing the data.

To my surprise, it has not only matched the performance of MySQL in preliminary tests, it was 17% faster on many queries. I created a table with six columns and inserted 1 million rows of random data to test the performance of each system. Using the MyISAM storage engine under MySQL, a "Select *" query took 30 seconds to execute on my old test machine. The same query using the Didget Management System only took 25 seconds to complete.

If I switched out the storage engine under MySQL to use the InnoDB engine, the same query took 1 minute and 20 seconds. I was surprised that the InnoDB engine with transactional support was so much slower than the MyISAM engine for this simple query. I have yet to implement the transaction feature using Didgets so I could not do a comparison test but I am confident that our transaction overhead will not be as dramatic as it was under MySQL.

I am also confident that the Didget Management System will provide a very easy mechanism to create, query, and share database tables. It will also be much easier to administer since we can provide lightning fast queries without having to index columns or do complicated joins across multiple tables.

In essence, the Didget Management System is a radically different architecture to the traditional RDBMS way of storing structured data in multiple tables. Since development of the database features are still in its infancy, there is much work yet to be done but I am confident that this will become a major feature of our new general-purpose data management system.

Stay tuned for further developments....

Sunday, February 24, 2013

Connecting the Dots

If you look up at the sky on a moonless night, far away from any city lights, you will see many thousands of individual stars. An asterism is a group of those stars that can be connected together in our minds to form a stick figure. Constellations are ancient asterisms that gained popular names like Virgo or Ursa Major. Other asterisms that just make up a portion of a constellation have also been given popular names like "The Big Dipper" or "Orion's Belt". People who star gaze and either find some of these popular asterisms or form their own, are looking for "patterns" among the thousands of stars.

Searching for patterns is also common when we deal with all that data that exists as individual files or database records on our hard drives, flash memory cards, or DVDs. Sometimes these patterns are already established for us. A popular software package may consist of a dozen separate executable files along with their configuration files and documentation. They are often copied into one or two folders or directories during an installation process to keep them together. Sometimes installation programs copy them into common folders like /usr/bin so that they get all mixed in with other programs and they are not so easy to sort out and figure which files belong to which programs.

But even files that seem to be completely independent of other kinds of data (e.g. a photo or a song) can often be grouped together with other files to form ad hoc groups (e.g. a photo or a music album). We are constantly trying to make connections between different data points to form new and interesting patterns. Facebook and other social media sites provide mechanisms to form some of these patterns. A user posts messages, pictures, documents, videos, and other personal information in order to tell a story about their life, their interests, and their friends. It is the connections between lots of individual pieces of data that can lead to new interactions and help us make decisions.

The current trend in "Big Data" and various forms of analytics is all about finding patterns in large amounts of data to drive business decisions. Analyze a million customer orders to look for patterns of shopping behaviour when it is cold outside in order to figure out what items to put on sale when the next big storm hits. Analyze emails sent by everyone over 65 years old in Florida to figure out what political messages will most likely sway the most voters.

The trick to establishing meaningful patterns among millions or billions of individual data points lies in the ability to quickly analyze each point and determine if it has a significant connection to another point. The system that is used to store the information is a critical component to being able to quickly check lots of data points for a certain condition in order to sift the wheat from the chaff. The system must not only be able to match things like strings or numbers, but it must provide some kind of context in order to make more meaningful connections.

For example, if someone wanted to analyze a group of messages to gain intelligence about military hardware, the word "Tank" would be a meaningful keyword to search for. However, such a "brute force" search might turn up every message that deals with water tanks, gas tanks, and R&B music. It is much more meaningful if the search was conducted using "Vehicle=Tank" instead.

The Didget Management System was designed to not only manage large numbers of data points, but to also aid in making connections between points in order to find new patterns. By attaching many searchable tags to any given piece of data and by providing context for every single tag, the system makes it easy to find all the data that share a common attribute. It can also rank various connections between any two points based on the number of attributes they share in order to give hints about more relevant connections.

Big Data Analytics is all about finding hidden patterns and unknown correlations in large amounts of data. This means that specialized queries must be conducted against all that data to try and find meaningful patterns. When the data is created and stored, the nature of such queries is largely unknown. In other words, the data must be stored in such a way as to make as wide as possible, a variety of potential queries.

The speed at which a query can execute is a major factor in finding that "needle in a haystack". If a big data set consists of 10 billion data points and every query takes several hours to complete, then it becomes very hard to conduct lots of different types of queries, looking for a pattern. If, on the other hand, such a query can execute in a minute or less, then it becomes practical to conduct a wide variety of queries hoping that a meaningful pattern just "pops out in front of you".

Several other "big data" projects like Hadoop, MapReduce, HBase, Cassandra, and MongoDB have been structured to be spread across a cluster of nodes so that the processing of data can occur in parallel. This can greatly reduce the time necessary to perform a query. Such systems can be very complex to set up and administer, however. Our system has been designed to greatly simplify such configurations.

But finding patterns should not just be exclusive to large companies with big data sets. Individual users could greatly benefit from finding meaningful patterns among a few million pieces of information. If I got a message from Mary about her vacation in Hawaii, it would be helpful if there was an "about" button next to her name that when pushed would bring up a list of every message, photo, and document that she had sent me or was about her. Likewise it would be helpful if the message itself had hyper-links in it that when clicked would bring up my own photos of Hawaii or information about scuba diving or whale watching. These links could be generated automatically by the system based on tags already present on other Didgets.

Saturday, January 19, 2013

Silos of Information

As I stated earlier, the Didget Management System was designed to offer an alternative to conventional data management systems that tend to manage just a subset of all data and to build walls around any extra metadata they may generate. With such systems, a given set of a few million pieces of data (files and/or database rows) will often be fragmented into several of these "silos of information".

To illustrate this using a real world example, consider the following:

A user has a 2TB hard drive nearly full of data. Since the average size of each piece of data on the drive is about 1 million bytes, this represents nearly 2 million different files. Out of all that data, there exists three important pieces of information that are from the user's friend Bob. Bob has sent the user an email; the user has taken a picture of Bob and transferred that picture from his camera to the hard drive; and Bob has also authored a document that the user downloaded from Bob's web site.

The email was transferred from the email server to the user's computer running Windows 7 by Microsoft Outlook and stored in a .pst file somewhere in the file system hierarchy. The picture was imported into Google's Picasa photo manager and was "tagged" with "Bob" using their facial recognition feature. This tag was embedded within the .jpg file using the EXIF space reserved for such tags. The document was also stored somewhere on the file system and the user set an extended attribute of "Author=Bob" on the document file using a special document manager program.

Now the user wants to do a general search for everything on his drive that has to do with his friend Bob and hopes to come up with all three pieces of information. He wants a program that will comb through all his data and find those pieces.

1) The program will prompt the user for what to base the search on. The user just types "Bob" since there is no standard schema that helps identify "Bob" as a person's first name.

2) The program must now be able to do a complete folder hierarchy traversal, looking for any instances of the string "Bob". It might find a bunch of files that contain "Bob" in their file name, like "bobset.exe" or "stuff.bob". It would need to show those to the user since it doesn't know if they might be relevant.

3) For every file the program searches, it would need to peek at its extended attributes to see if any contained the word "Bob". Like file names that matched, it would need to display a file called "Photo1.jpg" that had an extended attribute "Activity=Bobbing for apples". For every .jpg file, it would need to know how to open and search the EXIF data portions, also looking for any tags that might have "Bob" in them.

4) It would need to be able to parse through any .pst files by following the Microsoft specification, looking for any emails that might come from, or be about Bob.

Each of these things represents a different "silo" of information that would need to be accessed and understood by the program doing the search. The .pst database file; the file extended attributes; and the .jpg - EXIF file format information are examples of these silos. There are many other silos like .db files, html or xml files, registry files, INI files, and .doc files. Accessing each of them requires knowledge about their format and the rules for parsing them.

If instead of using those systems to store data; Microsoft's Outlook, Google's Picasa, and the document manager were all built on top of the Didget Management System then things could be much simpler. The email could be stored in a "Message Didget", the picture could be stored in a "Photo Didget", and the document would be stored in a "Document Didget". Each of these Didgets would have a tag ".person.FirstName = Bob" attached to it.

Now any application could look for stuff about Bob without missing anything or getting all kinds of unintentional results. It would also find all three items in less than 1 second instead of the painfully slow search in the previous example.

Sunday, January 6, 2013

A Non-Hierarchical Data Management System

Those of you who have been following this blog may be wondering why I have never called the Didget Management System an "Object Store". This has been intentional, since I believe that Didgets offer many features that other kinds of persistent objects simply do not, and I wanted to avoid confusion. Once someone hears the word "Object" they tend to get all kinds of notions in their head about what a Didget is and what it should do.

But the reality is that a Didget has more in common with an object than with a traditional file. One of the things that really sets our system apart from other kinds of object stores like Amazon S3, is that we are designing it to be a replacement for local file systems as well as a cloud storage system. If we are successful, in another ten years all the data stored on your laptop, desktop, mobile device, and in various cloud storage containers will be in the form of Didgets.

Although it will be very easy to overlay our system with a traditional hierarchical namespace to provide backward compatibility with legacy systems, our native storage design is anything but hierarchical. This is somewhat like having a file system volume with just a single folder (i.e. root) where every file in the volume is stored. Since files use simple names as their unique identifier, such a system is impractical (if not impossible) for a traditional file system with millions of files stored on it. With Didgets it is not only possible, but very practical to store a hundred million of them within a single container without a hierarchical naming model.

Every Didget within a container has its own unique 64-bit ID that is used to access it. For systems that interact with data without needing its identifier to be in human readable form, it is easier and faster to store IDs as numbers rather than names like "C:\Windows\System\Drivers\adpahci.sys". With our system, is also very easy to find groups of Didgets that match a given criteria or to narrow down a simple search to find the single Didget you are looking for.

But we have hierarchical namespaces for a reason, so it is worth reviewing how we got here.

With file systems, a file's identifier is its name. You create, open, move, copy, and delete a file by passing its name into a file API function. Since its name is its identifier, you can't have two different files with the exact same name. This means you have to come up with a unique name for every file - a task that gets increasingly harder as the number of files increases. Early file systems like FAT that were not case sensitive and restricted names to 8.3 format made this task even harder. Even with long file name support that allows very specific and descriptive names, devices like cameras tend to want to create your pictures with names like "Photo_001.jpg", "Photo_002.jpg", "Photo_003.jpg", etc..

To get around naming conflicts and to add a simple categorization facility, file system designers came up with a hierarchical directory (or folder) model. A file's name only needed to be unique within a given folder, and its full path name became its unique identifier. The file name and folder name could be easily human readable and provide clues for navigation that were intuitive for many users. The folder system also made it easy to copy, move, or delete whole folders or entire folder trees using simple commands.

But file names and folder hierarchies have a number of problems associated with them. Changing the name of any file or any folder in its path will change its unique identifier and thus invalidate any stored references to it. The human readable names cannot be translated from one language to another without causing the same problem. An unprotected file might have its contents overwritten by a completely unrelated file that just happens to have the same name. If I want to store photos I have downloaded, I might have both a "/home/photos/download" and a "/home/download/photos" folder and have files in both - causing confusion.

Didgets operate in a completely different manner. Each Didget can have a name (or multiple names) attached to it as a name tag. When a file is converted to a Didget, each folder name may be attached to it as a separate folder tag. Unlike file paths, the ordering of tags don't make any difference. So if we were to overlay a hierarchical namespace to the Didget system, a command like "ls /home/andy/documents/projects/projectX/*" would give the same results as "ls /documents/andy/projectX/project/home/*".

You could leave out folder tags in a search and just get more results. For example: "ls /andy/*.jpg" would return all the JPG photos that were stored in any path that had "andy" as one of its folders. New folder tags could be added or existing tags deleted at any time without having "moved" the Didget or changing its unique identifier in any way. Existing tags can also be modified or translated to another language with the same lack of consequences.

Such a system provides a much more flexible mechanism for categorizing and finding data. As previous posts have shown, we can find all matching Didgets much faster than conventional file systems can find matching files.