Wednesday, May 9, 2012

Prerequisites

As stated earlier, I have invented a new system (Didget Management) that I think can eventually replace conventional file systems. Before going into any details about the various features of my new system, I thought I would first discuss the requirements of any data management system that hopes to have any chance of unseating the reigning champion - the traditional file system.

Of course, any departure from over 50 years of computing tradition will be met with a certain amount of pain. No matter how many processes are put in place to ease the migration of data from one system to another, existing users and programs must adapt to a whole new way of managing data. Some features of conventional systems can be emulated to provide some level of backward compatibility, but nevertheless there will be a learning curve and the new system will break some old ways of doing things.

Obviously, the new system must offer some very compelling features in order to make the pain worth it. The new system must not only solve a number of existing problems, but it must also open up lots of new opportunities. As the Internet has proven, if you provide enough value, widespread adoption is possible in spite of many hurdles. The Internet went from a curiosity to an integral part of the computing landscape in a relatively short time once users and developers realized the power of these inter-connected servers and the opportunities they opened up.

If it aint broke don't fix it...

Although in a previous post I enumerated quite a few problems with conventional file systems, they also have a number of features that I think work very well. My new system must be able to provide these features with equivalent speeds and ease of use.

1) Block Based Storage.

File systems rely heavily on the block storage nature of the physical storage devices they control. Hard drives, flash drives, and optical disks are all block based storage mediums. Like file systems, the Didget Manager makes heavy use of a block based architecture.

2) Variable-Length Data Streams.

Each file in a file system has a data stream that consists of a set of bytes arranged serially that represents information stored in a digital format. The data stream may contain structured or unstructured data. Numerous formats have been invented over the years that programs rely heavily upon to work properly. Just like a file, a Didget has a variable length byte stream that can contain any kind of data. Any existing file can be converted to a Didget without modifying its data stream.

3) Robust, Yet Simple API.

Applications must be able to create, delete, access, modify, and perform queries against any number of data elements (e.g. files). Like file systems, the Didget Management System will release a robust set of APIs that make it very easy for applications to create new Didgets and query or otherwise manipulate existing ones.

4) Support for Massive Numbers.

Modern file systems like NTFS can handle billions of files within a single volume if the underlying storage is sufficiently large enough to hold them. Like volumes do with files, each Chamber can handle billions of individual Didgets.

5) Fast Access.

File systems have been finely tuned over the years to provide quick response to commands from applications to create, open, read, write, and close files. The Didget Manager is able to perform similar operations with just as much speed as conventional file systems. For batch operations where thousands of new Didgets are created at once, we can even do it faster.

No comments:

Post a Comment