Storage networking has emerged in the last two years as a compelling technology for increasing the availability and scalability of data storage. According to the Storage Network Industry Association, a storage area network is one “whose primary purpose is the transfer of data between computer systems and storage elements, and among storage elements”. With several product classes and variations within each class, storage networking is a dynamic industry that has been confusing for consumers and also those within the industry.
This article attempts to clarify some of the confusion around storage networking technology by dissecting it into three basic components: wiring, filing and storing. This analysis differs from the typical market-space approach of separating SAN (storage area network) and NAS (network attached storage) products, which helped to create the confusion in the first place by trying to draw distinctions between products with a great deal of overlapping technology.
Wiring, simply stated, is the connectivity technology used to connect storage devices and subsystems to host systems. It includes such diverse technologies as network cabling, host adapters, switches/hubs, and also logical connectivity components such as media access protocols, flow control, virtual network technologies and network security. In short, wiring is everything involved in the transportation of data across a storage network.
Filing is the intelligent process that organizes stored data. Typically, filing is done by file systems and database systems that determine where data is placed on storage devices and how it is retrieved. Filing also includes any additional information, or metadata, that is stored with the data, as well as how the stored information is represented to applications and users. Filing is tightly integrated with operating systems as the primary system-storage application. Even though it is often thought of as a key part of a system platform, filing is purely a logical process and has no inherent dependency on hardware. In general, filing is the primary role of NAS technology, although NAS products can certainly incorporate sophisticated storing and wiring technology also.
Storing technology has both physical and logical components. Physical components include devices such as disk drives and enclosures that provide power, cooling and connectivity. Logical components of storing include volume management software such as mirroring and RAID, and any other virtualization techniques that allow raw storage to be combined or partitioned. The logical component of storing also includes storage protocols and commands for communications between storage devices and systems. Storing is distinct from filing in that storing is a device-oriented function and filing is an application-oriented function that makes use of real or virtual storage.
Figure 1 illustrates common implementation options for wiring, filing and storing. The filing function traditionally resides in a computer system, but in NAS environments, it resides within a file server appliance, as shown. The wiring function is everything between the computer system and the disk subsystem, including storage network adapter cards. Network adapters in SANs are referred to as host bus adapters, or HBAs, and in NAS environments they are typically Ethernet network interface cards (NICs). HBAs and NICs control network operations for their respective networks. The storing function is associated mostly with disk devices and subsystems, but it also includes an important piece that runs as higher-level device driver code for storage network adapters. This driver code manages storage I/O commands and storage protocol information that is transported in storage network transmission frames.
Figure 1 Relative implementations of filing, wiring and storing functions in a storage network
A Closer Look at Wiring
Wiring has been the source of confusion and controversy recently as competing Fibre Channel and Gigabit Ethernet companies are vying for position in the market. The core of this discussion is an argument whether or not Gigabit Ethernet is a viable technology for storage networking. Before launching into polemics and politics, an examination of wiring requirements is in order.
Wiring needs to be reliable, flexible and have ample bandwidth for bursty storage transmissions. In general, the reliability of storage networks needs to exceed the reliability of most data networks. Systems applications such as file systems and databases do not tolerate errors when reading or writing data to storage devices. To emphasize this point, storage I/O errors usually result in system crashes, so storage networks must be rock-solid.
The flexibility requirement of storage network wiring is a reaction to the limited capabilities of traditional bus-based storage and its weakness to server failures and bottlenecks. Compared to storage buses, data networks have provided superior flexibility for many years. Flexibility in storage networking allows servers and storage to be upgraded and replaced independently of each other, facilitating systems management. Flexibility also refers to cost. A flexible technology provides multiple cost/benefit scenarios to meet the various needs of the market.
Bandwidth requirements for storage networks have three elements: wire speed, maximum throughput and low-latency. Storage transmissions have relatively short time-out parameters that do not tolerate delays regardless of their origin. Traditional data networking technologies have not had to address these requirements and therefore have not been developed to meet them.
As companies become more familiar with storage networking, they look for ways to connect servers and storage across longer distances for the purposes of data availability and protection. Some of the possibilities for doing this are extending private storage networks over dark fibre, running storage communications over public WDM (wave division multiplexing) networks, sending storage network traffic over ATM or IP networks, or using a single consistent network for both storage and MAN/WAN networks.
Fibre Channel vs. Gigabit Ethernet
Based on products available in the market today, Fibre Channel has a clear advantage over Gigabit Ethernet because the latter does not yet support storage traffic. But pretending for a moment that current Gigabit Ethernet products were used for storage networks, Fibre Channel would compare favorably as a more reliable transport with better bandwidth metrics. On the other hand, there is no question that Gigabit Ethernet would best Fibre Channel in terms of flexibility. In addition, there is no question that connecting storage traffic to MANs and WANs would be much simpler if storage traffic were running on an Ethernet/IP network.
Fibre Channel was the first widely available, high-throughput wiring technology. Built on the previous experiences of other networking and storage I/O technologies, Fibre Channel was anticipated by its inventors to become the next great backbone technology. Built from the bottom up for high reliability with minimal reliance on system CPUs, storage networking was not originally seen as an important market for the technology. In fact, some of the technology’s early advocates resisted the requirement to support storage. Fibre Channel skeptics who favor Gigabit Ethernet sometimes point out that Fibre Channel was a rootless technology looking for a market – and found one in storage networking.
Despite this history, the fact remains that Fibre Channel adopted an industry standard version of serial SCSI called FCP (Fibre Channel Protocol) that has proven to be an excellent protocol for storage traffic. On the strength of this protocol, Fibre Channel has compatibility with nearly all the file systems and database systems used in popular servers today. Implementation difficulties some companies have had with Fibre Channel have very little to do with the FCP protocol. The agreement on a single, working storage protocol may prove to be the biggest advantage Fibre Channel has in its future competition with Gigabit Ethernet.
However, it is important to realize that FCP is a storing function that is transported by Fibre Channel’s underlying wiring technology. There is no question that the same or a similar protocol could be used as the storing protocol on Ethernet-based storage networks. Therefore, the importance of the storage protocol should not be overestimated. The challenge for the Ethernet storage industry is whether or not they can agree on a common storage protocol.
Gigabit Ethernet: the SAN Changeling?
Ethernet was developed before the reliability design goals of storage networks were attainable. As a result, it has not had the characteristics needed for storage networking. Detractors of Gigabit Ethernet for storage networking reflect an assumption that Ethernet’s capabilities were cast in stone and cannot be changed. However, Ethernet historically has been very successful adapting new technologies in response to new requirements. An important and relevant example is Ethernet’s adoption of switched, full-duplex communications. Looking forward, Ethernet’s roadmap continues to include innovation, including a wire speed boost to 10 Gb/s in 2001.
Fibre Channel supporters point to Ethernet’s lack of flow control and the inability to support high network utilization. This is probably more FUD than reality as the Gigabit Ethernet standard incorporated flow control as a necessity of supporting gigabit transmissions. Gigabit Ethernet’s flow control might not be as sophisticated as Fibre Channel’s, but it does effectively limit the number of frames sent over a network and as a result, enables much higher network utilization.
Another weakness of Gigabit Ethernet is the potential enormous drain on CPU cycles needed to process the TCP/IP protocol stack. This is a fairly serious shortcoming, that is being addressed by opportunistic companies developing network processors that accelerate TCP/IP processing in hardware ASICs. It is unclear how soon this technology will come to market and how effective it might be.
It is a matter of speculation and opinion whether or not Ethernet’s continued improvements will include the changes necessary to make the technology more efficient and reliable for storage networking. However, the questions have more to do with execution of development plans more than they do with the raw potential of the technology. Comparing the network transport capabilities of both Fibre Channel and Gigabit Ethernet, it does not appear that Fibre Channel’s advantages in storage networks are great enough to discount Gigabit Ethernet as a serious competitor.
Fibre Channel’s Self-imposed Threat
To paraphrase Walt Kelly, creator of the Pogo cartoon strip, the Fibre Channel industry “has seen the enemy and it is us”. In its relatively short history, the Fibre Channel industry seems to have developed an industrial strength case of schizophrenia where participating companies hail the arrival of compatibility while ensuring their products are technically differentiated on fundamental capabilities like exchanging network routing information between competing company’s switches. This type of schizoid behavior is certainly nothing new, but it does not necessarily signal the beginning of a great industry either.
Another example of competition within the industry has been the evolution of two distinct types of Fibre Channel networks: shared-media loop networks and switched fabric networks. The access methods of these two networks are strikingly different. Various companies have developed proprietary solutions that integrate loops and fabrics, while the standards-based solution is primarily a theoretical poster-child.
As a result, the complexity of the environment has made it very difficult for the industry to add skilled, competent workers to support their customers. The lack of Fibre Channel talent is viewed as a very serious threat to short-term success of the industry.
What is the Incumbent?
One of the more interesting questions in this discussion is which technology is moving into the other’s territory. Fibre Channel supporters sometime paint a picture of Ethernet/IP industrial monopolies attempting to move into their market with promises of vapourware. Ethernet/IP companies point to the long history of Ethernet/IP as the networking technology of choice.
By now, it appears that IP networking has already won the role of WAN carrier for storage networking traffic. Several initiatives, including one joining Brocade and Cisco, have emerged that indicate a clear roadmap for Fibre Channel bridging over IP WANs. While there are competing technologies within the Fibre Channel industry for WAN bridging with ATM, this appears to be uninteresting to outside observers.
The question about which came first is more relevant than it might appear because it indicates whether or not storage networking will evolve as its own market or if it will be subsumed by the broader data-networking market. If storage networking develops its own identity, it might be possible for it to maintain its own specialized Fibre Channel wiring technology. If the storage market becomes identified with data networking, Fibre Channel will have a very difficult time competing with the big Ethernet/IP machine.
Can Fibre Channel Survive and Thrive?
Obsolescence is a way of life and an ever-present threat to technology professionals. We live and work in an environment where many products are obsolete far in advance of their expected useful life. The cost of replacing an infrastructure technology is enormous. Fortunately, there are often ways to smoothly transition from one technology to another and avoid abrupt service interruptions.
The possibility that Gigabit Ethernet will be used for storage networking is very real. While it is not clear that a concerted, unified effort by the Ethernet/IP industry exists, there is little question that such an effort could succeed in toppling Fibre Channel as the wiring technology of choice for storage networks. Fortunately, Gigabit Ethernet and Fibre Channel use some of the same fibre optical cables, which means cabling can be selected that is compatible with both, supporting future technology changes from one to the other.
Nonetheless, CIOs who need to make strategic storage decisions are in a difficult position. Lessons learned from past experiences will likely provide the best guidance about how to proceed. The unfortunate news is that it is likely to become more confusing over the next couple years as more players enter the fray. Proof of interoperability and compatibility will be the responsibility of the purchaser. In a few years, after the dust settles, the situation will no doubt be righted, but for now caution is advised and options should be left open.
Marc Farley has spent the last 10 years specializing in storage networking technology, summarizing his knowledge in “Building Storage Networks”, published by Osborne/McGraw-Hill. His experiences include systems engineering, strategic planning and corporate management. Mr. Farley can be reached at [email protected].
