在資訊爆炸的時代，儲存系統在日常生活與商業市場中扮演著相當重要的角色。由於愈來愈多的資料以數位的方式儲存於儲存媒介中，儲存系統需要更有效率的機制來管理數位資料與儲存裝置，而不只是單純地持續增加儲存媒介來擴充所需要的儲存空間。儲存虛擬化的概念，將底層的儲存裝置聚集成單一虛擬裝置，隱藏儲存系統底層中所有儲存裝置的組織複雜度，被視為此問題的解決方式。藉由儲存虛擬化之虛擬層，使用者可以根據自身的需求，動態地調整虛擬裝置的大小；也可藉由虛擬化所提供的方法，更有效率地管理數位資料。

在Linux作業系統上，邏輯磁碟管理員LVM2 (Logical Volume Manager 2) 被視為儲存虛擬化的實現方法之一。LVM2 包括了三項組成元素：核心空間的device-mapper、用戶空間的device-mapper函式庫(libdevmapper)，以及用戶空間的LVM2工具集。在此篇論文中，除了介紹近年來各種虛擬化技術之外，我們的目標將會著重於探討核心空間之device-mapper與其核心演算法，藉此了解儲存虛擬化於Linux作業系統上之實現方式；並嘗試最佳化device-mapper之核心演算法；以及測試device-mapper之效能。

In the era of explosive information, storage subsystem is becoming more and more important in our daily life and commercial markets. Because more and more data are recorded in the digital form and stored in the storage device, an intelligent mechanism is required to make the management of the digital data and storage devices more eficiently rather than simply keep increasing more storage equipment into a system. The concept of storage virtualization was introduced to solve this problem, by aggregating all the physical devices into a single virtual storage device and hidding the complexity of underlying block devices. Through this virtual layer, users can dynamically allocate and resize their virtual storage device to satisfy their need, and they can also use the methods provided by the virtual layer to organize data more efficiently.

Linux Logical Volume Manager 2 (LVM2) is an implementation of storage virtualization on the Linux operation system. It includes three components: the kernel-space devicemapper, the user-space device-mapper support library (libdevmapper), and the user-space LVM2 toolset. This thesis will focus on the kernel-space device-mapper, which provides virtualization mechanism for user-space logical volume manager. The organization of this thesis is composed of : (1) Introduce novel technologies in the recent years, (2) Provide an advanced document about the internals of device-mapper, (3) Try optimizing the mapping table algorithm, and (4) Evaluate the performance of device-mapper.

List of Tables ii
List of Figures iii
List of Listings iv
Chapter 1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Organization of this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Chapter 2 RelatedWork 7
2.1 Block Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 Block layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Storage Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 In-Band Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 Out-of-Band Virtualization . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Networked Storage Virtualization . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.1 NASD: Network-Attached Secure Disks . . . . . . . . . . . . . . . . 15
2.3.2 Petal: Distributed Virtual Disks . . . . . . . . . . . . . . . . . . . . 16
2.3.3 V:Drive: A Storage Virtualization System . . . . . . . . . . . . . . . 18
2.4 Storage Subsystem Virtualization . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 Linux Logical Volume Manager (LVM) . . . . . . . . . . . . . . . . 19
2.4.2 Linux Device-Mapper . . . . . . . . . . . . . . . . . . . . . . . . . 21
Chapter 3 Preparation 24
3.1 Linux Block Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.1 Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.2 Block I/O Submission . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Device-Mapper Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Device-Mapper Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter 4 Device-Mapper Internals 32
4.1 I/O Control Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.1.1 Mapped Devices Management . . . . . . . . . . . . . . . . . . . . . 32
4.1.2 Commands of ioctl Interface . . . . . . . . . . . . . . . . . . . . . . 33
4.1.3 Creating a Full Mapped Device . . . . . . . . . . . . . . . . . . . . 33
4.2 Mapped Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2.1 Creating a Mapped Device . . . . . . . . . . . . . . . . . . . . . . . 35
4.2.2 Suspending/Resuming a Mapped Device . . . . . . . . . . . . . . . 36
4.2.2.1 Suspending a Mapped Device . . . . . . . . . . . . . . . . 36
4.2.2.2 Resuming a Mapping Device . . . . . . . . . . . . . . . . 36
4.2.3 Swapping a Mapping Table . . . . . . . . . . . . . . . . . . . . . . . 37
4.2.4 Dispatching a Block I/O . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2.5 Ending a Block I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3 Mapping Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3.1 Mapping Algorithm - B+ Tree . . . . . . . . . . . . . . . . . . . . . 39
4.3.1.1 B+ Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3.1.2 B+ Tree Mapping Table . . . . . . . . . . . . . . . . . . . 40
4.3.2 Mapping Table Optimization . . . . . . . . . . . . . . . . . . . . . . 41
4.3.3 Loading the Table . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.3.4 Searching a Target Device . . . . . . . . . . . . . . . . . . . . . . . 43
4.4 Target Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5 Target Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5.1 Target Type Operations . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5.2 Target Type Management . . . . . . . . . . . . . . . . . . . . . . . . 46
4.5.3 Linear Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.5.3.1 Linear I/O Mapping . . . . . . . . . . . . . . . . . . . . . 48
4.5.4 Striped Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.5.4.1 Striped I/O Mapping . . . . . . . . . . . . . . . . . . . . . 49
4.5.5 Special Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Chapter 5 Implementation and Performance Evaluation 60
5.1 Optimizing the Mapping Table . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.1.1 Applying Binary Search to Mapping Table . . . . . . . . . . . . . . 60
5.1.2 CSS-Tree Based Mapping Table . . . . . . . . . . . . . . . . . . . . 61
5.1.2.1 CSS-Tree Algorithm . . . . . . . . . . . . . . . . . . . . . 62
5.1.2.2 CSS-Tree Based Mapping Table . . . . . . . . . . . . . . 63
5.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.1 Null Block Device . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.2 Null Target Type Module . . . . . . . . . . . . . . . . . . . . . . . . 68
5.3 Experiment Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.3.1 Null Block Device Performance . . . . . . . . . . . . . . . . . . . . 69
5.3.2 The Performance of Original Device-Mapper . . . . . . . . . . . . . 70
5.3.3 The Performance of Optimized Device-Mapper . . . . . . . . . . . . 71
Chapter 6 Conclusion and FutureWork 79
6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Appendix A Difference File : Binary Searching to Mapping Table Algorithm 84
Appendix B Difference File : Cache-Sensitive Search Tree Based Mapping Table 86
Appendix C Null Block Device Implementation 95
Appendix D Null Target Type Module 102

[1] G. A. Gibson and R. V. Meter, “Network Attached Storage Architecture.” Communications
of the ACM, no. 11, Nov 2000.
[2] D. A. Patterson, G. Gibson, and R. H. Katz, “A Case for Redundant Arrays of Inexpensive
Disks (RAID).” in Proceedings of the ACM SIGMOD International Conference on
Management of Data, June 1988, pp. 109-116.
[3] G. A. Gibson, D. F. Nagle, W. Courtright II, N. Lanza, P. Mazaitis, M. Unangst, and
J. Zelenka “NASD Scalable Storage Systems”, in Proceedings of USENIX 1999, Linux
Workshop, Monterey, CA, June 19.
[4] E. K. Lee and C. A. Thekkath, “Petal: Distributed Virtual Disks”, in Proceedings of
the 7th International Conference on Architectural Support for Programming Languages
and Operating Systems, ASPLOS-VII, October 1996, pp. 82-92, ACM.
[5] A. Brinkmann, M. Heidebuer, F. M. auf der Heide, U. Ruckert, K. Salzwedel, and
M.Vodisek, “V:Drive - Costs and Benefits of an Out-of-Band Storage Virtualization
System”, in Proceedings of the 12th NASA Goddard, 21st IEEE Conference on Mass
Storage Systems & Technologies (MSST), April 2004, pp. 153-157,13-16.
[6] D. Comer “The Ubiquitous B-Tree”, in ACM Computer Surveys, June 1979, pp. 121-
137, ACM.
[7] Garcia-Molina, H. and Salem, K. “Main Memory Database Systems: An Overview”, in
IEEE Transactions on Knowledge and Data Engineering 4, December 1992, pp. 509-
512, IEEE.
[8] Smith, A. J. “Cache Memories”, in ACM Comput. Survey, September 1982, pp. 473-530,
ACM.
[9] Graefe, G. and Larson, P. “B-Tree Indexes and CPU Caches”, in Proceedings of the 17th
international Conference on Data Engineering, April 2001, pp. 349-358, ACM.
[10] Rao, J. and Ross, K. A “Cache Conscious Indexing for Decision-Support in Main Memory”,
in Proceedings of the 25th international Conference on Very Large Data Bases,
September 1999, pp. 78-89, ACM.
[11] Rao, J. and Ross, K. A “Making B+-Trees Cache Conscious in Main Memory”, in Proceedings
of the 2000 ACM SIGMOD international Conference on Management of Data,
May 2000, pp. 475-486, ACM.
[12] Tobin J. Lehman and Michael J. Carey “A Study of Index Structures for Main Memory
Database Management Systems”, in Proceedings of the 12th international Conference
on Very Large Data Bases, August, 1986, pp. 294-303, ACM.
[13] Andersson, A,. “A Note on Searching a Binary Searching Tree” Software - Practice and
Experience, Vol 21, No. 10, pp. 1125-1128, October 1991
[14] Daniel P. Bovet and Marco Cesati, “Understanding the Linux Kernel, Third Edition.”
O’Reilly November 2005.
[15] J. Corbet, A. Rubini, and G. Kroah-Hartman, “Linux Device Drivers, Third Edition.”
O’Reilly, February 2005.
[16] Storage Networking Industry Association (SNIA). “Shared Storage Model: A framework
for describing storage architectures.” http://www.snia.org/tech activities/shared
storage model/SNIA-SSM-text-2003-04-13.pdf
[17] R. Peglar. “Storage Virtualization I: What, Why, Where and How.” [Online]. Available:
http://www.snia.org/education/tutorials/2006/spring/virtualization/Virtualization
I What Why Where How.pdf
[18] Mel Gorman. “CodeViz Project” [Online]. Available: http://www.csn.ul.ie/∼mel/
projects/codeviz/
[19] Z. Meggyesi. Fibre Channel Overview. [Online]. Available: http://hsi.web.cern.ch/HSI/
fcs/spec/overview.htm.
[20] Ravi Kiran UVS. “Linux Block Device Architecture” [Online]. Available: http://www.
geocities.com/ravikiran uvs/articles/blkdevarch.html
[21] “LVM2 resource page.” [Online]. Available: http://sources.redhat.com/lvm2
[22] “Device-mapper resource page.” [Online]. Available: http://sourceware.org/dm/
[23] SuSE Inc. “The Logical Volume Manager (LVM).” [Online]. Available: http://www.
novell.com/products/linuxenterpriseserver8/whitepapers/LVM.pdf
[24] IBM Inc. “Enterprise Volume Management System.” [Online]. Available: http://evms.
sourceforge.net/
[25] Corbet. “idr - Integer ID Management.” [Online]. Available: http://lwn.net/Articles/
103209/
[26] “Iometer Project.” [Online]. Available: http://www.iometer.org/