隨著Flash Drives快速發展，Flash記憶體被廣泛的運用再當今儲存環境上且效果卓越，然而受限於其昂貴的cost-per-byte，Flash記憶體尚且無法完全取代Hard Disk Drive (HDD)，許多利用小容量Flash記憶體來加速HDD儲存環境的探討與研究隨之興起，Storage Caching與Storage Tiering便是其中主流的兩種方法且各自發展成熟，在各種系統上皆有許多的應用實現供使用者選擇，然而這兩個方法由於策略上的限制，皆會面臨效能低落的情況，Storage Tiering對於short duration data無法及時反應；Storage Caching則是無法辨識出long-duration data。我們觀察到兩個機制是從不同的角度切入來提升系統儲存環境的I/O效能，且能互相彌補彼此在I/O特性上的缺陷，然而現今能夠提供使用者同時具有Storage Caching與Storage Tiering儲存環境的方法與實現少之又少。本論文中，我們實現了一個供使用者在Linux系統上可建立一個同時擁有Storage Caching機制與Storage Tiering機制的儲存環境，我們稱為Hybrid Storage Caching and Tiering on Linux，在這個環境上，我們明確了Storage Caching與Storage Tiering各自所要優化的數據類型，並透過兩個機制之間的偕同處理使得儲存環境能夠更廣泛的運用在現今複雜多變的應用中並取得一致性的高效能，我們也實作了許多優化機制來減少Hybrid Caching and Tiering中的overhead。我們使用fio benchmark模擬OLTP workload來進行I/O實驗，從實驗的結果可以證實Hybrid Caching and Tiering在各種不同的SSD size配置下皆能獲得比Caching-Only或Tiering-Only更高的效能。

With the rapid development of flash drives, flash memory is widely used in today's storage environment and the effect of excellence. However, limited by high cost-per-byte of flash drives can’t replace Hard Disk Drive (HDD) completely. Use of small capacity flash drive to accelerate the HDD storage environment become more popular. Flash-based Storage Caching and Storage Tiering are fully developed and user can choose many implementation at different system. However, two solutions would face low performance respectively when running specific I/O type. Storage Tiering can’t response in real time when short-duration I/O happened. Storage Caching can’t identify critical data for long-duration optimization and suffer the cache pollution problem easy. We observed that Storage Caching and Tiering are improve I/O performance from different granularity levels and frequency, and they are complementary. But rarely chosen for user to build a storage environment that combine Storage caching and Storage tiering the same time. In this thesis, we proposed and implement a package that combine Storage Caching and Storage Tiering based on Linux system, we called Hybrid Storage Caching and Tiering on Linux (Hybrid Caching and Tiering). In this storage environment, we assigned the data that Storage Caching and Tiering focus respectively. Through cooperation mechanism of Storage Caching and Storage Tiering can more widely be used on various kinds of application and provide consistent high performance. We also implement many optimization methods to reduce Hybrid Caching and Tiering overhead. We evaluated IOPS and SSD access ratio with Caching-Only method, Tiering-Only method and Hybrid Caching and Tiering method. The experiment result shows Hybrid Caching and Tiering methods improves the IOPS more than others.

論文審定書 i
中文摘要 iii
Abstract iv
Contents vi
List of Figures ix
List of Tables xii
Chapter 1. Introduction 1
1.1 Background 1
1.2 Motivation 2
1.2.1 Limitations of Storage Caching 2
1.2.2 Limitations of Storage Tiering 2
1.2.3 Our Proposed Methods 3
Chapter 2. Related Works 4
2.1 Preliminary Knowledge 4
2.1.1 Linux I/O Subsystem Architecture 4
2.1.2 Block Device 8
2.1.3 Bio 8
2.2 Auto-Tiering Storage 10
2.2.1 Hardware-based Tiered 11
2.2.2 Software-based Tiered 12
2.2.3 Other Tiered Research 12
2.3 Storage Caching 13
2.3.1 Data Allocation Policy 15
2.3.2 Write Cache Algorithm 15
2.3.3 Other SSD Caching Research 16
2.4 Different of Storage Caching and StorageTiering 17
2.5 Current of Combine Storage Caching and Storage Tiering 18
Chapter 3. Hybrid Storage Caching and Tiering on Linux 19
3.1 Overview of Hybrid Caching and Tiering on Linux 20
3.1.1 Register Hybrid Caching and Tiering 20
3.1.2 Hybrid Caching and Tiering Architecture 22
3.2 Btier 23
3.2.1 Tiering Mapping Table 24
3.2.2 Tiering Block Allocate 26
3.2.3 Btier I/O Flow 27
3.2.4 Tier Block Migration 28
3.3 Caching on Hybrid Caching and Tiering 30
3.3.1 Cache Block Meta-data 30
3.3.2 Mapping Mechanism in Storage Caching 31
3.3.3 Binary Representation 39
3.3.4 Cache Block Allocate 41
3.3.5 Replacement Policy 42
3.3.6 Read I/O Flow 43
3.3.7 Write I/O Flow 46
3.3.8 Lazy Adaptive Replacement Cache (LARC) 49
3.3.9 Storage Caching Read/Write Share Ratio 52
3.3.10 Pending bio 53
3.4 Hybrid Caching and Tiering on Linux 56
3.4.1 Bio’s Operate on Hybrid Caching and Tiering 56
3.4.2 Cooperation Mechanism of Caching and Tiering 58
3.4.3 Hybrid Caching and Tiering I/O Flow 60
Chapter 4. Optimization Mechanism of Hybrid Caching and Tiering 62
4.1 Spatially Optimized Write Cache Algorithm 62
4.1.1 Spatial Locality of HDD 63
4.1.2 Flush and Destage 63
4.1.3 Sequential Node 64
4.1.4 Spatially Optimized Write Cache Algorithm Operation 66
4.2 Automatic Dirty Cache Block Write Back 69
4.2.1 Auto Write Back Working Flow 69
4.2.2 Timing Relationship 71
Chapter 5. Experiment Result 74
5.1 Experimental Setup 74
5.2 Performance Analysis 76
5.3 Other Experiment 83
5.3.1 Simulate OLTP Data Temporal Locality Transfer 83
5.3.2 Hybrid Caching and Tiering with PCIE-SSD Caching, SATA-SSD Performance Tier and HDD Capacity Tier 87
Chapter 6. Conclusion 89
Chapter 7. Future Work 90
7.1 Synergistic Mechanism of Caching and Tiering 90
7.2 Dynamic Adjustment Size of Caching and Performance Tiering 90
Chapter 8. References 92

[1] EMC Corporation: “Fully Automated Storage Tiering for Virtual Pools (FAST VP)”, (2013)
http://www.emc.com/storage/symmetrix-vmax/fast.htm
[2] Hitachi Data Systems Corporation: “Dynamic Tiering”, (2013) http://www.hds.com/products/storage-software/hitachi-dynamic-tiering.html
[3] G. Schmidt, B. Dufrasne, J. Jamsek, P. Kimmel, H. Matsuno, F. Morais, L. Oxenham, A. Rainero, and D. Senin: “IBM System Storage DS8000 Easy Tier”, (2012)
http://www.redbooks.ibm.com/redpapers/pdfs/redp4667.pdf
[4] Hewlett-Packard Development Company: “HP 3PAR Adaptive Optimization Software - Overview & Features”, (2013)
http://h18006.www1.hp.com/storage/software/3par/aos/index.html
[5] Dell Corporation: “The Architectural Advantages of Dell SC Series Compellent Automated Tiered Storage”, (2016)
http://www.dell.com/learn/us/en/4402c/shared-content~solutions~en/documents~dell-compellent-tiered-storage.pdf
[6] Intel Corporation: “Smart Response Technology”
https://www.intel.com/content/www/us/en/architecture-and-technology/smart-response-technology.html
[7] btier. [Online]. Available: https://sourceforge.net/projects/tier/?source=navbar
[8] LVMTS. [Online]. Available: https://github.com/tomato42/lvmts
[9] Strunk, John D. "Hybrid Aggregates: Combining SSDs and HDDs in a single storage pool." ACM SIGOPS Operating Systems Review 46.3 (2012): 50-56.
[10] Chen, Feng, David A. Koufaty, and Xiaodong Zhang. "Hystor: making the best use of solid state drives in high performance storage systems." Proceedings of the international conference on Supercomputing. ACM, 2011.
[11] Guerra, Jorge, et al. "Cost Effective Storage using Extent Based Dynamic Tiering." FAST. Vol. 11. 2011.
[12] Tsai, Chai-Hao, Jerry Chou, and Yeh-Ching Chung. "Value-based tiering management on heterogeneous block-level storage system." Cloud Computing Technology and Science (CloudCom), 2012 IEEE 4th International Conference on. IEEE, 2012.
[13] Dot Hill System “Real-Time Storage Tiering for Real-World Workloads”
https://www.dothill.com/wp-content/uploads/2012/08/RealStorWhitePaper8.14.12.pdf
[14] Pritchett, Timothy, and Mithuna Thottethodi. "SieveStore: a highly-selective, ensemble-level disk cache for cost-performance." ACM SIGARCH Computer Architecture News. Vol. 38. No. 3. ACM, 2010.
[15] Huang, Sai, et al. "Improving flash-based disk cache with lazy adaptive replacement." ACM Transactions on Storage (TOS) 12.2 (2016): 8.
[16] Gill, Binny S., and Dharmendra S. Modha. "Wow: wise ordering for writes-combining spatial and temporal locality in non-volatile caches." Proceedings of the 4th conference on USENIX Conference on File and Storage Technologies-Volume 4. USENIX Association, 2005.
[17] Gill, Binny S., et al. "STOW: a spatially and temporally optimized write caching algorithm." Proceedings of the 2009 conference on USENIX Annual technical conference. USENIX Association, 2009.
[18] Fan, Ziqi, et al. "I/O-Cache: A Non-volatile Memory Based Buffer Cache Policy to Improve Storage Performance." Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), 2015 IEEE 23rd International Symposium on. IEEE, 2015.
[19] Matthews, Jeanna, et al. "Intel® Turbo Memory: Nonvolatile disk caches in the storage hierarchy of mainstream computer systems." ACM Transactions on Storage (TOS) 4.2 (2008): 4.
[20] Zhang, Xuechen, Kei Davis, and Song Jiang. "iTransformer: Using SSD to improve disk scheduling for high-performance I/O." Parallel & Distributed Processing Symposium (IPDPS), 2012 IEEE 26th International. IEEE, 2012.
[21] Fan, Ziqi, et al. "I/O-Cache: A Non-volatile Memory Based Buffer Cache Policy to Improve Storage Performance." Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), 2015 IEEE 23rd International Symposium on. IEEE, 2015.
[22] M. Srinivasan. Flashcache : A Write Back Block Cache for Linux. https://github.com/facebook/flashcache/blob/master/ doc/flashcache-doc.txt.
[23] dm-cache. [Online]. Available: https://github.com/mingzhao/dm-cache
[24] Anand Lal Shimpi (May 11, 2011). "Intel Z68 Chipset & Smart Response Technology (SSD Caching) Review". AnandTech. Retrieved November 4, 2011.
[25] Kgil, Taeho, David Roberts, and Trevor Mudge. "Improving NAND flash based disk caches." Computer Architecture, 2008. ISCA'08. 35th International Symposium on. IEEE, 2008.
[26] Oh, Yongseok, et al. "Caching less for better performance: balancing cache size and update cost of flash memory cache in hybrid storage systems." FAST. Vol. 12. 2012.
[27] Hayashi, Shinichi, and Norihisa Komoda. "Evaluation of SSD Tier Method and SSD Cache Method in Tiered Storage System." The papers of Technical Meeting on “Information Systems”, IEE Japan, IS-13-32. 2013.
[28] EMC Corporation: “EMC VNX2 Multicore FAST Cache”, (2016)
https://www.emc.com/collateral/white-papers/h12208-vnx-multicore-fast-cache-wp.pdf
[29] DataCore Software Corporation: “DataCore SANsymphony”
https://www.datacore.com/products/datacore-sansymphony
[30] Yang, Qing, and Jin Ren. "I-CASH: Intelligently coupled array of SSD and HDD." High Performance Computer Architecture (HPCA), 2011 IEEE 17th International Symposium on. IEEE, 2011.
[31] HITACHI T Corporation: “The Benefits of Server-Side Flash with VMware vSphere Flash Read Cache and FlashSoft Software from SanDisk on Hitachi Compute Blade 500”, (2016)
https://www.hds.com/en-us/pdf/white-paper/benefits-of-server-side-flash-with-vmware-vsphere-on-hitachi-compute-blade-500.p
[32] Axboe, Jens. "Fio-flexible i/o tester synthetic benchmark." URL https://github. com/axboe/fio (Accessed: 2015-06-13) (2005).