傳統的資料庫系統僅支援儲存文、數字資料。我們只能依據視訊資料的編號、標題或敘述來存取儲存在傳統資料庫系統中的視訊資料。視訊資料中，最明顯的資訊之一便是會隨著時間而變動位置的物體。每一部影片內的物體之間，均存在著空間及時間上的關係。時間關係能藉由畫面先後順序來確定，而空間關係則可藉由同一個畫面間，物體間的相對關係來確定。在設計一個以內容為主的視訊資料庫系統時，最困難的是如何將移動物體間的時間-空間關係完整地儲存及描述出來。許多以內容為主的視訊搜尋研究，不是忽略了物體間的時間關係，就是僅記錄在單一畫面中，物體間的空間關係。根據觀察，我們認為一個以內容為主的視訊資料庫系統，不僅要有一個好的索引結構、查詢處理機制，還要有一個方便的使用者介面，以滿足視訊資料的需求及特性。在本篇論文中，我們針對視訊資料，設計且實作了一個查詢處理的方法。在提出的方法中，我們考慮了三種查詢類型：物件查詢、空間-時間關係查詢及移動查詢。物件查詢是去搜尋特定物體；空間-時間關係查詢是去搜尋在時間-空間關係上，滿足使用者要求的物體；而移動查詢則是去搜尋以特定移動動作的物體。此外，我們同時也考慮了三個設計項目，分別是索引的建置、查詢的處理及介面的設計。當視訊資料庫中的資料量越來越大時，針對內容來做搜尋的處理時間亦將大量增加。所以我們需要設計一個適當的索引結構，以加速搜尋的時間。我們針對空間及時間上的關係，分別提出了兩種索引結構。針對時間上的索引，我們運用Time Index中的概念，設計了一個新的時間索引結構：shot-based B+-tree；針對空間上的索引，我們利用R-tree，不僅對同一個畫面中的物體做索引，同時也為同一物體的開始及結束之空間關係作索引。透過以上的索引結構，我們能更快、更準確地搜尋滿足特定時間-空間關係物體。在查詢的處理時，我們提出了一個簽章檔案結構，並藉由此簽章結構，將絕對不可能是答案的視訊資料給過濾掉。找到可能是答案的物體後，我們利用一個稱為binary string的表示法，去表示物體間的時間-空間關係。透過binary string去比對物體間的時間-空間關係是否滿足使用者的查詢。最後，我們設計了一個包含上述概念且方便的使用者介面。我們的系統是架構在Pentium III 550 的機器上，主記憶體有256 MB，作業系統為Windows 2000 Professional版，後端資料庫為Access 2000，並利用Delphi 6撰寫了約1萬行的程式碼。根據我們的經驗，我們所提出的方法可以有效地處理使用者的查詢，提供了更快的搜尋能力及更方便的使用者介面。

Traditional database systems only support textual and numerical data. Video data stored in these database systems can only be retrieved through their video identifiers, titles or descriptions. In the video data, frame-by-frame object change is one of the most obvious information. Each video contains temporal and spatial relationships between content objects. The temporal relationships can be specified between frame sequences and the spatial relationships can be specified by the relationships between objects in a single frame. The difficulty in designing a content-based video database system is how to store and describe the relationships between moving objects completely. Many researches on content-based video retrieval represented the content of video as a set of frames, but they either left out the temporal ordering of frames in the shot or only stored the relationships between objects in a single frame. According to these observations, we conclude that a content-based video database system requires video indexing, query processing and a convenient user interface to fit the requirements and characteristics of videos. In this thesis, we design and implement a query processing strategy for video data. In the proposed strategy, we consider three query types: the exact object match, the spatial-temporal object retrieval and the motion query, where a exact object match is to find the video files which contain the specific objects, a spatial-temporal objects retrieval is to retrieve the object pairs that satisfy some spatial-temporal relationships and a motion query is to find the set of frames which contain the object movements. Moreover, we consider three design issues: the video indexing, the video query processing and the video query interface. When there are a large number of videos in a video database and each video contains many shots, frames and objects, the processing time for content retrieval is tremendous. Thus, we need a proper video indexing strategy to speed up the searching time. In order to fulfill the spatial-temporal relationships of objects between different frames, we give the indexes both in the spatial and temporal axes. In the temporal index file structure, we propose the shot-based B+-tree to index the temporal data. In the spatial index file structure, we use R-tree to store not only the relationships between objects in one frame, but also the relationships of one object when the object first and last appears in the shot. Based on this strategy, we can describe the status of a moving object in details. For the part of query processing, we propose a signature file structure to filter out the videos that absolutely can not be the answer. After that, in order to determine whether the answer exists in the candidate videos, we use a multi-dimensional string, called binary string, to represent the spatial-temporal relationships between objects. Then, the video query processing problem will become a binary string matching problem. Finally, we design and implement an user-friendly user interface. Our system is performed on a Pentium III machine with one CPU clock rate of 550 MHz, 256 MB of main memory, running under Windows 2000 Professional edition, used Access 2000 database and coded in Delphi 6 with about 10,000 lines. From our experience, we show that the proposed system can support an efficient query processing, a fast searching capabilities and an user-friendly user interface.

TABLE OF CONTENTS Page
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .i
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
1.Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Temporal Concepts . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Temporal Data Models . . . . .. . . . . . . . . . . . . . . . . . . 2
1.3 Spatial-Temporal Query Types . . . . . . . . . . . . . . . . . . . 3
1.4 The Video Query Language . . . . . . . . . . . . . . . . . . . . . 4
1.5 Indexing Temporal Data . . . . . . . . . . . . . . . . . . . . . . 5
1.5.1 Problems in Indexing Temporal Data . . . . . . . . . . . . . . . 6
1.5.2 Video Indexing and Retrieving . . . . . . . . . . . . . . . . . . 7
1.6 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.7 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . 12
2. A Survey of Temporal and Video Indexing Strategies . . . . .. . . . 15
2.1 Temporal Index . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.1 Time Index . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.2 Time Index + . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.3 ITB + -Trees . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1.4 TGF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Video Index . . . . . . . . . .. . . . . . . . . . . . . . . . . . 24
2.2.1 3D-List . . . . . . . . . . .. . . . . . . . . . . . . . . . . . 24
2.2.2 STCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.3 Binary String Encoding . . . . . . . . . . . . . . . . . . . . . 29
3. The Video Indexing Structure . . . . . . . . . . . . . . . . . . .. 32
3.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Video Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2.1 The Analysis of Frame Information . . . . . . . . . . . . . .. . 34
3.2.2 The Format of Raw Data . . . . . . . . . . . . . . . . . . . . . 37
3.2.3 The Structure of the Index . . . . . . . . . . . . . . . . . . . 40
3.2.3.1 Temporal Index Structure . . . . . . . . . . . . . . . . . . . 40
3.2.3.2 Spatial Index Structure . . . . . . . . . . . . . . . . . . . 42
3.2.4 The Flowchart of Building Indices . . . . . . . . . . . . . . . 47
4. Query Processing . . . . . . . . . . . . . . . . . . .. . . . . . . 51
4.1 The Signature Technique . . . . . . . . . . . . . . . . . . . . . 51
4.1.1 Build the Video Signature File . . . . . . . . . . . . . . . . . 53
4.1.2 An Example of the Signature File . . . . . . . . . . . . . . . . 54
4.2 The Spatial-Temporal Relationship Match Strategy . . . . . . . . . 57
4.3 The Flowchart of the Query Processing . . . . . . .. . . . . . . . 63
5. The System Query Interface . . . . . . . . . . . . . . . . . . . . 66
5.1 Video Query Interface . . . . . . . . .. . . . . . . . . . . . . . 66
5.1.1 Building Indexes . . . . . . . . . . . . . . . . . . . . . . . . 66
5.1.2 Video Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 72
6.Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 79
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

[1] James F. Allen, ``Maintaining Knowledge about Temporal Intervals," Communications of the ACM (CACM), Vol. 26, No. 11, pp. 832--843, Nov. 1983.
[2] Farshid Arman, Remi Depommier, Arding Hsu and Ming-Yee Chiu, ``Content-
Based Browsing of Video Sequences," Proc. of the 2nd ACM Int'l Conf. on Multimedia, pp. 97--103, 1994.
[3] Norbert Beckmann, Hans-Peter Kriegel, Ralf Schneider and Bernhard Seeger,
``The R-tree: An Efficient and Robust Access Method for Points and Rectangles," SIGMOD RECORD, Vol. 19, No. 2, pp. 322--331, June 1990.
[4] Alberto Del Bimbo, Enrico Vicario and Daniele Zingoni, ``Sequence Retrieval
by Contents through Spatio Temporal Indexing," IEEE Simposium on Visual
Languages, pp. 88--92, 1993.
[5] Rasa Bliujute, Christian S. Jensen, Simonas Saltenis, and Giedrius Slivinskas, ``R-Tree Based Indexing of Now-Relative Bitemporal Data," Proc. of the 24th Int'l Conf. on VLDB, pp. 345--356, 1998.
[6] Chueh-Wei Chang and Suh-Yin Lee, ``Video Content Representation, Indexing,
and Matching in Video Information Systems," Journal of Visual Communication
and Image Representation, Vol. 8, No. 2, pp. 107--120, June 1997.
[7] Ramez Elmasri, Gene T. J. Wuu and Yeong-Joon Kim, ``The Time Index: An
Access Structure for Temporal Data," Proc. of the 16th Int'l Conf. on VLDB,
pp. 1--12, 1990.
[8] Myron Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ashley, Qian
Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic,
David Steele and Peter Yanker, ``Query by Image and Video Concept: The QBIC
System," IEEE Computer, pp. 23--32, Sept. 1995.
[9] Cheng Hian Goh, Hongjun Lu, Beng-Chin Ooi, and Kian-Lee Tan, ``Indexing
Temporal Data using Existing B+-Trees," Data and Knowledge Engineering,
Vol. 18, No. 2, pp. 147--165, Mar. 1996.
[10] Antonin Guttman, ``R-Trees: A Dynamic Index Structure for Spatial Searching," Proc. of ACM SIGMOD Int'l Conf. on Management of Data, pp. 47--57, 1984.
[11] Mohand-Said Hacid, Cyril Decleir, and Jacques Kouloumdjian, ``A Database
Approach for Modeling and Querying Video Data," IEEE Trans. on Knowledge
and Data Engineering, Vol. 12, No. 5, pp. 729--750, Oct. 2000.
[12] Arun Hampapur, Terry E. Weymouth and Ramesh Jain, ``Digital Video Segmentation," Proc. of the 2nd ACM Int'l Conf. on Multimedia, pp. 357--364, 1994.
[13] Christian S. Jensen and Richard T. Snodgrass, ``Temporal Data Management,"
IEEE Trans. on Knowledge and Data Engineering, Vol. 11, No. 1, pp. 36--44, Jan. 1999.
[14] Christian S. Jensen, James Cli ord, Ramez Elmasri, Shashi K. Gadia, Pat Hayes, Sushil Jajodia, Curtis Dyreson, Fabio Grandi, Wolfgang Kafer, Nick Kline, Nikos Lorentzos, Yannis Mitsopoulos, Angelo Montanari, Daniel Nonen, Elisa Peressi, Barbara Pernici, John F. Roddick, Nandlal L. Sarda, Maria Rita Scalas, Arie Segev, Richard T. Snodgrass, Mike D. Soo, Abdullah Tansel, Paolo Tiberio and Gio Wiederhold, ``A Consensus Glossary of Temporal Database Concepts, " SIG-MOD RECORD, Vol. 23, No. 1, pp. 52--64, March 1994.
[15] George Kollios, Vassilis J. Tsotras, Dimitrios Gunopulos, Alex Delis and Marios Hadjieleftheriou, ``Indexing Animated Objects Using Spatiotemporal Access Methods," IEEE Trans. on Knowledge and Data Engineering, Vol. 13, No. 5, pp. 758--777, Oct. 2001.
[16] Vram Kouramajian, Ibrahim Kamel, Ramez Elmasri and Syed Waheed, ``The
Time Index + : an Incremental Access Structure for Temporal Databases," Proc.
of the 3rd int'l Conf. on Information and Knowledge Management, pp. 296--303,
1994.
[17] Tony C. T. Kuo and Arbee L. P. Chen, ``Content-Based Query Processing for
Video Databases," IEEE Trans. on Multimedia, Vol. 2, No. 1, pp. 1--13, Mar.
2000.
[18] Chiang Lee and Te-Ming Tseng, ``Temporal Grid File: A File Structure for
Interval Data," Data and Knowledge Engineering, Vol. 26, No. 1, pp. 71--97,
May 1998.
[19] Chih-Chin Liu and Arbee L. P. Chen, ``3D-List: A Data Structure for Efficient Video Query Processing," IEEE Trans. on Knowledge and Data Engineering, Vol. 14, No. 1, pp. 106--122, Feb. 2002.
[20] Thomas D. C. Little and Arif Ghafoor, ``Interval-Based Conceptual Models for Time-Dependent Multimedia Data," IEEE Trans. on Knowledge and Data Engineering, Vol. 5, No. 4, pp. 551--563, Aug. 1993.
[21] S. Y. Lee, M. C. Yang and J. W. Chen, ``Signature File as a Spatial Filter for Iconic Image Database,"Journal of Visual Languages and Computing, Vol. 3, pp. 373--397, 1992.
[22] Dimitris Papadias, Nikos Mamoulis and Vasilis Delis, ``Approximate Spatio-
Temporal Retrieval," ACM Trans. on Information Systems, Vol. 19, No. 1, pp. 53--96, Jan. 2001.
[23] C. S. Roberts, ``Partial-match Retrieval via the Method of Superimposed Codes," Proc. of the IEEE, Vol. 67, No. 12, pp. 1624--1641, Dec. 1979.
[24] Timos Sellis, Nick Roussopoulos and Christos Faloutsos, ``The R + -Tree: A Dynamic Index for Multi-Dimensional Objects," Proc. of the 13th int'l Conf. on
VLDB, pp. 507--518, 1987.
[25] Kim Shearer, Horst Bunke and Svetha Venkatesh, ``Video Indexing and Similarity Retrieval by Largest Common Subgraph Detection Using Decision Trees," Pattern Recognition, Vol. 34, No. 5, pp. 1075--1091, May 2001.
[26] Stephen W. Smoliar and HongJiang Zhang, ``Content-Based Video Indexing and
Retrieval," IEEE Multimedia, Vol. 1, No. 2, pp. 62--72, Summer 1994.
[27] Theodoros Tzouramanis, Michael Vassilakopoulos and Yannis Manolopoulos,
``Overlapping Linear Quadtrees and Spatio-Temporal Query Processing," The
Computer Journal, Vol. 43, No. 4, pp. 325--343, May 2000.
[28] Michael Vazirgiannis, Yannis Theodoridis and Timos Sells, ``Spatio-Temporal Composition and Indexing for Large Multimedia Applications," Multimedia Systems, Vol. 6, No. 4, pp. 284--298, July 1998.
[29] Delis Vasilis and Hadzilacos Thanasis, ``Binary String Relations: A Foundation for Spatiotemporal Knowledge Representation," Proc. of the 8th int'l Conf. on Information and Knowledge Management, pp. 27--37, 1999.