Collections of digital documents can nowadays be found everywhere in institutions, universities or companies. Examples are Web sites or intranets. But searching them for information can still be painful. Searches often return either large numbers of matches or no suitable matches at all.Such document collections can vary a lot in size and how much structure they carry. What they have in common is that they typically do have some structure and that they cover a limited range of topics. The second point is significantly different from the Web in general.The type of search system that we propose in this book can suggest ways of refining or relaxing the query to assist a user in the search process. In order to suggest sensible query modifications we would need to know what the documents are about. Explicit knowledge about the document collection encoded in some electronic form is what we need. However, typically such knowledge is not available. So we construct it automatically.

Kruschwitz 'Intelligent Document Retrieval: Exploiting Markup Structure' Table of Contents Foreword V Preface VII List of Figures XIII List of Tables XV 1 Introduction 1 1.1 Introductory Examples 4 1.2 Using Markup to Extract Knowledge 8 1.3 Applying the Extracted Knowledge 15 1.4 Structure of the Book 17 Part I The Model 21 2 Related Work 23 2.1 Information Retrieval 24 2.2 Information Extraction 26 2.3 Clustering 27 2.4 Classification 29 2.5 Web Search Techniques 31 2.6 Ontologies 34 2.7 Layout Analysis 36 2.8 Web Search Studies 36 2.9 Navigating Concept Hierarchies 38 2.10 Dialogue Systems 41 2.11 Usability Issues 42 2.12 Concluding Remarks on Related Work 43 3 Data Analysis and Domain Model Construction 45 3.1 Documents 45 3.2 Concepts 47 3.3 A Domain Model Based on Concepts 51 3.4 Model Structure 53 3.5 Model Construction 54 3.6 Using the Model for Query Modification 58 3.7 Implementational Issues 60 4 Incorporating Additional Knowledge 63 4.1 Internal Knowledge 63 4.2 External Knowledge 67 5 A Dialogue System for Partially Structured Data 69 5.1 Dialogue as Movement in Space 70 5.2 Dialogue Example 71 5.3 Static vs. Dynamic Clusters 73 5.4 Real User Queries 73 5.5 Properties 75 5.5.1 Document Properties 76 5.5.2 System Properties 76 5.5.3 Goal Description 77 5.6 Dialogue 78 5.6.1 High Level Dialogue States 78 5.6.2 Low Level Dialogue States 80 5.6.3 Constructing Potential Choices 85 5.6.4 Dialogue Strategies 89 5.6.5 Customization 89 Part II Practical Applications 91 6 UKSearch - Intelligent Web Search 93 6.1 Indexing Web Pages 94 6.2 The UKSearch System 98 6.2.1 Indexing and Model Construction 100 6.2.2 Dialogue Strategy 102 6.3 Sample Domain 1: Essex University 107 6.3.1 Index Tables 108 6.3.2 Domain Model 109 6.3.3 Concepts it vs. Real User Queries 111 6.4 Sample Domain 2: BBC News 112 6.4.1 Index Tables 115 6.4.2 Domain Model 116 6.4.3 Adjusted Dialogue Strategy 117 6.5 Implementational Issues 117 7 UKSearch - Evaluation and Discussion 121 7.1 Log Analysis 121 7.1.1 System Setup 122 7.1.2 Results 124 7.1.3 Discussion 125 7.2 Investigating Domain Model Relations 125 7.2.1 Task and Setup 125 7.2.2 Results 127 7.2.3 Discussion 128 7.3 Task-Based Evaluation: Essex University 129 7.3.1 Search Tasks 129 7.3.2 Experimental Setup 133 7.3.3 Procedure 134 7.3.4 Results 134 7.3.5 Discussion 140 7.4 Task-Based Evaluation: BBC News 141 7.4.1 Search Tasks 142 7.4.2 Experimental Setup and Procedure 143 7.4.3 Results 143 7.4.4 Discussion 151 8 YPA - Searching Classified Directories 157 8.1 System Overview 158 8.2 Indexing Classified Advertisements 159 8.2.1 Structure of the Backend 160 8.2.2 Domain Model Construction 161 8.3 Dialogue Strategy in the YPA 162 8.3.1 Properties 165 8.3.2 Dialogue Setup 166 8.3.3 Dialogue Function 168 8.3.4 Calculation of Potential Choices 168 8.4 Implementational Issues 171 9 Future Directions and Conclusions 173 9.1 Towards Evolving Domain Models 173 9.2 Dialogue Management 176 9.3 An Outl