NOTE: The commands for creating and dropping indexes are not part of standard SQL.
virtual or derived tables because they present the user with what appear to be tables;
Table 2 summarizes the syntax (or structure) of various SQL statements. This sum-
7The full syntax of SQL is described in many voluminous documents of hundreds of pages.
type of constructs can be specified in each of the six clauses. Which of the six
2. Describe conceptually how an SQL retrieval query will be executed by speci-
3. Discuss how NULLs are treated in comparison operators in SQL. How are
4. Discuss how each of the following constructs is used in SQL, and discuss the
a. Nested queries.
b. Joined tables and outer joins.
c. Aggregate functions and grouping.
d. Triggers.
e. Assertions and how they differ from triggers.
f. Views and their updatability.
g. Schema change commands.
query results if each query is applied to the database in Figure A.2.
a. For each department whose average employee salary is more than
b. Suppose that we want the number of male employees in each department
6. Specify the following queries in SQL on the database schema in Figure A.5.
a. Retrieve the names and major departments of all straight-A students
b. Retrieve the names and major departments of all students who do not
7. In SQL, specify the following queries on the database in Figure A.1 using the
a. Retrieve the names of all employees who work in the department that has
b. Retrieve the names of all employees whose supervisor’s supervisor has
c. Retrieve the names of employees who make at least $10,000 more than
8. Specify the following views in SQL on the COMPANY database schema
a. A view that has the department name, manager name, and manager
b. A view that has the employee name, supervisor name, and employee
c. A view that has the project name, controlling department name, number
d. A view that has the project name, controlling department name, number
9. Consider the following view, DEPT_SUMMARY, defined on the COMPANY
a. SELECT *
b. SELECT D, C
c. SELECT D, AVERAGE_S
d. UPDATE DEPT_SUMMARY
e. DELETE FROM DEPT_SUMMARY
There are many books that describe various aspects of SQL. For example, two refer-
( . . . ,
( . . . ,
In this chapter we discuss the two formal languages forthe relational model: the relational algebra and the
Recall that a data model must include a set of operations to manipulate the data-
The relational algebra is very important for several reasons. First, it provides a for-
From Chapter 6 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
RDBMSs in use today do not provide user interfaces for relational algebra queries,
The relational algebra is often considered to be an integral part of the relational data
In Sections 6 and 7 we describe the other main formal language for relational data-
1SQL is based on tuple relational calculus, but also incorporates some of the operations from the rela-
mathematical logic called predicate calculus.2 In tuple relational calculus, variables
For the reader who is interested in a less detailed introduction to formal relational
2In this chapter no familiarity with first-order predicate calculus—which deals with quantified variables
The result is shown in Figure 1(a).
Notice that all the comparison operators in the set {=, <, ≤, >, ≥, ≠} can apply to
In general, the result of a SELECT operation can be determined as follows. The is applied independently to each individual tuple t in R. This
selected. All the selected tuples appear in the result of the SELECT operation. The
■ (NOT cond) is TRUE if cond is FALSE; otherwise, it is FALSE.
The SELECT operator is unary; that is, it is applied to a single relation. Moreover,
Hence, a sequence of SELECTs can be applied in any order. In addition, we can
In SQL, the SELECT condition is typically specified in the WHERE clause of a query.
The resulting relation is shown in Figure 1(b). The general form of the PROJECT
If the attribute list includes only nonkey attributes of R, duplicate tuples are likely to
The result is shown in Figure 1(c). Notice that the tuple <‘F’, 25000> appears only
In SQL, the PROJECT attribute list is specified in the SELECT clause of a query. For
Figure 2(a) shows the result of this in-line relational algebra expression.
It is sometimes simpler to break down a complex sequence of operations by specify-
result relations. This can be useful in connection with more complex operations
These two operations are illustrated in Figure 2(b).
If no renaming is applied, the names of the attributes in the resulting relation of a
where the symbol ρ (rho) is used to denote the RENAME operator, S is the new rela-
In SQL, a single query typically represents a complex relational algebra expression.
4As a single relational algebra expression, this becomes Result ← πSsn (σDno=5 (EMPLOYEE) ) ∪
Several set theoretic operations are used to merge the elements of two sets in vari-
Figure 4 illustrates the three operations. The relations STUDENT and INSTRUCTOR
The resulting relations from this sequence of operations are shown in Figure 5. The
Dependent_name Sex . . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
Lname Ssn Essn Dependent_name Sex . . .Bdate
Essn value match the Ssn value of the EMPLOYEE tuple. The ACTUAL_DEPENDENTS
The first operation is illustrated in Figure 6. Note that Mgr_ssn is a foreign key of the
. . . . . .
. . .
. . .
. . .
. . .
. . .
. . .
The general form of a JOIN operation on two relations5 R(A1, A2, …, An) and S(B1,
The result of the JOIN is a relation Q with n + m attributes Q(A1, A2, …, An, B1, B2,
5Again, notice that R and S can be any relations that result from general relational algebra expressions.
The same query can be done in two steps by creating an intermediate table DEPT as
If the attributes on which the natural join is specified already have the same names in
6NATURAL JOIN is basically an EQUIJOIN followed by the removal of the superfluous attributes.
Notice that if no combination of tuples satisfies the join condition, the result of a S will have between zero and
As we can see, a single JOIN operation is used to combine data from two relations so
outer joins (see Section 4.4). Informally, an inner join is a type of match and com-
This combines each project tuple with its controlling department tuple into a single
In SQL, JOIN can be realized in several different ways. The first method is to specify in the WHERE clause, along with any other selection condi-
Finally, apply the DIVISION operation to the two relations, which gives the desired
The preceding operations are shown in Figure 8(a).
In general, the DIVISION operation is applied to two relations R(Z) ÷ S(X), where
Figure 8(b) illustrates a DIVISION operation where X = {A}, Y = {B}, and Z = {A, B}.
The DIVISION operation can be expressed as a sequence of π, ×, and – operations as
A query tree is a tree data structure that corresponds to a relational algebra expres-
Figure 9 shows a query tree for Query 2: For every project located in ‘Stafford’, list the
In Figure 9, the three leaf nodes P, D, and E represent the three relations PROJECT,
are represented by internal tree nodes. The query tree signifies an explicit order of
where F1, F2, …, Fn are functions over the attributes in relation R and may involve
REPORT ← ρ(Ssn, Net_salary, Bonus, Tax)(πSsn, Salary – Deduction, 2000 * Years_service,
(EMPLOYEE)).
7There is no single agreed-upon notation for specifying aggregate functions. In some cases a “script A” is
resulting relation has the grouping attributes plus one attribute for each element in
ρR(Dno, No_of_employees, Average_sal)(Dno ℑ COUNT Ssn, AVERAGE Salary (EMPLOYEE))
If no grouping attributes are specified, the functions are applied to all the tuples in
It is important to note that, in general, duplicates are not eliminated when an aggre-
8Note that this is an arbitrary notation we are suggesting. There is no standard notation.
It is relatively straightforward in the relational algebra to specify all employees
To retrieve all employees supervised by Borg at level 2—that is, all employees e��
To get both sets of employees supervised at levels 1 and 2 by ‘James Borg’, we can
The results of these queries are illustrated in Figure 11. Although it is possible to
An operation called the transitive closure of relations has been proposed to compute
9In SQL, the option of eliminating duplicates before applying the aggregate function is available by
A set of operations, called outer joins, were developed for the case where the user
The LEFT OUTER JOIN operation keeps every tuple in the first, or left, relation R in R
filled or padded with NULL values. The result of these operations is shown in Figure
relation are also kept in the result relation T(X, Y, Z). It is therefore the same as a
Two tuples t1 in R and t2 in S are said to match if t1[X]=t2[X]. These will be com-
All the tuples from both relations are included in the result, but tuples with the same
Notice that the same person may still appear twice in the result. For example, we
The following are additional examples to illustrate the use of the relational algebra
Query 1. Retrieve the name and address of all employees who work for the
11Note that OUTER UNION is equivalent to a FULL OUTER JOIN if the join attributes are all the com-
This query could be specified in other ways; for example, the order of the JOIN and
Query 2. For every project located in ‘Stafford’, list the project number, the
In this example, we first select the projects located in Stafford, then join them with
Query 3. Find the names of employees who work on all the projects controlled
In this query, we first create a table DEPT5_PROJS that contains the project numbers
Query 4. Make a list of project numbers for projects that involve an employee
Query 5. List the names of all employees with two or more dependents.
Query 6. Retrieve the names of employees who have no dependents.
We first retrieve a relation with all employee Ssns in ALL_EMPS. Then we create a
Query 7. List the names of managers who have at least one dependent.
In this query, we retrieve the Ssns of managers in MGRS, and the Ssns of employees
As we mentioned earlier, the same query can be specified in many different ways in
It has been shown that any retrieval that can be specified in the basic relational alge-
12When queries are optimized, the system will choose a particular sequence of operations that corre-
Our examples refer to the database shown in Figures A.1 and A.3. We will use the
The condition EMPLOYEE(t) specifies that the range relation of tuple variable t is
The above query retrieves all attribute values for each selected EMPLOYEE tuple t. To
Informally, we need to specify the following information in a tuple relational calcu-
■ For each tuple variable t, the range relation R of t. This value is specified by
■ A condition to select particular combinations of tuples. As tuple variables
■ A set of attributes to be retrieved, the requested attributes. The values of
Before we discuss the formal syntax of tuple relational calculus, consider another
Query 0. Retrieve the birth date and address of the employee (or employees)
In tuple relational calculus, we first specify the requested attributes t.Bdate and
where t1, t2, …, tn, tn+1, …, tn+m are tuple variables, each Ai is an attribute of the rela-
1. An atom of the form R(ti), where R is a relation name and ti is a tuple vari-
2. An atom of the form ti.A op tj.B, where op is one of the comparison opera-
3. An atom of the form ti.A op c or c op tj.B, where op is one of the compari-
Each of the preceding atoms evaluates to either TRUE or FALSE for a specific combi-
■ Rule 1: Every atom is a formula.
13Also called a well-formed formula, or WFF, in mathematical logic.
a. (F1 AND F2) is TRUE if both F1 and F2 are TRUE; otherwise, it is FALSE.
b. (F1 OR F2) is FALSE if both F1 and F2 are FALSE; otherwise, it is TRUE.
c. NOT (F1) is TRUE if F1 is FALSE; it is FALSE if F1 is TRUE.
d. NOT (F2) is TRUE if F2 is FALSE; it is FALSE if F2 is TRUE.
■ An occurrence of a tuple variable in a formula F that is an atom is free in F.
■ All free occurrences of a tuple variable t in F are bound in a formula F� of the
The tuple variable d is free in both F1 and F2, whereas it is bound to the (∀∀) quan-
We can now give Rules 3 and 4 for the definition of a formula we started earlier:
■ Rule 3: If F is a formula, then so is (∃∃t)(F), where t is a tuple variable. The
■ Rule 4: If F is a formula, then so is (∀∀t)(F), where t is a tuple variable. The
The (∃∃) quantifier is called an existential quantifier because a formula (∃∃t)(F) is
(∀∀t)(F) is TRUE if every possible tuple that can be assigned to free occurrences of t
Query 1. List the name and address of all employees who work for the
The only free tuple variables in a tuple relational calculus expression should be those
Query 2. For every project located in ‘Stafford’, list the project number, the
In Q2 there are two free tuple variables, p and m. Tuple variable d is bound to the
Several tuple variables in a query can range over the same relation. For example, to
Query 3�. List the name of each employee who works on some project con-
changed to some. In this case we need two join conditions and two existential
Query 4. Make a list of project numbers for projects that involve an employee
Compare this with the relational algebra version of this query in Section 5. The
In the next section we discuss the relationship between the universal and existential
Query 3. List the names of employees who work on all the projects controlled
We want to make sure that a selected employee e works on all the projects controlled
1. For the formula F� = (∀∀x)(F) to be TRUE, we must have the formula F be
2. Using the same line of reasoning, we do not want to consider tuples in the
3. Formula F1, hence, is of the form NOT(x.Dnum=5) OR F2. In the context of
4. Finally, F2 gives the condition that we want to hold for a selected EMPLOYEE
In English, Q3 gives the following condition for selecting an EMPLOYEE tuple e: For
Using the general transformation from universal to existential quantifiers given in
We now give some additional examples of queries that use quantifiers.
Query 6. List the names of employees who have no dependents.
Query 7. List the names of managers who have at least one dependent.
This query is handled by interpreting managers who have at least one dependent as
is unsafe because it yields all tuples in the universe that are not EMPLOYEE tuples,
An expression is said to be safe if all values in its result are from the domain of the
Domain calculus differs from tuple calculus in the type of variables used in formu-
A formula is made up of atoms. The atoms of a formula are slightly different from
1. An atom of the form R(x1, x2, …, xj), where R is the name of a relation of must be a tuple in the relation whose name is R,
2. An atom of the form xi op xj, where op is one of the comparison operators in
3. An atom of the form xi op c or c op xj, where op is one of the comparison
As in tuple calculus, atoms evaluate to either TRUE or FALSE for a specific set of val-
In a similar way to the tuple relational calculus, formulas are made up of atoms,
Query 0. List the birth date and address of the employee whose name is ‘John
An alternative shorthand notation, used in QBE, for writing this query is to assign
Query 1. Retrieve the name and address of all employees who work for the
A condition relating two domain variables that range over attributes from two rela-
14Note that the notation of quantifying only the domain variables actually used in conditions and of
Query 2. For every project located in ‘Stafford’, list the project number, the
Query 6. List the names of employees who have no dependents.
Q6 can be restated using universal quantifiers instead of the existential quantifiers,
Query 7. List the names of managers who have at least one dependent.
As we mentioned earlier, it can be shown that any query that can be expressed in the
In Sections 1 through 3, we introduced the basic relational algebra operations and
are applied be union (or type) compatible; these include UNION, INTERSECTION, and
We discussed some important types of queries that cannot be stated with the basic
The last two sections described the basic concepts behind relational calculus, which
We discussed the syntax of relational calculus queries using both tuple and domain
2. What is union compatibility? Why do the UNION, INTERSECTION, and
3. Discuss some types of queries for which renaming of attributes is necessary
4. Discuss the various types of inner join operations. Why is theta join
5. What role does the concept of foreign key play when specifying the most
6. What is the FUNCTION operation? What is it used for?
7. How are the OUTER JOIN operations different from the INNER JOIN opera-
8. In what sense does relational calculus differ from relational algebra, and in
9. How does tuple relational calculus differ from domain relational calculus?
10. Discuss the meanings of the existential quantifier (∃∃) and the universal
11. Define the following terms with respect to the tuple calculus: tuple variable,
12. Define the following terms with respect to the domain calculus: domain vari-
13. What is meant by a safe expression in relational calculus?
14. When is a query language called relationally complete?
to the database state in Figure A.1.
16. Specify the following queries on the COMPANYrelational database schema
a. Retrieve the names of all employees in department 5 who work more than
b. List the names of all employees who have a dependent with the same first
c. Find the names of all employees who are directly supervised by ‘Franklin
d. For each project, list the project name and the total hours per week (by all
e. Retrieve the names of all employees who work on every project.
f. Retrieve the names of all employees who do not work on any project.
g. For each department, retrieve the department name and the average
h. Retrieve the average salary of all female employees.
i. Find the names and addresses of all employees who work on at least one
j. List the last names of all department managers who have no dependents.
17. From the chapter “The Relational Data Model and Relational Database
a. For each flight, list the flight number, the departure airport for the first leg
b. List the flight numbers and weekdays of all flights or flight legs that
c. List the flight number, departure airport code, scheduled departure time,
d. List all fare information for flight number ‘CO197’.
e. Retrieve the number of available seats for flight number ‘CO197’ on
18. Consider the LIBRARY relational database schema shown in Figure 14, which
a. How many copies of the book titled The Lost Tribe are owned by the
b. How many copies of the book titled The Lost Tribe are owned by each
c. Retrieve the names of all borrowers who do not have any books checked
d. For each book that is loaned out from the Sharpstown branch and whose
e. For each library branch, retrieve the branch name and the total number
f. Retrieve the names, addresses, and number of books checked out for all
g. For each book authored (or coauthored) by Stephen King, retrieve the
19. Specify the following queries in relational algebra on the database schema
a. List the Order# and Ship_date for all orders shipped from Warehouse# W2.
b. List the WAREHOUSE information from which the CUSTOMER named
c. Produce a listing Cname, No_of_orders, Avg_order_amt, where the middle
d. List the orders that were not shipped within 30 days of ordering.
e. List the Order# for orders that were shipped from all warehouses that the
20. Specify the following queries in relational algebra on the database schema
a. Give the details (all attributes of trip relation) for trips that exceeded
b. Print the Ssns of salespeople who took trips to Honolulu.
c. Print the total trip expenses incurred by the salesperson with SSN = ‘234-
21. Specify the following queries in relational algebra on the database schema
a. List the number of courses taken by all students named John Smith in
b. Produce a list of textbooks (include Course#, Book_isbn, Book_title) for
c. List any department that has all its adopted books published by ‘Pearson
22. Consider the two tables T1 and T2 shown in Figure 15. Show the results of
a. T1 T1.P = T2.A T2
b. T1 T1.Q = T2.B T2
c. T1 T1.P = T2.A T2
d. T1 T1.Q = T2.B T2
e. T1 ∪ T2
23. Specify the following queries in relational algebra on the database schema in
a. For the salesperson named ‘Jane Doe’, list the following information for
b. List the Serial# and Model of cars that have no options.
c. Consider the NATURAL JOIN operation between SALESPERSON and
d. Write a query in relational algebra involving selection and one set opera-
24. Specify queries a, b, c, e, f, i, and j of Exercise 16 in both tuple and domain
25. Specify queries a, b, c, and d of Exercise 17 in both tuple and domain rela-
26. Specify queries c, d, and f of Exercise 18 in both tuple and domain relational
27. In a tuple relational calculus query with n tuple variables, what would be the
28. Rewrite the domain relational calculus queries that followed Q0 in Section 7
29. Consider this query: Retrieve the Ssns of employees who work on at least
■ x is a tuple variable that ranges over the PROJECT relation.
■ P ≡ EMPLOYEE with Ssn=123456789 works on PROJECT x.
■ (∀∀ x)(P(x)) ≡ NOT(∃∃x)(NOT(P(x))).
30. Show how you can specify the following relational algebra operations in
a. σA=C(R(A, B, C))(R(A, B, C))
d. R(A, B, C) ∪ S(A, B, C)
f. R(A, B, C) = S(A, B, C)
g. R(A, B, C) × S(D, E, F)
31. Suggest extensions to the relational calculus so that it may express the fol-
32. A nested query is a query within a query. More specifically, a nested query is
a. List the names of all employees who work in the department that has the
b. List the names of all employees whose supervisor’s supervisor has
c. List the names of employees who make at least $10,000 more than the
33. State whether the following conclusions are true or false:
a. NOT (P(x) OR Q(x)) → (NOT (P(x)) AND (NOT (Q(x)))
the RA interpreter on the COMPANY database schema in Figure A.2.
a. List the names of all employees in department 5 who work more than 10
b. List the names of all employees who have a dependent with the same first
c. List the names of employees who are directly supervised by Franklin
d. List the names of employees who work on every project.
e. List the names of employees who do not work on any project.
f. List the names and addresses of employees who work on at least one proj-
g. List the names of department managers who have no dependents.
35. Consider the following MAILORDER relational schema describing the data
a. Retrieve the names of parts that cost less than $20.00.
b. Retrieve the names and cities of employees who have taken orders for
c. Retrieve the pairs of customer number values of customers who live in
d. Retrieve the names of customers who have ordered parts from employees
e. Retrieve the names of customers who have ordered parts costing less than
f. Retrieve the names of customers who have not placed an order.
g. Retrieve the names of customers who have placed exactly two orders.
36. Consider the following GRADEBOOK relational schema describing the data
a. Retrieve the names of students enrolled in the Automata class during the
b. Retrieve the Sid values of students who have enrolled in CSc226 and
c. Retrieve the Sid values of students who have enrolled in CSc226 or
d. Retrieve the names of students who have not enrolled in any class.
e. Retrieve the names of students who have enrolled in all courses in the
37. Consider a database that consists of the following relations.
a. Retrieve the part numbers that are supplied to exactly two projects.
b. Retrieve the names of suppliers who supply more than two parts to proj-
c. Retrieve the part numbers that are supplied by every supplier.
d. Retrieve the project names that are supplied by supplier ‘S1’ only.
e. Retrieve the names of suppliers who supply at least two different parts
38. Specify and execute the following queries for the database in Exercise 16 of
a. Retrieve the names of students who have enrolled in a course that uses a
b. Retrieve the names of courses in which the textbook has been changed at
c. Retrieve the names of departments that adopt textbooks published by
d. Retrieve the names of departments that adopt textbooks written by
e. Retrieve the names of students who have never used a book (in a course)
39. Repeat Laboratory Exercises 34 through 38 in domain relational calculus
Conceptual modeling is a very important phase indesigning a successful database application.
In this chapter, we follow the traditional approach of concentrating on the database
From Chapter 7 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
This chapter is organized as follows: Section 1 discusses the role of high-level con-
The material in Sections 8 and 9 may be excluded from an introductory course.
Figure 1 shows a simplified overview of the database design process. The first step
1A class is similar to an entity type in many ways.
the known functional requirements of the application. These consist of the user-
Once the requirements have been collected and analyzed, the next step is to create a
the data requirements of the users and includes detailed descriptions of the entity
During or after the conceptual schema design, the basic data model operations can
The next step in database design is the actual implementation of the database, using
The last step is the physical design phase, during which the internal storage struc-
■ The company is organized into departments. Each department has a unique
The ER model describes data as entities, relationships, and attributes. In Section 3.1
2The Social Security number, or SSN, is a unique nine-digit identifier assigned to each individual in the
value for each of its attributes. The attribute values that describe each entity become
Figure 3 shows two entities and the values of their attributes. The EMPLOYEE entity
Several types of attributes occur in the ER model: simple versus composite, single-
types and illustrate their use via examples. Then we discuss the concept of a NULL
Composite versus Simple (Atomic) Attributes. Composite attributes can be
3Zip Code is the name used in the United States for a five-digit postal code, such as 76019, which can
need to subdivide it into component attributes. For example, if there is no need to
Single-Valued versus Multivalued Attributes. Most attributes have a single
Stored versus Derived Attributes. In some cases, two (or more) attribute val-
NULL Values. In some cases, a particular entity may not have an applicable value
Complex Attributes. Notice that, in general, composite and multivalued attrib-
ponents of a composite attribute between parentheses () and separating the compo-
Entity Types and Entity Sets. A database usually contains groups of entities that
4For those familiar with XML, we should note that complex attributes are similar to complex elements in
each. A few individual entities of each type are also illustrated, along with the values
An entity type is represented in ER diagrams5 (see Figure 2) as a rectangular box
An entity type describes the schema or intension for a set of entities that share the
Key Attributes of an Entity Type. An important constraint on the entities of an
5We use a notation for ER diagrams that is close to the original proposed notation (Chen 1976). Many
has one or more attributes whose values are distinct for each individual entity in the
Some entity types have more than one key attribute. For example, each of the
In our diagrammatic notation, if two attributes are underlined separately, then each
Value Sets (Domains) of Attributes. Each simple attribute of an entity type is
Mathematically, an attribute A of entity set E whose value set is V can be defined as
We refer to the value of attribute A for entity e as A(e). The previous definition cov-
The value set provides all possible values. Usually only a small number of these val-
1. An entity type DEPARTMENT with attributes Name, Number, Locations,
2. An entity type PROJECT with attributes Name, Number, Location, and
3. An entity type EMPLOYEE with attributes Name, Ssn, Sex, Address, Salary,
4. An entity type DEPENDENT with attributes Employee, Dependent_name, Sex,
6The power set P (V ) of a set V is the set of all subsets of V.
So far, we have not represented the fact that an employee can work on several proj-
In Figure 8 there are several implicit relationships among the various entity types. In
This section is organized as follows: Section 4.1 introduces the concepts of relation-
Informally, each relationship instance ri in R is an association of entities, where the
Degree of a Relationship Type. The degree of a relationship type is the number
Relationships as Attributes. It is sometimes convenient to think of a binary
8This concept of representing relationship types as attributes is used in a class of data models called
Role Names and Recursive Relationships. Each entity type that participates
Role names are not technically necessary in relationship types where all the partici-
Cardinality Ratios for Binary Relationships. The cardinality ratio for a binary
9N stands for any number of related entities (zero or more).
world rule is that an employee can work on several projects and a project can have
Cardinality ratios for binary relationships are represented on ER diagrams by dis-
Participation Constraints and Existence Dependencies. The participation
cardinality ratio and participation constraints, taken together, as the structural
We will discuss constraints on higher-degree relationships in Section 9.
Notice that attributes of 1:1 or 1:N relationship types can be migrated to one of the
For a 1:N relationship type, a relationship attribute can be migrated only to the
In ER diagrams, both a weak entity type and its identifying relationship are distin-
10The identifying entity type is also sometimes called the parent entity type or the dominant entity
weak entity type representation if there are many attributes. If the weak entity par-
We can now refine the database design in Figure 8 by changing the attributes that
■ MANAGES, a 1:1 relationship type between EMPLOYEE and DEPARTMENT.
■ CONTROLS, a 1:N relationship type between DEPARTMENT and PROJECT.
13The rules in the miniworld that determine the constraints are sometimes called the business rules,
type DEPENDENT. The participation of EMPLOYEE is partial, whereas that of
After specifying the above six relationship types, we remove from the entity types in
Figure 2 displays the COMPANY ER database schema as an ER diagram. We now
In Figure 2 the cardinality ratio of each binary relationship type is specified by
constraint is specified by a single line for partial participation and by double lines
Figure 14 summarizes the conventions for ER diagrams. It is important to note that
As a general practice, given a narrative description of the database requirements, the
. . .
In general, the schema design process should be considered an iterative refinement
■ Section 9 discusses choices concerning the degree of a relationship.
In this section, we describe one alternative ER notation for specifying structural
14In some notations, particularly those used in object modeling methodologies such as UML, the (min,
max relationship instances in R at any point in time. In this method, min = 0 implies partial participation,
Figure 15 displays the COMPANY database schema using the (min, max) notation.14 Usually, one uses
. . .
notation is more precise, and we can use it to specify some structural constraints for
Figure 15 also displays all the role names for the COMPANY database schema.
In UML class diagrams, a class (similar to an entity type in ER) is displayed as a box
In UML, there are two types of relationships: association and aggregation.
value. Association (relationship) names are optional in UML, and relationship
The operations given in each class are derived from the functional requirements of
Weak entities can be modeled using the construct called qualified association (or
This section is not meant to be a complete description of UML class diagrams, but
In Section 4.2 we defined the degree of a relationship type as the number of partic-
The ER diagram notation for a ternary relationship type is shown in Figure 17(a),
some part to project j. The existence of three relationship instances (s, p), ( j, p), and
It is also possible to represent the ternary relationship as a regular entity type by
Another example is shown in Figure 18. The ternary relationship type OFFERS rep-
semester to the courses offered during that semester by any instructor. These ternary
Although in general three binary relationships cannot replace a ternary relationship,
Notice that it is possible to have a weak entity type with a ternary (or n-ary) identi-
There are two notations for specifying structural constraints on n-ary relationships,
Recall that the relationship set of SUPPLY is a set of relationship instances (s, j, p),
The second notation is based on the (min, max) notation displayed in Figure 15 for
15This notation allows us to determine the key of the relationship relation.
the various types of attributes, which can be nested arbitrarily to produce complex
We also briefly discussed stored versus derived attributes. Then we discussed the ER
We presented two methods for specifying the structural constraints on relationship
We noted that, alternatively, another method of specifying structural constraints is
Entity-Relationship schemas can be represented diagrammatically as ER diagrams.
2. List the various cases where use of a NULL value would be appropriate.
3. Define the following terms: entity, attribute, attribute value, relationship
4. What is an entity type? What is an entity set? Explain the differences among
5. Explain the difference between an attribute and a value set.
6. What is a relationship type? Explain the differences among a relationship
7. What is a participation role? When is it necessary to use role names in the
8. Describe the two alternatives for specifying structural constraints on rela-
9. Under what conditions can an attribute of a binary relationship type be
10. When we think of relationships as attributes, what are the value sets of these
11. What is meant by a recursive relationship type? Give some examples of
12. When is the concept of a weak entity used in data modeling? Define the
13. Can an identifying relationship of a weak entity type be of a degree greater
14. Discuss the conventions for displaying an ER schema as an ER diagram.
15. Discuss the naming conventions used for ER schema diagrams.
a. The university keeps track of each student’s name, student number, Social
(B.A., B.S., …, Ph.D.). Some user applications need to refer to the city,
b. Each department is described by a name, department code, office num-
c. Each course has a course name, description, course number, number of
d. Each section has an instructor, semester, year, course, and section num-
e. A grade report has a student, section, letter grade, and numeric grade (0,
17. Composite and multivalued attributes can be nested to any number of levels.
18. Show an alternative design for the attribute described in Exercise 17 that uses
19. Consider the ER diagram in Figure 20, which shows a simplified schema for
20. We can consider many entity types to describe database environment and
21. Design an ER schema for keeping track of information about votes taken in
22. A database is being constructed to keep track of the teams and games of a
23. Consider the ER diagram shown in Figure 21 for part of a BANK database.
a. List the strong (nonweak) entity types in the ER diagram.
b. Is there a weak entity type? If so, give its name, partial key, and identifying
c. What constraints do the partial key and the identifying relationship of the
d. List the names of all relationship types, and specify the (min, max) con-
e. List concisely the user requirements that led to this ER schema design.
f. Suppose that every customer must have at least one account but is
24. Consider the ER diagram in Figure 22. Assume that an employee may work
25. Consider the ER diagram in Figure 23. Assume that a course may or may not
two to four courses. Supply (min, max) constraints on this diagram. State
26. Consider an entity type SECTION in a UNIVERSITY database, which
27. Cardinality ratios often dictate the detailed design of a database. The cardi-
1. STUDENT ______________ SOCIAL_SECURITY_CARD
2. STUDENT ______________ TEACHER
3. CLASSROOM ______________ WALL
4. COUNTRY ______________ CURRENT_PRESIDENT
5. COURSE ______________ TEXTBOOK
6. ITEM (that can
7. STUDENT ______________ CLASS
8. CLASS ______________ INSTRUCTOR
9. INSTRUCTOR ______________ OFFICE
10. EBAY_AUCTION
28. Consider the ER schema for the MOVIES database in Figure 24.
Assume that MOVIES is a populated database. ACTOR is used as a generic
a. There are no actors in this database that have been in no movies.
b. There are some actors who have acted in more than ten movies.
c. Some actors have done a lead role in multiple movies.
d. A movie can have only a maximum of two lead actors.
e. Every director has been an actor in some movie.
f. No producer has ever been an actor.
g. A producer cannot be an actor in some other movie.
h. There are movies with more than a dozen actors.
i. Some producers have been a director as well.
j. Most movies have one director and one producer.
k. Some movies have one director but several producers.
l. There are some actors who have done a lead role, directed a movie, and
m. No movie has a director who also acted in that movie.
29. Given the ER schema for the MOVIES database in Figure 24, draw an instance
30. Illustrate the UML Diagram for Exercise 16. Your UML design should
a. A student should have the ability to compute his/her GPA and add or
b. Each department should be to able add or delete courses and hire or ter-
c. Each instructor should be able to assign or change a student’s grade for a
Note: Some of these functions may be spread over multiple classes.
32. Consider a MAIL_ORDER database in which employees take orders for parts
33. Consider a MOVIE database in which data is recorded about the movie indus-
■ Each movie is identified by title and year of release. Each movie has a
■ Production companies are identified by name and each has an address. A
34. Consider a CONFERENCE_REVIEW database in which researchers submit
■ Authors of papers are uniquely identified by e-mail id. First and last
■ Reviewers of papers are uniquely identified by e-mail address. Each
■ Each paper is assigned between two and four reviewers. A reviewer rates
35. Consider the ER diagram for the AIRLINE database shown in Figure 20. Build
Entity – Relationship (ER) modeling concepts aresufficient for representing many database schemas
In this chapter, we describe features that have been proposed for semantic data mod-
2EER has also been used to stand for Extended ER model.
1CAD/CAM stands for computer-aided design/computer-aided manufacturing.
From Chapter 8 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
We present the UML class diagram notation for representing specialization and gen-
an employee. We call each of these subgroupings a subclass or subtype of the
3A class/subclass relationship is often called an IS-A (or IS-AN) relationship because of the way we
An entity cannot exist in the database merely by being a member of a subclass; it
An important concept associated with subclasses (subtypes) is that of type inheri-
4In some object-oriented programming languages, a common restriction is that an entity (or object) has
attributes) of the subclass. Similarly, a subclass can participate in specific relation-
Figure 2 shows a few entity instances that belong to subclasses of the {SECRETARY,
There are two main reasons for including class/subclass relationships and specializa-
The second reason for using subclasses is that some relationship types may be par-
Notice that the generalization process can be viewed as being functionally the inverse
So far we have introduced the concepts of subclasses and superclass/subclass rela-
superclass or subclass represents a collection of entities of the same type and hence
First, we discuss constraints that apply to a single specialization or a single general-
each of the specializations. However, a specialization may also consist of a single
If all subclasses in a specialization have their membership condition on the same
6Such an attribute is called a discriminator in UML terminology.
Two other constraints may apply to a specialization. The first is the disjointness (or
The second constraint on specialization is called the completeness (or totalness)
Notice that the disjointness and completeness constraints are independent. Hence,
7The notation of using single or double lines is similar to that for partial or total participation of an entity
Certain insertion and deletion rules apply to specialization (and generalization) as a
The reader is encouraged to make a complete list of rules for insertions and dele-
A subclass itself may have further subclasses specified on it, forming a hierarchy or a
one parent, which results in a tree structure or strict hierarchy. In contrast, for a
Figure 7 shows another specialization lattice of more than one level. This may be
The requirements for the part of the UNIVERSITY database shown in Figure 7 are the
1. The database keeps track of three types of persons: employees, alumni, and
2. Every employee has a salary, and there are three types of employees: faculty,
3. Each faculty has a rank, whereas each staff member has a staff position.
4. Students are further classified as either graduate or undergraduate, with the
In such a specialization lattice or hierarchy, a subclass inherits the attributes not
8In some models, the class is further restricted to be a leaf node in the hierarchy or lattice.
It is important to note here that some models and languages are limited to single
Although we have used specialization to illustrate our discussion, similar concepts
Now we elaborate on the differences between the specialization and generalization
It is possible to arrive at the same hierarchy or lattice from the other direction. In
In structural terms, hierarchies or lattices resulting from either process may be iden-
9Our use of the term category is based on the ECR (Entity-Category-Relationship) model (Elmasri et al.
A category has two or more superclasses that may represent distinct entity types,
classes. This represents the constraint that an OWNER may be a COMPANY, a BANK,
Attribute inheritance works more selectively in the case of categories. For example,
A category can be total or partial. A total category holds the union of all entities in
The superclasses of a category may have different key attributes, as demonstrated by
It is important to note that some modeling methodologies do not have union types.
In this section, we first give an example of a database schema in the EER model to
GRAD_STUDENT is a subclass of STUDENT, with the defining predicate Class = 5.
10We assume that the quarter system rather than the semester system is used in this university.
date [St_date]. A grant is related to one principal investigator [PI] and to all
Conceptual database design should be considered as an iterative refinement process
■ If a subclass has few specific (local) attributes and no specific relationships, it
■ Similarly, if all the subclasses of a specialization/generalization have few spe-
is a set or collection of entities; this includes any of the EER schema constructs of
A specialization Z = {S1, S2, …, Sn} is a set of subclasses that have the same super-
Otherwise, Z is said to be partial. Z is said to be disjoint if we always have
Otherwise, Z is said to be overlapping.
A subclass S of C is said to be predicate-defined if a predicate p on the attributes of
A specialization Z (or generalization G) is said to be attribute-defined if a predicate
11The use of the word class here differs from its more common use in object-oriented programming lan-
is used to specify membership in each subclass Si in Z. Notice that if ci ≠ cj for i ≠ j,
A category T is a class that is a subset of the union of n defining superclasses D1, D2,
In database design, we are mainly concerned with specifying concrete classes whose
. . .
. . .
In this section we discuss in general terms some of the modeling concepts of the ER
■ KR is generally broader in scope than semantic data models. Different forms
12An ontology is somewhat similar to a conceptual schema, but with more knowledge, rules, and excep-
In general, the objects of a class should have a similar type structure. However, some
In the EER model, entities are classified into entity types according to their basic
At the object level, the values of key attributes are used to distinguish among entities
an object to form the whole object. The second case is when we represent an aggre-
The abstraction of association is used to associate objects from several independent
One way to represent this situation is to create a higher-level aggregate class com-
To represent this situation correctly in the ER model as described here, we need to
The study of ontologies attempts to describe the structures and relationships that
In this definition, a conceptualization is the set of concepts that are used to repre-
Usually the concepts used to describe ontologies are quite similar to the concepts we
13This definition is given in Gruber (1995).
world from reality in order to store and manage data. An ontology is usually consid-
Next, we showed how to display these new constructs in an EER diagram. We also
2. Define the following terms: superclass of a subclass, superclass/subclass rela-
3. Discuss the mechanism of attribute/relationship inheritance. Why is it use-
4. Discuss user-defined and predicate-defined subclasses, and identify the dif-
5. Discuss user-defined and attribute-defined specializations, and identify the
6. Discuss the two main types of constraints on specializations and generaliza-
7. What is the difference between a specialization hierarchy and a specialization
8. What is the difference between specialization and generalization? Why do we
9. How does a category differ from a regular shared subclass? What is a cate-
10. For each of the following UML terms (see Section 6), discuss the correspon-
11. Discuss the main differences between the notation for EER schema diagrams
12. List the various data abstraction concepts and the corresponding modeling
13. What aggregation feature is missing from the EER model? How can the EER
14. What are the main similarities and differences between conceptual database
15. Discuss the similarities and differences between an ontology and a database
Specify all constraints that should hold on the database. Make sure that the
17. Consider the BANK ER schema in Figure A.1 (at end of the chapter), and
18. The following narrative describes a simplified version of the organization of
19. Identify all the important concepts represented in the library database case
Books can be checked out for 21 days. Members are allowed to have only five
borrow books at least ten times during the year. The top 1 percent of mem-
20. Design a database to keep track of information for an art museum. Assume
■ The museum has a collection of ART_OBJECTS. Each ART_OBJECT has a
■ ART_OBJECTS are categorized based on their type. There are three main
Draw an EER schema diagram for this application. Discuss any assumptions
21. Figure 12 shows an example of an EER diagram for a small private airport
10. EMPLOYEE SSN
22. Show how the UNIVERSITY EER schema in Figure 9 may be represented in
23. Consider the entity sets and attributes shown in the table below. Place a
a. The left side has a relationship with the right side.
b. The right side is an attribute of the left side.
c. The left side is a specialization of the right side.
d. The left side is a generalization of the right side.
24. Draw a UML diagram for storing a played game of chess in a database. You
1. The game of chess is played between two players.
2. The game is played on an 8 × 8 board like the one shown below:
3. The players are assigned a color of black or white at the start of the game.
4. Each player starts with the following pieces (traditionally called chess-
a. king
d. 2 bishops
5. Every piece has its own initial position.
6. Every piece has its own set of legal moves based on the state of the game.
a. A piece may move to an empty square or capture an opposing piece.
another piece (queen, rook, bishop, or knight).
Note: Some of these functions may be spread over multiple classes.
25. Draw an EER diagram for a game of chess as described in Exercise 24. Focus
26. Which of the following EER diagrams is/are incorrect and why? State clearly
a.
b.
c.
27. Consider the following EER diagram that describes the computer systems at
points earned by individual students in their classes. The data requirements are summarized as
■ Each student is identified by a unique identifier, first and last name, and an e-mail address.
■ Each instructor teaches certain courses each term. Each course is identified by a course num-
■ Students are enrolled in each course taught by the instructor.
29. Consider an ONLINE_AUCTION database system in which members (buyers
■ A member may be a buyer or a seller. A buyer has a shipping address
■ Buyers make bids for items they are interested in. Bid price and time of
30. Consider a database system for a baseball organization such as the major
■ Within the players group is a subset of players called pitchers. Pitchers
■ Teams are uniquely identified by their names. Teams are also described by
■ Teams have one manager, a number of coaches, and a number of players.
31. Consider the EER diagram for the UNIVERSITY database shown in Figure 9.
32. Consider the EER diagram for the small AIRPORT database shown in Figure
A survey of semantic data modeling appears in Hull and King (1987). Eick (1991)
Fensel (2000, 2003) discuss the Semantic Web and application of ontologies.
This chapter discusses how to design a relationaldatabase schema based on a conceptual schema
We outline a seven-step algorithm in Section 1 to convert the basic ER model con-
From Chapter 9 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
assume that the mapping will create tables with simple single-valued attributes.
Step 1: Mapping of Regular Entity Types. For each regular (strong) entity type
If multiple keys were identified for E during the conceptual design, the information
The relations that are created from the mapping of entity types are sometimes called
Step 2: Mapping of Weak Entity Types. For each weak entity type W in the ER
If there is a weak entity type E2 whose owner is also a weak entity type E1, then E1
necessary. The primary key of the DEPENDENT relation is the combination {Essn,
Step 3: Mapping of Binary 1:1 Relationship Types. For each binary 1:1 rela-
1. Foreign key approach: Choose one of the relations—S, say—and include as
2. Merged relation approach: An alternative mapping of a 1:1 relationship
3. Cross-reference or relationship relation approach: The third option is to
Step 4: Mapping of Binary 1:N Relationship Types. For each regular binary
An alternative approach is to use the relationship relation (cross-reference) option
Step 5: Mapping of Binary M:N Relationship Types. For each binary M:N
Notice that we can always map 1:1 or 1:N relationships in a manner similar to M:N
Step 6: Mapping of Multivalued Attributes. For each multivalued attribute A,
In our example, we create a relation DEPT_LOCATIONS (see Figure 3(d)). The
Step 7: Mapping of N-ary Relationship Types. For each n-ary relationship
. . .
. . .
. . .
simple components of composite attributes) as attributes of S. The primary key of S
One of the main points to note in a relational schema, in contrast to an ER schema,
For example, to form a relation that includes the employee name, project name, and
In the relational schema we create a separate relation for each multivalued attribute.
Next, we discuss the mapping of EER model constructs to relations by extending the
Step 8: Options for Mapping Specialization or Generalization. Convert
■ Option 8A: Multiple relations—superclass and subclasses. Create a rela-
■ Option 8B: Multiple relations—subclass relations only. Create a relation
■ Option 8C: Single relation with one type attribute. Create a single relation
Description Mflag Drawing_no Batch_no Pflag List_priceSupplier_nameManufacture_date
■ Option 8D: Single relation with multiple type attributes. Create a single
must hold for each Li. This specifies a foreign key from each Li to L, as well as an>
shown as an oval and stands for a specific task performed by the system. An actor,
Interaction Diagrams. The next two types of UML behavioral diagrams, interaction
F. Sequence Diagrams. Sequence diagrams describe the interactions between
showing the flow of messages between objects. Within the sequence diagram, an
G. Collaboration Diagrams. Collaboration diagrams represent interactions
H. Statechart Diagrams. Statechart diagrams describe how an object’s state
To describe the behavior of an object, it is common in most object-oriented tech-
The important elements of a statechart diagram shown in Figure 10 are as follows:
■ States. Shown as boxes with rounded corners, they represent situations in
■ Transitions. Shown as solid arrows between the states, they represent the
■ Start/Initial State. Shown by a solid circle with an outgoing arrow to a state.
ing into it from a state.
Statechart diagrams are useful in specifying how an object’s reaction to a message
7See Harel (1987).
I. Activity Diagrams. Activity diagrams present a dynamic view of the system by
In this section we will briefly illustrate the use of some of the UML diagrams we
Imagine a scenario with students enrolling in courses that are offered by professors.
charge of processing student aid applications for which the students have to apply.
As mentioned previously, one of the first steps involved in designing a database is to
Next, we can design a sequence diagram to visualize the execution of the use cases. For
Enroll student [count < 50]
Enroll student/set count = 0Course enrollment Enrolling
Entry/register student
Section closing
Canceled
Cancel
CancelCancel Count = 50
Exit/closesection
Do/enroll students
Figure 11
A sample statechart
diagram for the
UNIVERSITY
database.
341
Practical Database Design Methodology and Use of UML Diagrams
getSeatsLeft
getPreq = true &&
[getSeatsLeft =
True]/update
Schedule
:Registration
requestRegistration getCourseListing
selectCourse addCourse getPreReq
:Student
:Catalog :Course :Schedule
Figure 12
A sequence diagram for the UNIVERSITY database.
The catalog is first browsed to get course listings. Then, when the student selects a
course to register in, prerequisites and course capacity are checked, and the course is
then added to the student’s schedule if the prerequisites are met and there is space in
the course.
These UML diagrams are not the complete specification of the UNIVERSITY data-
base. There will be other use cases for the various applications of the actors, includ-
ing registrar, student, professor, and so on. A complete methodology for how to
arrive at the class diagrams from the various diagrams we illustrated in this section
is outside our scope here. Design methodologies remain a matter of judgment and
personal preferences. However, the designer should make sure that the class dia-
gram will account for all the specifications that have been given in the form of the
use cases, statechart, and sequence diagrams. The class diagram in Figure 13 shows
a possible class diagram for this application, with the structural relationships and
the operations within the classes. These classes will need to be implemented to
develop the UNIVERSITY database, and together with the operations they will
implement the complete class schedule/enrollment/aid application. Only some of
the attributes and methods (operations) are shown in Figure 13. It is likely that
these class diagrams will be modified as more details are specified and more func-
tions evolve in the UNIVERSITY application.
342
Practical Database Design Methodology and Use of UML Diagrams
REGISTRATION
. . .
findCourseAdd()
cancelCourse()
addCourse()
viewSchedule()
. . .()
CATALOG
. . .
getPreReq()
getSeatsLeft()
getCourseListing()
. . .()
FINANCIAL_AID
aidType
aidAmount
assignAid()
denyAid()
SCHEDULE
. . .
updateSchedule()
showSchedule()
. . .()
STUDENT
. . .
requestRegistration()
applyAid()
. . .()
PROFESSOR
. . .
enterGrades()
offerCourse()
. . .()
COURSE
time
classroom
seats
. . .
dropCourse()
addCourse()
. . .()
PERSON
Name
Ssn
. . .
viewSchedule()
. . .()
Figure 13
The design of the UNIVERSITY database as a class diagram.
4 Rational Rose: A UML-Based Design Tool
4.1 Rational Rose for Database Design
Rational Rose is one of the modeling tools used in the industry to develop informa-
tion systems. It was acquired by IBM in 2003. As we pointed out in the first two sec-
tions of this chapter, a database is a central component of most information
systems. Rational Rose provides the initial specification in UML that eventually
leads to the database development. Many extensions have been made in the latest
versions of Rose for data modeling, and now it provides support for conceptual,
logical, and physical database modeling and design.
343
Practical Database Design Methodology and Use of UML Diagrams
4.2 Rational Rose Data Modeler
Rational Rose Data Modeler is a visual modeling tool for designing databases.
Because it is UML-based, it provides a common tool and language to bridge the
communication gap between database designers and application developers. This
makes it possible for database designers, developers, and analysts to work together,
capture and share business requirements, and track them as they change through-
out the process. Also, by allowing the designers to model and design all specifica-
tions on the same platform using the same notation, it improves the design process
and reduces the risk of errors.
The process modeling capabilities in Rational Rose allow the modeling of the
behavior of database applications as we saw in the short example above, in the form
of use cases (Figure 8), sequence diagrams (Figure 12), and statechart diagrams
(Figure 11). There is the additional machinery of collaboration diagrams to show
interactions between objects and activity diagrams to model the flow of control,
which we did not show in our example. The eventual goal is to generate the database
specification and application code as much as possible. The Rose Data Modeler can
also capture triggers, stored procedures, and other modeling concepts explicitly in
the diagram rather than representing them with hidden tagged values behind the
scenes. The Rose Data Modeler also provides the capability to forward engineer a
database in terms of constantly changing requirements and reverse engineer an
existing implemented database into its conceptual design.
4.3 Data Modeling Using Rational Rose Data Modeler
There are many tools and options available in Rose Data Modeler for data modeling.
Reverse Engineering. Reverse engineering of a database allows the user to create
a conceptual data model based on an existing database schema specified in a DDL
file. We can use the reverse engineering wizard in Rational Rose Data Modeler for
this purpose. The reverse engineering wizard basically reads the schema in the data-
base or DDL file and recreates it as a data model. While doing so, it also includes the
names of all quoted identifier entities.
Forward Engineering and DDL Generation. We can also create a data model
directly from scratch in Rose. Having created the data model,8 we can also use it to
generate the DDL for a specific DBMS. There is a forward engineering wizard in the
Rose Data Modeler that reads the schema in the data model or reads both the
schema in the data model and the tablespaces in the data storage model and gener-
ates the appropriate DDL code in a DDL file. The wizard also provides the option of
generating a database by executing the generated DDL file.
8The term data model used by Rational Rose Data Modeler corresponds to our notion of an application
model or conceptual schema.
344
Practical Database Design Methodology and Use of UML Diagrams
Conceptual Design in UML Notation. Rational Rose allows modeling of data-
bases using UML notation. ER diagrams most often used in the conceptual design
of databases can be easily built using the UML notation as class diagrams in
Rational Rose. For example, the ER schema of our COMPANY database from
Figure A.1 at the end of this chapter in Appendix: Figure can be redrawn in Rose
using UML notation as shown in Figure 14. The textual specification in Figure 14
can be converted to the graphical representation shown in Figure 15 by using the
data model diagram option in Rose.
Figure 15 is similar to Figure A.1, except that it is using the notation provided by
Rational Rose. Hence, it can be considered as an ER diagram using UML notation,
with the inclusion of methods and other details. Identifying relationships specify
that an object in a child class (DEPENDENT in Figure 15) cannot exist without a
Figure 14
A logical data model diagram definition in Rational Rose.
345
Practical Database Design Methodology and Use of UML Diagrams
corresponding parent object in the parent class (EMPLOYEE in Figure 15), whereas
non-identifying relationships specify a regular association (relationship) between
two independent classes. It is possible to update the schemas directly in their text or
graphical form. For example, if the relationship between the EMPLOYEE and
PROJECT called WORKS_ON was deleted, Rose would automatically update or
delete all the foreign keys in the relevant tables.
EMPLOYEE
Fname: Char(15)
Minit: Char(1)
Lname: Char(15)
Sex: Char(1)
Salary: Integer
Address: Char(20)
Ssn: Integer
Bdate: Date
Number: Integer
Project_number: Integer
Name: Char(15)
Employee_ssn: Integer
<>PK_T_00()
An important difference in Figure 15 from ER notation is that foreign key attributes
Converting Logical Data Model to Object Model and Vice Versa. Rational
Extensive Domain Support. The Rose Data Modeler allows database designers
SQL) and assign them to any column in the data model. Properties of the domain
Easy Communication among Design Teams. As mentioned earlier, using a
1. As an application involves more and more complexity of data in terms of
model the same information keeps increasing rapidly. It becomes difficult
2. The sheer size of some databases runs into hundreds of entity types and rela-
The above factors have given rise to many tools that come under the general cate-
1. Diagramming. This allows the designer to draw a conceptual schema dia-
2. Model mapping. The mapping is system-specific—most tools generate
3. Design normalization. This utilizes a set of functional dependencies that
We will not survey database design tools here, but only mention the following char-
1. An easy-to-use interface. This is critical because it enables designers to
2. Analytical components. Tools should provide analytical components for
3. Heuristic components. Aspects of the design that cannot be precisely
4. Trade-off analysis. A tool should present the designer with adequate com-
5. Display of design results. Design results, such as schemas, are often dis-
6. Design verification. This is a highly desirable feature. Its purpose is to ver-
Currently there is increasing awareness of the value of design tools, and they are
package must be chosen. We discussed some of the organizational criteria that come
Persistence Inc. PowerTier Mapping from O-O to relational model
Popkin Software Telelogic System Architect Data modeling, object modeling, process
Resolution Ltd. XCase Conceptual modeling up to code maintenance
2. Which of the six phases are considered the main activities of the database
3. Why is it important to design the schemas and applications in parallel?
4. Why is it important to use an implementation-independent data model dur-
5. Discuss the importance of requirements collection and analysis.
6. Consider an actual application of a database system of interest. Define the
7. Discuss the characteristics that a data model for conceptual schema design
8. Compare and contrast the two main approaches to conceptual schema
9. Discuss the strategies for designing a single conceptual schema from its
10. What are the steps of the view integration approach to conceptual schema
11. How would a view integration tool work? Design a sample modular architec-
12. What are the different strategies for view integration?
13. Discuss the factors that influence the choice of a DBMS package for the
14. What is system-independent data model mapping? How is it different from
15. What are the important factors that influence physical database design?
16. Discuss the decisions made during physical database design.
17. Discuss the macro and micro life cycles of an information system.
18. Discuss the guidelines for physical database design in RDBMSs.
19. Discuss the types of modifications that may be applied to the logical data-
20. What functions do the typical database design tools provide?
21. What type of functionality would be desirable in automated tools to support
22. What are the current relational DBMSs that dominate the market? Choose
23. A possible DDL corresponding to the figure above follows:
a. The choice of requiring Name to be NON NULL
24. What naming conventions can you develop to help identify foreign keys
25. What functions do the typical database design tools provide?
detailed discussion of physical design and transaction issues in reference to com-
Navathe and Kerschberg (1986) discuss all phases of database design and point out
Requirements collection and analysis is a heavily researched topic. Chatzoglu et al.
refinement of EER schemas for integration. Castano et al. (1998) present a compre-
Transaction design is a relatively less thoroughly researched topic. Mylopoulos et al.
. . .
In this chapter, we discuss the features of object-oriented data models and show how some of these
2Multimedia databases must store various types of multimedia objects, such as video, audio, images,
1Computer-aided design/computer-aided manufacturing and computer-integrated manufacturing.
From Chapter 11 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
Another reason for the creation of object-oriented databases is the vast increase in
and the ENCORE/ObServer project at Brown University. Commercially available
3Microelectronics and Computer Technology Corporation, Austin, Texas.
a database by making them persistent, and type and class hierarchies and inheritance.
An object typically has two components: state (value) and behavior (operations). It
8Palo Alto Research Center, Palo Alto, California.
sharing among concurrent programs, and recovery from failures. An OO database
The internal structure of an object in OOPLs includes the specification of instance
To encourage encapsulation, an operation is defined in two parts. The first part,
Another OO concept is operator overloading, which refers to an operation’s ability to
applied to. This feature is also called operator polymorphism. For example, an opera-
In the next several sections, we discuss in some detail the main characteristics of
It is inappropriate to base the OID on the physical address of the object in storage,
the efficiency of object retrieval. If the physical address of the object changes, an
1. One type constructor has been called the atom constructor, although this
2. A second type constructor is referred to as the struct (or tuple) constructor., and
3. Collection (or multivalued) type constructors include the set(T), list(T),
The atom constructor is used to represent all basic atomic values, such as integers,, where each aj is an attribute name
The main characteristic of a collection type is that its objects or values will be a
10Also called an instance variable name in OO terminology.
the concepts gradually in this section using a simpler notation. The type construc-
In Figure 1, the attributes that refer to other objects—such as Dept of EMPLOYEE or
Encapsulation of Operations. The concept of encapsulation is one of the main
The external users of the object are only made aware of the interface of the opera-
For database applications, the requirement that all objects be completely encapsu-
The term class is often used to refer to a type definition, along with the definitions of
12This definition of class is similar to how it is used in the popular C++ programming language. The
declared for each class, and the signature (interface) of each operation is included in
An operation is typically applied to an object by using the dot notation. For exam-
referenced by d is destroyed (deleted). The only exception is the constructor opera-
Specifying Object Persistence via Naming and Reachability. An ODBS is
If we first create a named persistent object N, whose state is a set (or possibly a bag)
Notice the difference between traditional database models and ODBs in this respect.
13Default names for the constructor and destructor operations exist in the C++ programming language. instead of
In traditional database models, such as the relational model, all objects are assumed
Simplified Model for Inheritance. Another main characteristic of ODBs is that
15Some systems, such as POET, automatically create the extent for a class.
A type is defined by assigning it a type name, and then defining a number of attrib-
A type in its simplest form has a type name and a list of visible (public) functions.
In the PERSON type, the Name, Address, Ssn, and Birth_date functions can be imple-
The concept of subtype is useful when the designer or user must create a new type
16In this section we will use the terms type and class as meaning the same thing—namely, the attributes
In general, a subtype includes all of the functions that are defined for its supertype
As another example, consider a type that describes objects in plane geometry, which
Notice that type definitions describe objects but do not generate objects on their
Constraints on Extents Corresponding to a Type Hierarchy. In most ODBs,
Classification then proceeds by assigning objects into additional subtypes that are
18This is called OBJECT in the ODMG model (see Section 3).
responding to the class OBJECT, directly or indirectly. In the ODMG model (see
An extent is a named persistent object whose value is a persistent collection that
The inheritance model discussed in this section is very simple. As we will see in
Polymorphism of Operations (Operator Overloading). Another characteris-
In OO databases, a similar situation may occur. We can use the
19In programming languages, there are several kinds of polymorphism. The interested reader is referred
example, by writing a general algorithm to calculate the area of a polygon) and then
Multiple Inheritance and Selective Inheritance. Multiple inheritance occurs
There are several techniques for dealing with ambiguity in multiple inheritance.
Selective inheritance occurs when a subtype inherits only some of the functions of
20In the ODMG model, type inheritance refers to inheritance of operations only, not attributes (see
■ Object identity. Objects have unique identities that are independent of their
■ Type constructors. Complex object structures can be constructed by apply-
■ Encapsulation of operations. Both the object structure and the operations
■ Programming language compatibility. Both persistent and transient
■ Type hierarchies and inheritance. Object types can be specified by using a
■ Extents. All persistent objects of a particular type can be stored in an extent.
■ Polymorphism and operator overloading. Operations and method names
In the following sections we show how these concepts are realized in the SQL stan-
SQL is the standard language for RDBMSs. SQL was first specified by Chamberlin
The following are some of the object database features that have been included in
■ Some type constructors have been added to specify complex objects. These
■ Inheritance mechanisms are provided using the keyword UNDER.
We now discuss each of these concepts in more detail. In our discussion, we will
Figure 4 illustrates some of the object concepts in SQL. We will explain the examples
the concept of ROW TYPE to directly create a structured attribute by using the key-
To allow for collection types in order to create complex-structured objects, four
An array type can have its elements referenced using the common notation of square
In general, the user can specify that system-generated object identifiers for the indi-
For example, in Figure 4(b), we declared a method Age() that calculates the age of
The code for implementing the method still has to be written. We can refer to the
SQL provides certain built-in functions for user-defined types. For a UDT called
In general, a UDT can have a number of user-defined functions associated with it.
Two types of functions can be defined: internal SQL and external. Internal functions
It is also possible to define virtual attributes as part of UDTs, which are computed
■ All attributes are inherited.
hierarchy.
instance is used.
restriction that the signature be the same.
arguments.
Consider the following examples to illustrate type inheritance, which are illustrated
In general, we specify the local attributes and any additional specific methods for
Another facility in SQL is table inheritance via the supertable/subtable facility. This
In SQL, –> is used for dereferencing and has the same meaning assigned to it in the
One of the reasons for the success of commercial relational DBMSs is the SQL stan-
In this section, we describe the ODMG object model and the ODL. In Section 4, we
overview of OQL in Section 5, and the C++ language binding in Section 6.
Objects and Literals. Objects and literals are the basic building blocks of the
An object has five aspects: identifier, name, lifetime, structure, and creation.
1. The object identifier is a unique system-wide identifier (or Object_id).23
Every object must have an object identifier.
2. Some objects may optionally be given a unique name within a particular
21The earlier versions of the object model were published in 1993 and 1997.
3. The lifetime of an object specifies whether it is a persistent object (that is, a
4. The structure of an object specifies how the object is constructed by using
5. Object creation refers to the manner in which an object can be created. This
In the object model, a literal is a value that does not have an object identifier.
1. Atomic literals26 correspond to the values of basic data types and are prede-
2. Structured literals correspond roughly to values that are constructed using
25In the ODMG model, atomic objects do not correspond to objects whose values are basic data types.
3. Collection literals specify a literal value that is a collection of objects or val-, bag, list,, where T is the type of objects or values in the collection.28
Figure 5 gives a simplified view of the basic types and type generators of the object
Figure 5 is a simplified version of the object model. For the full specifications, see
The result returned by this operation is Boolean and would be true if the identity of
28These are similar to the corresponding type constructors described in Section 1.3.
The other inheritance relationship, called EXTENDS inheritance, is specified by the
30The ODMG report also calls interface inheritance as type/subtype, is-a, and generalization/specializa-
be raised by the O.remove_element(E) operation if E is not an element in the collec-
Collection objects are further specialized into set, list, bag, array, and dictionary, which type generator can be that is the union of the type generator allows duplicate elements in.
E = O.retrieve_element_at(I), which simply retrieves the ith element of the array. Any
Figure 6 is a diagram that illustrates the inheritance hierarchy of the built-in con-, whose state would be sets of STUDENT objects. The programmer to manipulate an instance of type. Creating application classes is typically done by utilizing the object
It is important to note that all objects in a particular collection must be of the same
For example, in a UNIVERSITY database application, the user can specify an object
An attribute is a property that describes some aspect of an object. Attributes have
A relationship is a property that specifies that two objects in the database are related.
31As mentioned earlier, this definition of atomic object in the ODMG object model is different from the
By specifying inverses, the database system can maintain the referential integrity of
In addition to attributes and relationships, the designer can include operations in
occur during operation execution. The implementation of the operation will
A class with an extent can have one or more keys. A key consists of one or more
Next, we present the concept of factory object—an object that can be used to gen-
Finally, we discuss the concept of a database. Because an ODBMS can create many
34A composite key is called a compound key in the ODMG report.
Figure 10 shows one possible set of ODL class definitions for the UNIVERSITY data-
35The ODL syntax and data types are meant to be compatible with the Interface Definition language
of STUDENTs. At the same time, individual STUDENT and FACULTY objects will
CIRCLERECTANGLE . . .
to the M:N relationship between STUDENT and SECTION in Figure 9(b). The reason
Hence, the M:N relationship is mapped to the class GRADE, and a pair of 1:N rela-
36We will discuss alternative mappings for attributes of relationships in Section 4.
Because the previous example does not include any interfaces, only classes, we now
Multiple inheritance of interfaces by a class is allowed, as is multiple inheritance of
37This is similar to how an M:N relationship is mapped in the relational model and in the legacy network
another, thus enforcing the ODB equivalent of the relational referential integrity
The third major difference is that in ODB design, it is necessary to specify the oper-
Step 1. Create an ODL class for each EER entity type or subclass. The type of the
Declare an extent for each class, and specify any key attributes as keys of the extent.
Step 2. Add relationship properties or reference attributes for each binary relation-
Depending on the cardinality ratio of the binary relationship, the relationship prop-
If relationship attributes exist, a tuple constructor (struct) can be used to create a, which may be included
38This implicitly uses a tuple constructor at the top level of the type declaration, but in general, the tuple
Step 3. Include appropriate operations for each class. These are not available from
Step 4. An ODL class that corresponds to a subclass in the EER schema inherits (via
Step 5. Weak entity types can be mapped in the same way as regular entity types. An> or list> constructors. The attributes of the weak entity
Step 6. Categories (union types) in an EER schema are difficult to map to ODL. It is
Step 7. An n-ary relationship with degree n > 2 can be mapped into a separate class,
In Section 5.1 we will discuss the syntax of simple OQL queries and the concept of
The basic OQL syntax is a select … from … where … structure, as it is for SQL. For
In general, an entry point to the database is needed for each query, which can be any; PERSONS is of type set; FACULTY is of type; and so on.
Using the example in Q0, there are three syntactic options for specifying iterator
42This is similar to the tuple variables that range over tuples in SQL queries.
The named objects used as database entry points for OQL queries are not limited to
returns a reference to the collection of all persistent DEPARTMENT objects, whose. Similarly, suppose we had given (via the database bind
returns a reference to that individual object of type DEPARTMENT. Once an entry
The first expression Q2 returns an object of type FACULTY, because that is the type
Path expressions Q2 and Q2A return single values, because the attributes Chair (of even when applied to a single object, because that is the type of the
43Note that the latter two options are similar to the syntax for specifying tuple variables in SQL queries.
relationship Has_faculty of the DEPARTMENT class. The collection returned will
In general, an OQL query can return a result with a complex structure specified in
44As mentioned earlier, struct corresponds to the tuple constructor discussed in Section 1.3.
Note that OQL is orthogonal with respect to specifying path expressions. That is,
The order by clause is similar to the corresponding SQL construct, and specifies in
Specifying Views as Named Queries. The view mechanism in OQL uses the
For example, the following view V1 defines a named query Has_minors to retrieve the
Extracting Single Elements from Singleton Collections. An OQL query will,
Since a department name is unique across all departments, the result should be one
Collection Operators (Aggregate Functions, Quantifiers). Because many
The aggregate operators (min, max, count, sum, avg) operate over a collection.45 The
45These correspond to aggregate functions in SQL.
Notice that aggregate operations can be applied to any collection of the appropriate
The membership and quantification expressions return a Boolean type—that is, true
(E in C) returns true if element E is a member of collection C.
Q10 also illustrates a simpler way to specify the select clause of queries that return a.
One can also write queries that return true/false results. As an example, let us
Ordered (Indexed) Collection Expressions. As we discussed in Section 3.3,
Q14 illustrates the use of the first operator on a list collection that contains the
The select-from-order-by query returns a list of Computer Science students ordered
The Grouping Operator. The group by clause in OQL, although similar to the
Q16 retrieves the number of majors in each department. In this query, the students
where F1: E1, F2: E2, …, Fk: Ek is a list of partitioning (grouping) attributes and each
where Ti is the type returned by the expression Ei, partition is a distinguished field
Just as in SQL, a having clause can be used to filter the partitioned sets (that is, select
Note that the select clause of Q17 returns the average GPA of the students in the
returns a bag of student GPAs for that partition. The from clause declares an iterator.
The class library added to C++ for the ODMG standard uses the prefix d_ for class is can refer to both persistent and transient objects of class T.
In order to utilize the various built-in types in the ODMG object model such as col- specifies the operations to be inherited by all objects. specifies the operations of collections., D_List, D_Bag,, and D_Dictionary, and correspond to the collection types in the> whose instances would be sets of references to whose instances would be sets of strings.
46Presumably, d_ stands for database classes.
To specify relationships, the keyword rel_ is used within the prefix of type names; for
in the DEPARTMENT class, we are declaring that Majors_in and Has_majors are rela-
For the OML, the binding overloads the operation new so that it can be used to cre-
48We have only provided a brief overview of the C++ binding. For full details, see Cattell and Barry eds.
We then discussed the ODMG 3.0 standard for object databases. We started by
In Section 5, we presented an overview of the object query language (OQL). The
In 1997 Sun endorsed the ODMG API (Application Program Interface). O2 tech-
2. What primary characteristics should an OID possess?
3. Discuss the various type constructors. How are they used to create complex
4. Discuss the concept of encapsulation, and tell how it is used to create
5. Explain what the following terms mean in object-oriented database termi-
6. What is the relationship between a type and its subtype in a type hierarchy?
7. What is the difference between persistent and transient objects? How is per-
8. How do regular inheritance, multiple inheritance, and selective inheritance
9. Discuss the concept of polymorphism/operator overloading.
10. Discuss how each of the following features is realized in SQL 2008: object
11. In the traditional relational model, creating a table defined both the table
12. Describe the rules of inheritance in SQL 2008.
13. What are the differences and similarities between objects and literals in the
14. List the basic operations of the following built-in interfaces of the ODMG
15. Describe the built-in structured literals of the ODMG object model and the
16. What are the differences and similarities of attribute and relationship prop-
17. What are the differences and similarities of class inhertance via extends and
18. Discuss how persistence is specified in the ODMG object model in the C++
19. Why are the concepts of extents and keys important in database applica-
20. Describe the following OQL concepts: database entry points, path expres-
21. What is meant by the type orthogonality of OQL?
22. Discuss the general principles behind the C++ binding of the ODMG stan-
23. What are the main differences between designing a relational database and
24. Describe the steps of the algorithm for object database design by EER-to-OO
26. Compare inheritance in the EER model to inheritance in the OO model
27. Consider the UNIVERSITY EER schema in Figure A.1. Think of what opera-
28. Consider the COMPANY ER schema in Figure A.2. Think of what operations
29. Design an OO schema for a database application that you are interested in.
30. Consider the AIRPORT database described in Exercise 21 from the chapter
31. Map the COMPANY ER schema in Figure A.1 into ODL classes. Include
32. Using search engines and other sources, determine to what extent the vari-
There is a vast bibliography on OO databases, so we can only provide a representa-
The Object-Oriented Manifesto by Atkinson et al. (1990) is an interesting article
The O2 system is described in Deux et al. (1991), and Bancilhon et al. (1992)
Maier et al. (1986) and Butterworth et al. (1991) describe the design of GEM-
Cattell (1991) surveys concepts from both relational and object databases and dis-
. . .
. . .
Many electronic commerce (e-commerce) andother Internet applications provide Web inter-
From Chapter 12 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
of the Web page. The Web program must first submit a query to the airline database
In this chapter, we will focus on describing the XML data model and its associated
The information stored in databases is known as structured data because it is repre-
However, not all data is collected and inserted into carefully designed structured
Consider the following example. We want to collect a list of bibliographic references
Semistructured data may be displayed as a directed graph, as shown in Figure 1. The
1. The schema information—names of attributes, relationships, and classes
2. In the semistructured model, there is no requirement for a predefined
In addition to structured and semistructured data, a third category exists, known as
HTML uses a large number of predefined tags, which are used to specify a variety of
■ The … tags specify the boundaries of the document.
…
start tag has four attributes describing various character- and start tags have one and twoHTML has a very large number of predefined tags, and whole books are devoted to
1That is why it is known as HyperText Markup Language.
stands for table row and stands for table data.424
XML: Extensible Markup Language
formatted so that humans are able to easily understand the document contents, and
The example in Figure 2 illustrates a static HTML page, since all the information to
2 XML Hierarchical (Tree) Data Model
Attributes in XML provide additional information that describes elements, as we
Figure 3 shows an example of an XML element called . As in HTML, ele-
4SGML (Standard Generalized Markup Language) is a more general language for describing documents
425
XML: Extensible Markup Language
Figure 3.
ProductX 1 Bellaire 5
123456789 Smith 32.5
453453453 Joyce 20.0
ProductY 2 Sugarland 5
123456789 7.5
453453453 20.0
333445555 10.0
…
Complex elements are constructed from other elements hierarchically, whereas
426
XML: Extensible Markup Language
that can be exchanged among multiple users. In HTML, all tag names are prede-
It is straightforward to see the correspondence between the XML textual representa-, , , , , ,, and . The complex elements are the ones with the tag names, , and . In general, there is no limit on the levels of
It is possible to characterize three main types of XML documents:
■ Data-centric XML documents. These documents have many small data
■ Document-centric XML documents. These are documents with large
■ Hybrid XML documents. These documents may have parts that contain
XML documents that do not follow a predefined schema of element names and cor-
XML attributes are generally used in a manner similar to how they are used in
427
XML: Extensible Markup Language
3 XML Documents, DTD, and XML Schema
3.1 Well-Formed and Valid XML Documents
In Figure 3, we saw what a simple XML document may look like. An XML docu-
A well-formed XML document is syntactically correct. This allows it to be processed
A well-formed XML document can be schemaless; that is, it can have any tag names
A stronger criterion is for an XML document to be valid. In this case, the document
428
XML: Extensible Markup Language
Figure 4
When specifying elements, the following notation is used:
■ A * following the element name means that the element can be repeated zero
■ A + following the element name means that the element can be repeated one
■ A ? following the element name means that the element can be repeated zero
■ An element appearing without any of the preceding three symbols must
■ The type of the element is specified via parentheses following the element. If
■ The list of attributes that can appear within an element can also be specified
■ Parentheses can be nested when specifying elements.
ment.
>
] >
429
XML: Extensible Markup Language
We can see that the tree structure in Figure 1 and the XML document in Figure 3
When the value of the standalone attribute in an XML document is “no”, the docu-
Although XML DTD is quite adequate for specifying tree structures with required,
3.2 XML Schema
As with XML DTD, XML schema is based on the tree data model, with elements and
430
XML: Extensible Markup Language
Figure 5
Company Schema (Element Approach) – Prepared by Babak
Hojabri
431
XML: Extensible Markup Language
(continues)
432
Figure 5 (continued)
XML: Extensible Markup Language
1. Schema descriptions and XML namespaces. It is necessary to identify the
433
XML: Extensible Markup Language
the file used in this example, which is http://www.w3.org/2001/XMLSchema.
2. Annotations, documentation, and language used. The next couple of lines
3. Elements and types. Next, we specify the root element of our XML schema.
4. First-level elements in the COMPANY database. Next, we specify the three
5. Specifying element type and minimum and maximum occurrences. In
434
document that conforms to the schema specifications. If they are not speci-
6. Specifying keys. In XML schema, it is possible to specify constraints that
7. Specifying the structures of complex elements via complex types. The next
8. Composite (compound) attributes. Composite attributes from Figure A.3
This example illustrates some of the main features of XML schema. There are other
XML: Extensible Markup Language
435
XML: Extensible Markup Language
4 Storing and Extracting XML Documents from
Several approaches to organizing the contents of XML documents to facilitate their
1. Using a DBMS to store the documents as text. A relational or object DBMS
2. Using a DBMS to store the document contents as data elements. This
3. Designing a specialized system for storing native XML data. A new type of
4. Creating or publishing customized XML documents from preexisting
All of these approaches have received considerable attention. We focus on the fourth
436
based on flat files (relational model) and graph representations (ER model). But
5 XML Languages
5.1 XPath: Specifying Path Expressions in XML
The first XPath expression in Figure 6 returns the company root node and all its
www.company.com/info.XML
then the first XPath expression in Figure 6 can be written as
doc(www.company.com/info.XML)/company
This prefix would also be included in the other examples of XPath expressions.
XML: Extensible Markup Language
Figure 6
1. /company
2. /company/department
3. //employee [employeeSalary gt 70000]/employeeName
4. /company/employee [employeeSalary gt 70000]/employeeName
5. /company/project/projectWorker [hours ge 20.0]
437
XML: Extensible Markup Language
The second example in Figure 6 returns all department nodes (elements) and their
The third XPath expression in Figure 6 illustrates the use of //, which is conve-nient
The expression returns all employeeName nodes that are direct children of an
The fourth XPath expression in Figure 6 should return the same result as the previ-
When we need to include attributes in an XPath expression, the attribute name is
/company/*
The examples above illustrate simple XPath expressions, where we can only move
The main restriction of XPath path expressions is that the path that specifies the pat-
6We use the terms node, tag, and element interchangeably here.
438
retrieved. The XQuery language separates these two concerns, and provides more
5.2 XQuery: Specifying Queries in XML
FOR
There can be zero or more instances of the FOR clause, as well as of the LET clause in
LET $d := doc(www.company.com/info.xml)
$y IN $d/company/employee $y/employeeName/firstName, $y/employeeName/lastName,
$x/hours
1. Variables are prefixed with the $ sign. In the above example, $d, $x, and $y
2. The LET clause assigns a variable to a particular expression for the rest of the
3. The FOR clause assigns a variable to range over each of the individual items
4. The WHERE clause specifies additional conditions on the selection of items.
5. Finally, the RETURN clause specifies which elements or attributes should be
XML: Extensible Markup Language
439
XML: Extensible Markup Language
will return a sequence of elements each containing for employees who work more that 20 hours per week on project
Figure 7 includes some additional examples of queries in XQuery that can be speci-
The third query illustrates how a join operation can be performed by using more
XQuery has very powerful constructs to specify complex queries. In particular, it can
This concludes our brief introduction to XQuery. The interested reader is referred to
Figure 7
1. FOR $x IN $x/firstName, $x/lastName
2. FOR $x IN $x/employeeName/firstName, $x/employeeName/lastName
3. FOR $x IN $y/employeeName/firstName, $y/employeeName/lastName, $x/hours
440
XML: Extensible Markup Language
5.3 Other Languages and Protocols Related to XML
■ The Extensible Stylesheet Language (XSL) can be used to define how a doc-
■ The Extensible Stylesheet Language for Transformations (XSLT) can be used
■ The Web Services Description Language (WSDL) allows for the description
■ The Simple Object Access Protocol (SOAP) is a platform-independent and
■ The Resource Description Framework (RDF) provides languages and tools
6 Extracting XML Documents from Relational
6.1 Creating Hierarchical XML Views over Flat
This section discusses the representational issues that arise when converting data
We will use the simplified UNIVERSITY ER schema shown in Figure 8 to illustrate our
441
XML: Extensible Markup Language
Name
S-D
Students
Courses
Instructors
Major dept Department
1 1
1
1
N
N
S-S C-S S-1
D-1
D-C
DEPARTMENT
COURSE
SECTION
Name
Ssn
N
M 1
Class
YearNumber Qtr
Grade
STUDENT
Sections
Sections taught
N N
NameSsn
Number
Rank
SalaryINSTRUCTOR
Department
Course
Sections
Figure 8
types COURSE, SECTION, and STUDENT from Figure 8, and the relationships
At least three possible document hierarchies can be extracted from the database
442
XML: Extensible Markup Language
1
Number
Sections
Name
1
Number
Students
Qtr
YearSECTION
N
N
Name
Ssn
Grade
Class
Figure 10
Figure 11
S-D
Ssn
Name
Class
STUDENT
M N N 1
Number
Year Qtr
SECTION
Number
Name
COURSES-D
Course Sections
Grade
Figure 9
443
XML: Extensible Markup Language
1
Ssn
Sections
NameSTUDENT
1
Number
Qtr
Year
1
N
Grade
Class
COURSE
Course_number
Course_name
Figure 12
Figure 11 (continued)
In the second hierarchical document view, we can choose STUDENT as root (Figure 12). In this hierarchi-
444
XML: Extensible Markup Language
Figure 13
1
Ssn
Name
1
Number
Qtr
Year
1
N
Grade
Class
COURSE
Course_number
Course_name
Figure 14
The third possible way is to choose SECTION as the root, as shown in Figure 14.
445
XML: Extensible Markup Language
COURSE
INSTRUCTOR
1 1 N N
1 1NN
(a) (b)
STUDENT
DEPARTMENTSECTION COURSE
INSTRUCTOR INSTRUCTOR1
STUDENT
DEPARTMENTSECTION
(c)
STUDENT
DEPARTMENTSECTION
INSTRUCTOR COURSE1 INSTRUCTOR1 COURSE
1
N
Figure 15
6.2 Breaking Cycles to Convert Graphs into Trees
Suppose that we need the information in all the entity types and relationships in
In Figure 15(c), we have converted the initial graph to a hierarchy. We can do further
446
XML: Extensible Markup Language
6.3 Other Steps for Extracting XML Documents
In addition to creating the appropriate XML hierarchy and corresponding XML
1. It is necessary to create the correct query in SQL to extract the desired infor-
2. Once the query is executed, its result must be restructured from the flat rela-
3. The query can be customized to select either a single object or multiple
7 Summary
Review Questions
tured data?
2. Under which of the categories in 1 do XML documents fall? What about self-
447
3. What are the differences between the use of tags in XML versus HTML?
4. What is the difference between data-centric and document-centric XML
5. What is the difference between attributes and elements in XML? List some of
6. What is the difference between XML schema and XML DTD?
Exercises
in the relational database shown in Figure A.1 such that the XML document
8. Create XML schema documents and XML DTDs to correspond to the hier-
9. Consider the LIBRARY relational database schema in Figure A.5. Create an
10. Specify the following views as queries in XQuery on the company XML
a. A view that has the department name, manager name, and manager
b. A view that has the employee name, supervisor name, and employee
c. A view that has the project name, controlling department name, number
d. A view that has the project name, controlling department name, number
Selected Bibliography
XML: Extensible Markup Language
448
DEPT_LOCATIONS
Dnumber
Houston
Stafford
Bellaire
Sugarland
Dlocation
DEPARTMENT
Dname
Research
Administration
Headquarters 1
5
4
888665555
333445555
987654321
1981-06-19
1988-05-22
1995-01-01
Dnumber Mgr_ssn Mgr_start_date
WORKS_ON
Essn
123456789
123456789
666884444
453453453
453453453
333445555
333445555
333445555
333445555
999887777
999887777
987987987
987987987
987654321
987654321
888665555
3
1
2
2
1
2
30
30
30
10
10
3
10
20
20
20
40.0
32.5
7.5
10.0
10.0
10.0
10.0
20.0
20.0
30.0
5.0
10.0
35.0
20.0
15.0
NULL
Pno Hours
PROJECT
Pname
ProductX
ProductY
ProductZ
Computerization
Reorganization
Newbenefits
3
1
2
30
10
20
5
5
5
4
4
1
Houston
Bellaire
Sugarland
Stafford
Stafford
Houston
Pnumber Plocation Dnum
DEPENDENT
333445555
333445555
333445555
987654321
123456789
123456789
123456789
Joy
Alice F
M
F
M
M
F
F
1986-04-05
1983-10-25
1958-05-03
1942-02-28
1988-01-04
1988-12-30
1967-05-05
Theodore
Alice
Elizabeth
Abner
Michael
Spouse
Daughter
Son
Daughter
Spouse
Spouse
Son
Dependent_name Sex Bdate Relationship
EMPLOYEE
Fname
John
Franklin
Jennifer
Alicia
Ramesh
Joyce
James
Ahmad
Narayan
English
Borg
Jabbar
666884444
453453453
888665555
987987987
F
F
M
M
M
M
M
F
4
4
5
5
4
1
5
5
25000
43000
30000
40000
25000
55000
38000
25000
987654321
888665555
333445555
888665555
987654321
NULL
333445555
333445555
Zelaya
Wallace
Smith
Wong
3321 Castle, Spring, TX
291 Berry, Bellaire, TX
731 Fondren, Houston, TX
638 Voss, Houston, TX
1968-01-19
1941-06-20
1965-01-09
1955-12-08
1969-03-29
1937-11-10
1962-09-15
1972-07-31
980 Dallas, Houston, TX
450 Stone, Houston, TX
975 Fire Oak, Humble, TX
5631 Rice, Houston, TX
999887777
987654321
123456789
333445555
Minit Lname Ssn Bdate Address Sex DnoSalary Super_ssn
B
T
J
S
K
A
V
E
Houston
1
4
5
5
Essn
5
Figure A.1
449
XML: Extensible Markup Language
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.2
450
XML: Extensible Markup Language
EMPLOYEE
Fname Minit Lname
Name Address
Sex
Salary
Ssn
Bdate
Supervisor Supervisee
SUPERVISION1 N
Hours
WORKS_ON
CONTROLS
M N
1
DEPENDENTS_OF
Name
Location
N
1
PROJECT
DEPARTMENT
Locations
Name Number
Number
Number_of_employees
MANAGES
Start_date
WORKS_FOR
N
DEPENDENT
Sex Birth_date RelationshipName
Figure A.3
451
XML: Extensible Markup Language
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.4
452
XML: Extensible Markup Language
Publisher_nameBook_id Title
BOOK
BOOK_COPIES
BOOK_AUTHORS
Book_id Author_name
LIBRARY_BRANCH
PUBLISHER
Name Address Phone
BOOK_LOANS
Book_id Branch_id Card_no Date_out Due_date
BORROWER
Figure A.5
453
Introduction to SQL
In this chapter, we discuss some of the methods thathave been developed for accessing databases from
It is important to note that database programming is a very broad topic. There are
From Chapter 13 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
454
Introduction to SQL Programming Techniques
and changes to existing techniques are incorporated into newer system versions and
We start our presentation of database programming in Section 1 with an overview
1 Database Programming: Techniques
We now turn our attention to the techniques that have been developed for accessing
455
Introduction to SQL Programming Techniques
type SQL commands or queries directly over several lines, ended by a semicolon and“). Alternatively, a file of commands can be created. The system
The interactive interface is quite convenient for schema and constraint creation or
In this section, first we give an overview of the main approaches to database pro-
1.1 Approaches to Database Programming
1. Embedding database commands in a general-purpose programming lan-
2. Using a library of database functions. A library of functions is made avail-
3. Designing a brand-new language. A database programming language is
1Other prefixes are sometimes used, but this is the most common.
456
Introduction to SQL Programming Techniques
statements are added to the database language to convert it into a full-
In practice, the first two approaches are more common, since many applications are
1.2 Impedance Mismatch
Another problem occurs because the results of most queries are sets or multisets of
Impedance mismatch is less of a problem when a special database programming
457
Introduction to SQL Programming Techniques
1.3 Typical Sequence of Interaction
When a programmer or software engineer writes a program that requires access to a
1. When the client program requires access to a particular database, the pro-
2. Once the connection is established, the program can interact with the data-
3. When the program no longer needs access to a particular database, it should
A program can access multiple databases if needed. In some database programming
In the next three sections, we discuss examples of each of the three main approaches
2 Embedded SQL, Dynamic SQL, and SQLJ
2There are two-tier and three-tier architectures; to keep our discussion simple, we will assume a two-tier
458
Introduction to SQL Programming Techniques
SQL, are presented in Sections 2.1 through 2.3, and can be adapted to other pro-
2.1 Retrieving Single Tuples with Embedded SQL
Within an embedded SQL command, we may refer to specially declared C program
Suppose that we want to write C programs to process the COMPANY database in
4Our discussion here also applies to the C++ programming language, since we do not use any of the
459
Introduction to SQL Programming Techniques
Figure 1
0) int loop ;
because strings in C are terminated by a NULL character (\0), which is not part of
Notice that the only embedded SQL commands in Figure 1 are lines 1 and 7, which
Connecting to the Database. The SQL command for establishing a connection
CONNECT TO AS ;
In general, since a user or program can access several database servers, several con- to change from the
SET CONNECTION ;
Once a connection is no longer needed, it can be terminated by the following com-
DISCONNECT ;
In the examples in this chapter, we assume that the appropriate connection has
6SQL strings can also be mapped to char* types in C.
460
Introduction to SQL Programming Techniques
Communicating between the Program and the DBMS Using SQLCODE
462
Introduction to SQL Programming Techniques
CLOSE CURSOR command is issued to indicate that we are done with processing
An example of using cursors to process a query result with multiple records is command executes the query and fetches its result as a table into commands (line 9). We assume that
This example also illustrates how the programmer can update database records.
Figure 3
//Program Segment E2:
463
Introduction to SQL Programming Techniques
the clause FOR UPDATE OF in the cursor declaration and list the names of any specifies that the cur-
Notice that declaring a cursor and associating it with a query (lines 4 through 7 in command (line 8) is executed. Also notice that there is no need to include
General Options for a Cursor Declaration. Several options can be specified
DECLARE [ INSENSITIVE ] [ SCROLL ] CURSOR ] ] ] ;
We already briefly discussed the options listed in the last line. The default is that the and list
When the optional keyword SCROLL is specified in a cursor declaration, it is possi-
FETCH [ [ ] FROM ] INTO ;
The ORDER BY clause orders the tuples so that the FETCH command will fetch them
464
Introduction to SQL Programming Techniques
Figure 4
//Program Segment E3:
…
…
2.3 Specifying Queries at Runtime Using Dynamic SQL
Program segment E3 in Figure 4 reads a string that is input by the user (that string
Although including a dynamic update command is relatively straightforward in
In E3, the reason for separating PREPARE and EXECUTE is that if the command is to
465
Introduction to SQL Programming Techniques
well as generating the code for executing it. It is possible to combine the PREPARE
EXEC SQL EXECUTE IMMEDIATE :sqlupdatestring ;
This is useful if the command is to be executed only once. Alternatively, the pro-
2.4 SQLJ: Embedding SQL Commands in Java
Before being able to process SQLJ with Java in Oracle, it is necessary to import sev-
Figure 5
1) import java.sql.* ;
…“, ““, ““, true) ;
…
8This section assumes familiarity with object-oriented concepts and basic JAVA concepts.
466
Introduction to SQL Programming Techniques
Figure 6
1) string dname, ssn , fname, fn, lname, ln,
2) char sex, minit, mi ;
5. The format of this function call, which returns an object of type default context,10
is as follows:
public static DefaultContext
For example, we can write the statements in lines 6 through 8 in Figure 5 to connect using the login of and with automatic commitment of each command,11 and
In the following examples, we will not show complete Java classes or programs since
Notice that because Java already uses the concept of exceptions for error handling,
try {} catch () {}
is used to deal with exceptions that occur during the execution of . If is processed directly. Exceptions
10A default context, when set, applies to subsequent commands in the program until it is changed.
467
Introduction to SQL Programming Techniques
Figure 7
//Program Segment J1:
+ ” ” + salary)
that can be thrown by the code in a particular operation should be specified as part
()
In SQLJ, the embedded SQL commands within a Java program are preceded by
In J1 a single tuple is retrieved by the embedded SQLJ query; that is why we are able
2.5 Retrieving Multiple Tuples in SQLJ Using Iterators
1. A named iterator is associated with a query result by listing the attribute
2. A positional iterator lists only the attribute types that appear in the query
12We will not discuss iterators in more detail here.
468
Introduction to SQL Programming Techniques
Figure 8
//Program Segment J2A:
double salary) ;
e.lname + ” ” + e.salary) ;
In both cases, the list should be in the same order as the attributes that are listed in
When the iterator object is associated with a query (lines 11 and 12 in Figure 8), the
469
Introduction to SQL Programming Techniques
In Figure 8, the command (e.next()) in line 13 performs two functions: It gets
Next, consider the same example using positional iterators as shown in Figure 9
The positional iterator behaves in a manner that is more similar to embedded SQL INTO com-
Figure 9
//Program Segment J2B:
470
Introduction to SQL Programming Techniques
a fetch command returns a valid tuple from the query result. It is set to TRUE again
3 Database Programming with Function Calls:
Embedded SQL (see Section 2) is sometimes referred to as a static database pro-
In this section, we give an overview of two function call interfaces. We first discuss
The main advantage of using a function call interface is that it makes it easier to
3.1 Database Programming with SQL/CLI Using C
Before using the function calls in SQL/CLI, it is necessary to install the appropriate
13Our discussion here also applies to the C++ programming language, since we do not use any of the
471
Introduction to SQL Programming Techniques
When using SQL/CLI, the SQL statements are dynamically created and passed as
Each record is accessible to the program through a C pointer variable—called a
SQLAllocHandle(, , )
Figure 10
//Program CLI1:
SQL_NTS) else exit ;
SQL_NTS) ;
472
Introduction to SQL Programming Techniques
In this function, the parameters are as follows:
■ indicates the type of record being created. The possible val-
■ indicates the container within which the new handle is being
■ is the pointer (handle) to the newly created record of type.
When writing a C program that will include database calls through SQL/CLI, the
1. The library of functions comprising SQL/CLI must be included in the C pro-
2. Declare handle variables of types SQLHSTMT, SQLHDBC, SQLHENV, and
3. An environment record must be set up in the program using is the NULL handle SQL_NULL_HANDLE (NULL pointer) when
4. A connection record is set up in the program using SQLAllocHandle. In line
5. A statement record is set up in the program using SQLAllocHandle. In line
6. The statement is prepared using the SQL/CLI function SQLPrepare. In line
14To keep our presentation simple, we will not show description records here.
473
Introduction to SQL Programming Techniques
statement handle stmt1. The question mark (?) symbol in line 10 represents
7. Before executing the query, any parameters in the query string should be
8. Following these preparations, we can now execute the SQL statement refer-
9. In order to determine where the result of the query is returned, one common
10. Finally, in order to retrieve the column values into the C program variables,
15An alternative technique known as unbound columns uses different SQL/CLI functions, namely
474
Introduction to SQL Programming Techniques
Figure 11
//Program Segment CLI2:
SQL_NTS) else exit ;
SQL_NTS) ;
As we can see, using dynamic function calls requires a lot of preparation to set up
In CLI1 a single tuple is selected by the SQL query. Figure 11 shows an example of
475
Introduction to SQL Programming Techniques
3.2 JDBC: SQL Function Calls for Java Programming
Because Java is object-oriented, its function libraries are implemented as classes.
JDBC is designed to allow a single Java program to connect to several different data-
To load a JDBC driver explicitly, the generic Java function for loading a class can be
Class.forName(“oracle.jdbc.driver.OracleDriver”)
This will register the driver with the driver manager and make it available to the
-Djdbc.drivers = oracle.jdbc.driver
17This section assumes familiarity with object-oriented concepts and basic Java concepts.
476
Introduction to SQL Programming Techniques
Figure 12
//Program JDBC1:
…
The following are typical steps that are taken when writing a Java application pro-
1. The JDBC library of classes must be imported into the Java program. These
2. Load the JDBC driver as discussed previously (lines 4 to 7). The Java excep-
3. Create appropriate variables as needed in the Java program (lines 8 and 9).
477
Introduction to SQL Programming Techniques
4. The Connection object. A connection object is created using the
jdbc:oracle::/
An alternative form is
getConnection(url, dbaccount, password)
Various properties can be set for a connection object, but they are mainly
5. The Statement object. A statement object is created in the program. In
Figure 13
//Program Segment JDBC2:
…
throws SQLException, IOException {
dno.tostring() ;
478
Introduction to SQL Programming Techniques
objects. In line 14 in Figure 12, a query string with a single parameter—indi-
6. Setting the statement parameters. The question mark (?) symbol in line 14
7. Before executing a PreparedStatement query, any parameters should be
8. Following these preparations, we can now execute the SQL statement refer-
9. The ResultSet object. In line 19, the result of the query is returned in an
479
Introduction to SQL Programming Techniques
(“Lname”, “Salary”) with the get… functions. In our examples, we used
In general, the programmer can check for SQL exceptions after each JDBC function
Notice that JDBC does not distinguish between queries that return single tuples and
In example JDBC1, a single tuple is selected by the SQL query, so the loop in lines 20
This concludes our brief introduction to JDBC. The interested reader is referred to
4 Database Stored Procedures
This section introduces two additional topics related to database programming. In
4.1 Database Stored Procedures and Functions
480
Introduction to SQL Programming Techniques
main part of the DBMS software package—is located. Although this is suitable for
Stored procedures are useful in the following circumstances:
■ If a database program is needed by several applications, it can be stored at
■ Executing a program at the server can reduce data transfer and communica-
■ These procedures can enhance the modeling power provided by views by
In general, many commercial DBMSs allow stored procedures and functions to be
CREATE PROCEDURE ()
;
The parameters and local declarations are optional, and are specified only if needed.
CREATE FUNCTION () ;
If the procedure (or function) is written in a general-purpose programming lan-
CREATE PROCEDURE ()
EXTERNAL NAME ;
In general, each parameter should have a parameter type that is one of the SQL data
481
Introduction to SQL Programming Techniques
Because the procedures and functions are stored persistently by the DBMS, it
CALL () ;
If this statement is called from JDBC, it should be assigned to a statement object of
4.2 SQL/PSM: Extending SQL for Specifying Persistent Stored
SQL/PSM is the part of the SQL standard that specifies how to write persistent
In this section, we discuss the SQL/PSM constructs for conditional (branching)
The conditional branching statement in SQL/PSM has the following form:
IF THEN THEN THEN
END IF ;
Consider the example in Figure 14, which illustrates how the conditional branch
SQL/PSM has several constructs for looping. There are standard while and repeat
20We only give a brief introduction to SQL/PSM here. There are many other features in the SQL/PSM
482
Introduction to SQL Programming Techniques
Figure 14
//Function PSM1:
WHILE DO
END WHILE ;
There is also a cursor-based looping structure. The statement list in such a loop is
FOR AS CURSOR FOR DO
END FOR ;
Loops can have names, and there is a LEAVE statement to break a loop
5 Comparing the Three Approaches
1. Embedded SQL Approach. The main advantage of this approach is that the
483
Introduction to SQL Programming Techniques
queries have to be generated at runtime, the function call approach will be
2. Library of Function Calls Approach. This approach provides more flexibil-
3. Database Programming Language Approach. This approach does not suf-
6 Summary
In Section 2, we discussed the general technique known as embedded SQL, where
In Section 3, we discussed how function call libraries can be used to access SQL
484
Introduction to SQL Programming Techniques
by the next() operation. The get and set functions for retrieving attribute values
In Section 4 we gave a brief overview of stored procedures, and discussed SQL/PSM
Review Questions
2. What is JDBC? Is it an example of embedded SQL or of using function calls?
3. List the three main approaches to database programming. What are the
4. What is the impedance mismatch problem? Which of the three program-
5. Describe the concept of a cursor and how it is used in embedded SQL.
6. What is SQLJ used for? Describe the two types of iterators available in SQLJ.
Exercises
Figure A.3. Write a program segment to read a student’s name and print his
8. Repeat Exercise 7, but use SQLJ with Java as the host language.
9. Consider the library relational database schema in Figure A.4. Write a pro-
10. Repeat Exercise 9, but use SQLJ with Java as the host language.
11. Repeat Exercises 7 and 9, but use SQL/CLI with C as the host language.
12. Repeat Exercises 7 and 9, but use JDBC with Java as the host language.
13. Repeat Exercise 7, but write a function in SQL/PSM.
14. Create a function in PSM that computes the median salary for the
485
Introduction to SQL Programming Techniques
Selected Bibliography
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.1
486
Introduction to SQL Programming Techniques
Name Student_number Class Major
Smith 17 1 CS
Brown 8 2 CS
STUDENT
Course_name Course_number Credit_hours Department
Intro to Computer Science CS1310 4 CS
Data Structures CS3320 4 CS
Discrete Mathematics MATH2410 3 MATH
Database CS3380 3 CS
COURSE
Section_identifier Course_number Semester Year Instructor
85 MATH2410 Fall 07 King
92 CS1310 Fall 07 Anderson
102 CS3320 Spring 08 Knuth
112 MATH2410 Fall 08 Chang
119 CS1310 Fall 08 Anderson
135 CS3380 Fall 08 Stone
SECTION
Student_number Section_identifier Grade
17 112 B
17 119 C
8 85 A
8 92 A
8 102 B
8 135 A
GRADE_REPORT
Course_number Prerequisite_number
CS3380 CS3320
CS3380 MATH2410
CS3320 CS1310
PREREQUISITE
Figure A.2
487
Introduction to SQL Programming Techniques
Section_identifier SemesterCourse_number InstructorYear
SECTION
Course_name Course_number Credit_hours Department
COURSE
Name Student_number Class Major
STUDENT
Course_number Prerequisite_number
Student_number GradeSection_identifier
GRADE_REPORT
Figure A.3
488
Introduction to SQL Programming Techniques
Publisher_nameBook_id Title
BOOK
BOOK_COPIES
BOOK_AUTHORS
Book_id Author_name
LIBRARY_BRANCH
PUBLISHER
Name Address Phone
BOOK_LOANS
Book_id Branch_id Card_no Date_out Due_date
BORROWER
Figure A.4
489
Web Database
In this chapter, we direct our attention to how data-bases are accessed from scripting languages. Many
Basic HTML is useful for generating static Web pages with fixed text and other
There are various techniques for programming dynamic features into Web pages.
From Chapter 14 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
490
Web Database Programming Using PHP
available on most computer platforms. A PHP interpreter provides a Hypertext
This chapter is organized as follows. Section 1 gives a simple example to illustrate
1 A Simple PHP Example
PHP has libraries of functions for accessing databases stored under various types of
Consider the example shown in Figure 1(a), which prompts a user to enter the first
1. Suppose that the file containing PHP script in program segment P1 is stored
2. Line 0 shows the PHP start tag , shown on line 16. Text outside of these tags is
491
Enter your name:
SUBMIT NAME SUBMIT NAME
Enter your name:
(d)
John Smith
Welcome, John Smith
Web Database Programming Using PHP
Figure 1
(a)
0)
10) Enter your name:
printed as is. This allows PHP code segments to be included within a larger
3. Line 1 shows one way of posting comments in a PHP program on a single
4. The auto-global predefined PHP variable $_POST (line 2) is an array that
492
Web Database Programming Using PHP
can be associative arrays whose indexes can be any string values. For example,
5. When the Web page at http://www.myserver.com/example/greeting.php is
6. Line 8 shows one way of creating long text strings in an HTML file. We will
10) Enter your name:
494
Web Database Programming Using PHP
3. Here documents. Enclose a part of a document between a << (greater than), >= (greater than or equal), < (less than), and <= (less than
or equal).
2.2 PHP Arrays
Arrays are very important in PHP, since they allow lists of elements. They are used
frequently in forms that employ pull-down menus. A single-dimensional array is
used to hold the list of choices in the pull-down menu. For database query results,
two-dimensional arrays are used with the first dimension representing rows of a
table and the second dimension representing columns (attributes) within a row.
495
Web Database Programming Using PHP
Figure 3
Illustrating basic PHP array processing.
0) $teaching = array('Database' => ‘Smith’, ‘OS’ => ‘Carrick’,
1) $teaching[‘Graphics’] = ‘Benson’; $teaching[‘Data Mining’] = ‘Kam’;
‘;Course $i is $course[$i] \n”;
10) }
There are two main types of arrays: numeric and associative. We discuss each of
A numeric array associates a numeric index (or position or sequence number) with
Figure 3 gives two examples of array variables: $teaching and $courses. The first
If we only provide values (no keys) as array elements, the keys are automatically
There are several different techniques for looping through arrays in PHP. We illus-
496
Web Database Programming Using PHP
key and value of each element on a separate line. Lines 7 through 10 show how a tra-
The example in lines 7 through 10 also illustrates how an HTML table can be dis-
(table row) tag.
The count function (line 7) returns the current number of elements in the array.
2.3 PHP Functions
We now give two simple examples to illustrate basic PHP functions. In Figure 4, we
A second example is shown in Figure 5. Here we are using the $teaching array
The function call in line 11 would return the string: Smith is teaching Database,
497
Web Database Programming Using PHP
Figure 4
//Program Segment P1′:
Figure 5
0) function course_instructor ($course, $teaching_assignments) {
‘Graphics’ => ‘Kam’);
498
Web Database Programming Using PHP
‘Computer Architecture’. A few comments about this example and about PHP func-
■ The built-in PHP array function array_key_exists($k, $a) returns true
■ Function arguments are passed by value. Hence, in this example, the calls in
■ Return values of a function are placed after the RETURN keyword. A function
■ Scope rules for variable names apply as in other programming languages.
The previous discussion gives a brief overview of PHP functions. Many details are
2.4 PHP Server Variables and Forms
1. $_SERVER[‘SERVER_NAME’]. This provides the Web site name of the server
2. $_SERVER[‘REMOTE_ADDRESS’]. This is the IP (Internet Protocol) address
3. $_SERVER[‘REMOTE_HOST’]. This is the Web site name of the client user
4. $_SERVER[‘PATH_INFO’]. This is the part of the URL address that comes
499
Web Database Programming Using PHP
5. $_SERVER[‘QUERY_STRING’]. This provides the string that holds parame-
6. $_SERVER[‘DOCUMENT_ROOT’]. This is the root directory that holds the
These and other entries in the $_SERVER array are usually needed when creating the
Another important PHP auto-global built-in array variable is called $_POST. This
3 Overview of PHP Database Programming
There is a PHP database access function library that is part of PHP Extension and
We will discuss several functions that are part of PEAR DB in the context of some
3.1 Connecting to a Database. The func-
500
Web Database Programming Using PHP
Figure 6
0) require ‘DB.php’;
…
$q->getMessage()); }
9) $d->setErrorHandling(PEAR_ERROR_DIE);
10) $eid = $d->nextID(‘EMPLOYEE’);
…
string argument specifies the database information. The format for ‘string’ is:
://:@
In Figure 6, line 1 connects to the database that is stored using Oracle (specified via portion of the ‘string’ specifies the par-
■ MySQL. Specified as mysql for earlier versions and mysqli for later versions
■ Oracle. Specified as oc8i for versions 7, 8, and 9. This is used in line 1 of
■ SQLite. Specified as sqlite.
The above is not a comprehensive list.
501
Web Database Programming Using PHP
Following the in the string argument passed to DB::connect is followed by. These are followed by the where the
In line 1 of Figure 6, the user is connecting to the server at www.host.com/db1 using
Line 2 in Figure 6 shows how to check whether the connection to the database was
In general, most SQL commands can be sent to the database once a connection is
The PHP PEAR DB library offers an alternative to having to check for errors after
$d–>setErrorHandling(PEAR_ERROR_DIE)
will terminate the program and print the default error messages if any subsequent
3.2 Collecting Data from Forms and Inserting Records
Lines 10 through 12 in Figure 6 illustrate how this may be done. In this example, we
502
Web Database Programming Using PHP
input parameters called emp_name, emp_job, and emp_dno. These would be acces-
In the SQL INSERT command shown on lines 11 and 12 in Figure 6, the array
This example also illustrates another feature of PEAR DB. It is common in some
PHP has a function $d–>nextID to create a sequence of unique values for a partic-
The code for insert in lines 10 to 12 in Figure 6 may allow malicious strings to be
3.3 Retrieval Queries from Database Tables
The second query example is shown in lines 8 to 13 and illustrates a dynamic query.
1This would be similar to system-generated OID for object and object-relational database systems.
503
Web Database Programming Using PHP
Figure 7
0) require ‘DB.php’;
…
…
10) print “employees in dept $_POST[’emp_dno’] whose job is
11) while ($r = $q->fetchRow()) {
…
…
$POST[’emp_dno’]. If the user had entered ‘Engineer’ for the job and 5 for the
The last query (lines 14 to 17) shows an alternative way of specifying a query and
As we can see, PHP is suited for both database access and creating dynamic Web
2The $r variable is similar to the cursors and iterator variables.
504
Web Database Programming Using PHP
4 Summary
We gave only a very basic introduction to PHP. There are many books as well as
Review Questions
Where in the three-tier architecture does a PHP program execute? Where
2. What type of programming language is PHP?
3. Discuss the different ways of specifying strings in PHP.
4. Discuss the different types of arrays in PHP.
5. What are PHP auto-global variables? Give some examples of PHP auto-
6. What is PEAR? What is PEAR DB?
7. Discuss the main functions for accessing a database in PEAR DB, and how
8. Discuss the different ways for looping over a query result in PHP.
9. What are placeholders? How are they used in PHP database programming?
505
Web Database Programming Using PHP
Exercises
code to create the tables of this schema.
11. Write a PHP program that creates Web forms for entering the information
12. Write PHP Web interfaces for the queries specified in Exercise 18 from the
Selected Bibliography
506
Publisher_nameBook_id Title
BOOK
BOOK_COPIES
BOOK_AUTHORS
Book_id Author_name
LIBRARY_BRANCH
PUBLISHER
Name Address Phone
BOOK_LOANS
Book_id Branch_id Card_no Date_out Due_date
BORROWER
Figure A.1
507
Basics of Functional
for Relational Databases
Each relation schema consists of a number of attrib-utes, and the relational database schema consists of
There are two levels at which we can discuss the goodness of relation schemas. The
From Chapter 15 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
508
Basics of Functional Dependencies and Normalization for Relational Databases
physical storage) level—how the tuples in a base relation are stored and updated.
As with many design problems, database design may be performed using two
Relational database design ultimately produces a set of relations. The implicit goals
We start this chapter by informally discussing some criteria for good and bad rela-
1An exception in which this approach is used in practice is based on a model called the binary relational
509
Basics of Functional Dependencies and Normalization for Relational Databases
more general definitions of normal forms that can be directly applied to any given
Further study should continue the development of the theory related to the design
1 Informal Design Guidelines
Before discussing the formal theory of relational database design, we discuss four
■ Making sure that the semantics of the attributes is clear in the schema
■ Reducing the redundant information in tuples
■ Reducing the NULL values in tuples
■ Disallowing the possibility of generating spurious tuples
These measures are not always independent of one another, as we will see.
1.1 Imparting Clear Semantics to Attributes in Relations
510
Basics of Functional Dependencies and Normalization for Relational Databases
DEPARTMENT
DnumberDname
Ename Bdate Address Dnumber
EMPLOYEE
P.K.
P.K.
F.K.
Pname Pnumber Plocation Dnum
PROJECT F.K.
F.K.
DEPT_LOCATIONS
Dnumber Dlocation
P.K.
P.K.
Pnumber Hours
WORKS_ON
P.K.
F.K.
Ssn
Dmgr_ssn
Ssn
Figure 1
In general, the easier it is to explain the semantics of the relation, the better the rela-
511
Basics of Functional Dependencies and Normalization for Relational Databases
Ename
EMPLOYEE
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
999887777
123456789
333445555
453453453
987654321
987987987
888665555
666884444
123456789
123456789
333445555
453453453
333445555
333445555
999887777
987987987
999887777
987654321
987654321
3
1
2
2
1
3
10
20
10
30
30
30
20
20
40.0
32.5
7.5
10.0
20.0
10.0
10.0
10.0
35.0
30.0
5.0
20.0
15.0
Null
1937-11-10
1968-07-19
1965-01-09
1955-12-08
1972-07-31
1969-03-29
1941-06-20
Bdate
3321 Castle, Spring, TX
731 Fondren, Houston, TX 5
638 Voss, Houston, TX
5631 Rice, Houston, TX
980 Dallas, Houston, TX
450 Stone, Houston, TX
291Berry, Bellaire, TX
975 Fire Oak, Humble, TX
Address
4
5
5
4
1
4
5
Dnumber
Dname
DEPARTMENT
Research
Administration
Headquarters 888665555
333445555
987654321
Dnumber
5
1
4
DEPT_LOCATIONS
1
4
5
Dnumber
Houston
Dlocation
Bellaire
Stafford
Houston
Sugarland
5
5
PROJECT
ProductX
ProductY
ProductZ
Pname
1
Pnumber Plocation Dnum
3
2
20
10
Reorganization
30
5
5
5
1
4
4
Bellaire
Houston
Sugarland
Houston
Stafford
StaffordNewbenefits
Computerization
WORKS_ON
Pnumber Hours
Ssn
Dmgr_ssn
Ssn
Figure 2
512
Basics of Functional Dependencies and Normalization for Relational Databases
Ssn
EMP_PROJ
(b)
(a)
FD1
FD2
FD3
Pnumber Hours Ename Pname Plocation
Ename Ssn
EMP_DEPT
Figure 3
The semantics of the other two relation schemas in Figure 1 are slightly more com-
Guideline 1
Examples of Violating Guideline 1. The relation schemas in Figures 3(a) and
513
Basics of Functional Dependencies and Normalization for Relational Databases
(Ename), project name (Pname), and project location (Plocation). Although there is
1.2 Redundant Information in Tuples
One goal of schema design is to minimize the storage space used by the base rela-
Storing natural joins of base relations leads to an additional problem referred to as
Insertion Anomalies. Insertion anomalies can be differentiated into two types,
■ To insert a new employee tuple into EMP_DEPT, we must include either the
■ It is difficult to insert a new department that has no employees as yet in the
2These anomalies were identified by Codd (1972a) to justify the need for normalization of relations, as
514
Basics of Functional Dependencies and Normalization for Relational Databases
Ename
EMP_DEPT
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
Narayan, Ramesh K.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
999887777
123456789
333445555
453453453
987654321
666884444
987987987
888665555 1937-11-10
Ssn
1968-07-19
1965-01-09
1955-12-08
1972-07-31
1969-03-29
1941-06-20
1962-09-15
Bdate
3321 Castle, Spring, TX
731 Fondren, Houston, TX 5
638 Voss, Houston, TX
5631 Rice, Houston, TX
980 Dallas, Houston, TX
450 Stone, Houston, TX
291 Berry, Bellaire, TX
975 FireOak, Humble, TX
Address
4
5
5
4
1
4
5
Administration
Research
Research
Research
Administration
Headquarters
Administration
Research
987654321
333445555
333445555
333445555
987654321
888665555
987654321
333445555
Dnumber Dname Dmgr_ssn
Ssn
EMP_PROJ
123456789
123456789
666884444
453453453
453453453
333445555
333445555
333445555
333445555
999887777
999887777
987987987
987987987
987654321
987654321
888665555
3
1
2
2
1
2
30
30
30
10
10
3
10
20
20
20
Pnumber
40.0
32.5
7.5
10.0
10.0
10.0
10.0
20.0
20.0
30.0
5.0
10.0
35.0
20.0
15.0
Null
Hours
Narayan, Ramesh K.
Smith, John B.
Smith, John B.
Wong, Franklin T.
Wong, Franklin T.
Wong, Franklin T.
Wong, Franklin T.
English, Joyce A.
English, Joyce A.
Zelaya, Alicia J.
Jabbar, Ahmad V.
Zelaya, Alicia J.
Jabbar, Ahmad V.
Wallace, Jennifer S.
Wallace, Jennifer S.
Borg, James E.
Ename
ProductZ
ProductX
ProductY
ProductY
ProductZ
Reorganization
ProductX
ProductY
Newbenefits
Newbenefits
Computerization
Computerization
Newbenefits
Reorganization
Reorganization
Houston
Bellaire
Sugarland
Sugarland
Houston
Stafford
Houston
Bellaire
Sugarland
Stafford
Stafford
Stafford
Stafford
Stafford
Houston
Houston
Pname Plocation
Computerization
Redundancy Redundancy
Redundancy
Figure 4
attributes for employee. This violates the entity integrity for EMP_DEPT
515
Basics of Functional Dependencies and Normalization for Relational Databases
Deletion Anomalies. The problem of deletion anomalies is related to the second
Modification Anomalies. In EMP_DEPT, if we change the value of one of the
It is easy to see that these three anomalies are undesirable and cause difficulties to
Guideline 2
The second guideline is consistent with and, in a way, a restatement of the first
1.3 NULL Values in Tuples
3This is not as serious as the other problems, because all tuples can be updated by a single SQL query.
516
Basics of Functional Dependencies and Normalization for Relational Databases
also lead to problems with understanding the meaning of the attributes and with
■ The attribute does not apply to this tuple. For example, Visa_status may not
■ The attribute value for this tuple is unknown. For example, the Date_of_birth
■ The value is known but absent; that is, it has not been recorded yet. For exam-
Having the same representation for all NULLs compromises the different meanings
Guideline 3
Using space efficiently and avoiding joins with NULL values are the two overriding
1.4 Generation of Spurious Tuples
5This is because inner and outer joins produce different results when NULLs are involved in joins. The
517
Basics of Functional Dependencies and Normalization for Relational Databases
Ssn Pnumber Hours Pname Plocation
Ename
P.K.
EMP_PROJ1
Plocation
P.K.
EMP_LOCS
Ename
Houston
Bellaire
Sugarland
Bellaire
Stafford
Houston
Stafford
Houston
Plocation
(b)
(a)
EMP_PROJ1
Ssn
666884444
453453453
453453453
333445555
333445555
333445555
333445555
999887777
987987987
987987987
987654321
987654321
888665555
3
1
2
2
1
2
30
30
30
10
10
3
20
20
20
Pnumber
40.0
32.5
7.5
10.0
10.0
10.0
10.0
20.0
20.0
30.0
5.0
10.0
35.0
20.0
15.0
NULL
ProductZ
ProductX
ProductY
ProductY
ProductZ
Computerization
Reorganization
ProductX
ProductY
Newbenefits
Newbenefits
Computerization
Computerization
Newbenefits
Reorganization
Reorganization
Houston
Bellaire
Sugarland
Sugarland
Houston
Stafford
Houston
Bellaire
Sugarland
Stafford
Stafford
Stafford
Stafford
Stafford
Houston
Houston
Hours Pname Plocation
EMP_LOCS
EMP_PROJ relation in Figure 4, which are obtained by applying the appropriate
Suppose that we used EMP_PROJ1 and EMP_LOCS as the base relations instead of
518
Basics of Functional Dependencies and Normalization for Relational Databases
Ssn
123456789
123456789
123456789
123456789
666884444
666884444
453453453
453453453
453453453
453453453
453453453
333445555
333445555
333445555
333445555
2
1
1
3
2
2
2
2
2
2
2
3
1
1
2
3
Pnumber
7.5
32.5
32.5
40.0
40.0
20.0
20.0
7.5
7.5
20.0
10.0
20.0
20.0
10.0
10.0
10.0
Hours
ProductY
ProductX
ProductX
ProductZ
ProductZ
ProductX
ProductX
ProductY
ProductY
ProductY
ProductY
ProductY
ProductY
ProductY
ProductY
ProductZ
Pname
Sugarland
Bellaire
Bellaire
Houston
Houston
Bellaire
Bellaire
Sugarland
Sugarland
Sugarland
Sugarland
Sugarland
Sugarland
Sugarland
Sugarland
Houston
333445555 10 10.0 Computerization Stafford
333445555 20
*
*
*
*
*
*
*
*
*
*
*
10.0 Reorganization Houston
Smith, John B.
Smith, John B.
English, Joyce A.
Narayan, Ramesh K.
Wong, Franklin T.
Smith, John B.
English, Joyce A.
English, Joyce A.
Wong, Franklin T.
Smith, John B.
Smith, John B.
English, Joyce A.
Wong, Franklin T.
English, Joyce A.
Wong, Franklin T.
Narayan, Ramesh K.
Wong, Franklin T.
Wong, Franklin T.
Plocation Ename
*
*
Figure 6
they represent spurious information that is not valid. The spurious tuples are
Decomposing EMP_PROJ into EMP_LOCS and EMP_PROJ1 is undesirable because
Guideline 4
519
Basics of Functional Dependencies and Normalization for Relational Databases
matching attributes that are not (foreign key, primary key) combinations because
This informal guideline obviously needs to be stated more formally. There is a for-
1.5 Summary and Discussion of Design Guidelines
■ Anomalies that cause redundant work to be done during insertion into and
■ Waste of storage space due to NULLs and the difficulty of performing selec-
■ Generation of invalid and spurious data during joins on base relations with
In the rest of this chapter we present formal concepts and theory that may be used
2 Functional Dependencies
2.1 Definition of Functional Dependency
520
Basics of Functional Dependencies and Normalization for Relational Databases
relation schema R = {A1, A2, … , An}.
database as a single universal table; we use this concept only in developing the for-
Definition. A functional dependency, denoted by X → Y, between two sets of
This means that the values of the Y component of a tuple in r depend on, or are
Thus, X functionally determines Y in a relation schema R if, and only if, whenever
■ If a constraint on R states that there cannot be more than one tuple with a
■ If X → Y in R, this does not say whether or not Y → X in R.
A functional dependency is a property of the semantics or meaning of the attrib-
7This concept of a universal relation is important in the discussion of algorithms for relational database
521
Basics of Functional Dependencies and Normalization for Relational Databases
TEACH
Teacher
Smith
Hall
Brown
Bartram
Martin
Hoffman
Horowitz
Compilers
Data Structures
Data Management
Data Structures
Course Text
exist in the real world if the relationship changes. For example, the FD Zip_code →
Consider the relation schema EMP_PROJ in Figure 3(b); from the semantics of the
a. Ssn → Ename
These functional dependencies specify that (a) the value of an employee’s Social
A functional dependency is a property of the relation schema R, not of a particular
Given a populated relation, one cannot determine which FDs hold and which do
522
Basics of Functional Dependencies and Normalization for Relational Databases
Figure 8
A B C D
a1 b1 c1 d1
a1 b2 c2 d2
a2 b2 c2 d3
a3 b3 c4 d3
hold if there are tuples that show the violation of such an FD. See the illustrative
Figure 3 introduces a diagrammatic notation for displaying FDs: Each FD is dis-
We denote by F the set of functional dependencies that are specified on relation
3 Normal Forms Based on Primary Keys
■ Perform a conceptual schema design using a conceptual model such as ER or
■ Design the relations based on external knowledge derived from an existing
Following either of these approaches, it is then useful to evaluate the relations for
523
Basics of Functional Dependencies and Normalization for Relational Databases
this section on the first three normal forms for relation schemas and the intuition
We start by informally discussing normal forms and the motivation behind their
3.1 Normalization of Relations
Normalization of data can be considered a process of analyzing the given relation
■ A formal framework for analyzing relation schemas based on their keys and
■ A series of normal form tests that can be carried out on individual relation
Definition. The normal form of a relation refers to the highest normal form
Normal forms, when considered in isolation from other factors, do not guarantee a
524
Basics of Functional Dependencies and Normalization for Relational Databases
relation schema in the database is, say, in BCNF or 3NF. Rather, the process of nor-
■ The nonadditive join or lossless join property, which guarantees that the
■ The dependency preservation property, which ensures that each functional
The nonadditive join property is extremely critical and must be achieved at any
3.2 Practical Use of Normal Forms
Another point worth noting is that the database designers need not normalize to the
Definition. Denormalization is the process of storing the join of higher nor-
3.3 Definitions of Keys and Attributes
Before proceeding further, let’s look again at the definitions of keys of a relation
Definition. A superkey of a relation schema R = {A1, A2, … , An} is a set of
525
Basics of Functional Dependencies and Normalization for Relational Databases
The difference between a key and a superkey is that a key has to be minimal; that is,
If a relation schema has more than one key, each is called a candidate key. One of
Definition. An attribute of relation schema R is called a prime attribute of R if
In Figure 1, both Ssn and Pnumber are prime attributes of WORKS_ON, whereas
We now present the first three normal forms: 1NF, 2NF, and 3NF. These were pro-
3.4 First Normal Form
Consider the DEPARTMENT relation schema shown in Figure 1, whose primary key
■ The domain of Dlocations contains atomic values, but some tuples can have a
526
Basics of Functional Dependencies and Normalization for Relational Databases
Dname
DEPARTMENT
DEPARTMENT
Dnumber Dmgr_ssn Dlocations
Dname
Administration
Headquarters 1
5
4
Dnumber
888665555
333445555
987654321
Dmgr_ssn
{Houston}
{Bellaire, Sugarland, Houston}
{Stafford}
Dlocations
Dname
Research
Research
Administration
Headquarters
Bellaire
Sugarland
Houston
Stafford
Houston
5
5
5
4
1
Dnumber
333445555
333445555
333445555
987654321
888665555
Dmgr_ssn Dlocation
■ The domain of Dlocations contains sets of values and hence is nonatomic. In
In either case, the DEPARTMENT relation in Figure 9 is not in 1NF; in fact, it does
1. Remove the attribute Dlocations that violates 1NF and place it in a separate
9In this case we can consider the domain of Dlocations to be the power set of the set of single loca-
527
Basics of Functional Dependencies and Normalization for Relational Databases
2. Expand the key so that there will be a separate tuple in the original
3. If a maximum number of values is known for the attribute—for example, if it
Of the three solutions above, the first is generally considered best because it does
First normal form also disallows multivalued attributes that are themselves com-
EMP_PROJ(Ssn, Ename, {PROJS(Pnumber, Hours)})
The set braces { } identify the attribute PROJS as multivalued, and we list the com-
Notice that Ssn is the primary key of the EMP_PROJ relation in Figures 10(a) and
This procedure can be applied recursively to a relation with multiple-level nesting
528
Basics of Functional Dependencies and Normalization for Relational Databases
EMP_PROJ
Projs
EMP_PROJ1
Ssn Ename
EMP_PROJ2
HoursSsn Pnumber
EMP_PROJ
Ssn
123456789
666884444
453453453
333445555
999887777
987987987
987654321
888665555
Zelaya, Alicia J.
Jabbar, Ahmad V.
Wallace, Jennifer S.
Borg, James E.
32.5
7.5
20.0
20.0
10.0
10.0
10.0
30.0
35.0
5.0
15.0
NULL
English, Joyce A.
Narayan, Ramesh K.
Smith, John B.
Wong, Franklin T.
Ename
3
1
2
1
2
2
20
3
10
30
10
20
30
20
Pnumber Hours
Figure 10
existence of more than one multivalued attribute in one relation must be handled
PERSON (Ss#, {Car_lic#}, {Phone#})
This relation represents the fact that a person has multiple cars and multiple
PERSON_IN_1NF (Ss#, Car_lic#, Phone#)
529
Basics of Functional Dependencies and Normalization for Relational Databases
To avoid introducing any extraneous relationship between Car_lic# and Phone#, all
3.5 Second Normal Form
Definition. A relation schema R is in 2NF if every nonprime attribute A in R is
The test for 2NF involves testing for functional dependencies whose left-hand side
If a relation schema is not in 2NF, it can be second normalized or 2NF normalized
3.6 Third Normal Form
10This is the general definition of transitive dependency. Because we are concerned only with primary
530
Basics of Functional Dependencies and Normalization for Relational Databases
Ssn
EMP_PROJ
(b)
FD1
FD2
FD3
2NF Normalization
Pnumber Hours Ename Pname Plocation
Ssn
EP1
FD1
Pnumber Hours
Ename Ssn
ED1
Ssn
EP2
FD2
Ename Pnumber
EP3
FD3
Pname Plocation
Ename Ssn
EMP_DEPT
Dnumber
ED2
3NF Normalization
Figure 11
dependencies Ssn → Dnumber and Dnumber → Dmgr_ssn hold and Dnumber is nei-
Definition. According to Codd’s original definition, a relation schema R is in
The relation schema EMP_DEPT in Figure 3(a) is in 2NF, since no partial depen-
531
Basics of Functional Dependencies and Normalization for Relational Databases
Table 1 Summary of Normal Forms Based on Primary Keys and Corresponding Normalization
Normal Form Test Remedy (Normalization)
First (1NF) Relation should have no multivalued
Form new relations for each multivalued
Second (2NF) For relations where primary key con-
Decompose and set up a new relation for
Third (3NF) Relation should not have a nonkey
Decompose and set up a relation that
EMP_DEPT by decomposing it into the two 3NF relation schemas ED1 and ED2
Intuitively, we can see that any functional dependency in which the left-hand side is
4 General Definitions of Second
In general, we want to design our relation schemas so that they have neither partial
532
Basics of Functional Dependencies and Normalization for Relational Databases
any, into account. In this section we give the more general definitions of 2NF and
4.1 General Definition of Second Normal Form
The test for 2NF involves testing for functional dependencies whose left-hand side
Based on the two candidate keys Property_id# and {County_name, Lot#}, the func-
FD3: County_name → Tax_rate
In words, the dependency FD3 says that the tax rate is fixed for a given county (does
The LOTS relation schema violates the general definition of 2NF because Tax_rate is
11This definition can be restated as follows: A relation schema R is in 2NF if every nonprime attribute A
533
Basics of Functional Dependencies and Normalization for Relational Databases
Property_id#
LOTS
FD1
FD2
FD3
FD4
County_name Lot# Area Price Tax_rate
Property_id#
(b)
FD1
FD2
FD4
County_name Lot# Area Price
(c)
(d)
Property_id#
FD1
FD2
County_name Lot# Area
LOTS2
FD3
County_name Tax_rate
LOTS1B
FD4
Area Price
LOTS 1NF
LOTS1
LOTS1A LOTS1B
LOTS2 2NF
LOTS2 3NF
Candidate Key
Figure 12
534
Basics of Functional Dependencies and Normalization for Relational Databases
4.2 General Definition of Third Normal Form
According to this definition, LOTS2 (Figure 12(b)) is in 3NF. However, FD4 in LOTS1
Two points are worth noting about this example and the general definition of 3NF:
■ LOTS1 violates 3NF because Price is transitively dependent on each of the
■ This general definition can be applied directly to test whether a relation
4.3 Interpreting the General Definition
A relation schema R violates the general definition of 3NF if a functional depen-
■ A nonprime attribute determines another nonprime attribute. Here we typ-
■ A proper subset of a key of R functionally determines a nonprime attribute.
Therefore, we can state a general alternative definition of 3NF as follows:
Alternative Definition. A relation schema R is in 3NF if every nonprime attribute
■ It is fully functionally dependent on every key of R.
■ It is nontransitively dependent on every key of R.
535
Basics of Functional Dependencies and Normalization for Relational Databases
Property_id#
LOTS1A(a)
(b)
FD1
FD2
FD1
FD2
FD5
BCNF Normalization
County_name Lot# Area
Property_id#
LOTS1AX
A
B C
Area
LOTS1AY
Figure 13
5 Boyce-Codd Normal Form
The area of a lot that determines the county, as specified by FD5, can be represented
Definition. A relation schema R is in BCNF if whenever a nontrivial functional
536
Basics of Functional Dependencies and Normalization for Relational Databases
TEACH
Student
Smith
Smith
Smith
Mark
Navathe
Ammar
Schulman
Operating Systems
Database
Database
Theory
Wallace
Wallace
Wong
Zelaya
Mark
Ahamad
Omiecinski
Navathe
Database
Database
Operating Systems
Database
Course Instructor
Narayan Operating Systems Ammar
Figure 14
The formal definition of BCNF differs from the definition of 3NF in that condition
In practice, most relation schemas that are in 3NF are also in BCNF. Only if X → A
As another example, consider Figure 14, which shows a relation TEACH with the fol-
FD1: {Student, Course} → Instructor
Note that {Student, Course} is a candidate key for this relation and that the depen-
12This dependency means that each instructor teaches one course is a constraint for this application.
537
Basics of Functional Dependencies and Normalization for Relational Databases
decomposed into one of the three following possible pairs:
1. {Student, Instructor} and {Student, Course}.
2. {Course, Instructor} and {Course, Student}.
3. {Instructor, Course} and {Instructor, Student}.
All three decompositions lose the functional dependency FD1. The desirable decom-
A test to determine whether a decomposition is nonadditive (or lossless) is dis-
We make sure that we meet this property, because nonadditive decomposition is a
(Instructor, Course) and (Instructor, Student)
Note that if we designate (Student, Instructor) as a primary key of the relation
6 Multivalued Dependency
So far we have discussed the concept of functional dependency, which is by far the
538
Basics of Functional Dependencies and Normalization for Relational Databases
(a) EMP
Ename
Smith
Smith
Smith
John
Anna
John
X
Y
X
Y
Pname Dname
(b) EMP_PROJECTS
Ename
Smith
Smith
X
Y
Pname
EMP_DEPENDENTS
Ename
Smith
John
Dname
(c) SUPPLY
Sname
Smith
Smith
Adamsky
Walton
Adamsky
Adamsky
Smith
Bolt
Bolt
Nut
Bolt
Nut
Nail
Bolt
ProjY
ProjX
ProjY
ProjX
ProjZ
ProjX
ProjY
Part_name Proj_name
(d) R1
Smith
Smith
Adamsky
Walton
Adamsky
Bolt
Bolt
Nut
Nut
Nail
Bolt
Bolt
Nut
Nut
Nail
Part_name
R2
Smith
Smith
Adamsky
Walton
Adamsky
Proj_name
ProjY
ProjX
ProjY
ProjZ
ProjX
R3
ProjY
ProjX
ProjY
ProjZ
ProjX
Figure 15
EMP_DEPENDENTS.
For example, consider the relation EMP shown in Figure 15(a). A tuple in this EMP
13In an ER diagram, each would be represented as a multivalued attribute or as a weak entity type.
539
Basics of Functional Dependencies and Normalization for Relational Databases
ified as a multivalued dependency on the EMP relation, which we define in this sec-
6.1 Formal Definition of Multivalued Dependency
15 where
■ t3[X] = t4[X] = t1[X] = t2[X].
■ t3[Y] = t1[Y] and t4[Y] = t2[Y].
■ t3[Z] = t2[Z] and t4[Z] = t1[Z].
Whenever X →→ Y holds, we say that X multidetermines Y. Because of the symme-
An MVD X →→ Y in R is called a trivial MVD if (a) Y is a subset of X, or (b) X ∪ Y
If we have a nontrivial MVD in a relation, we may have to repeat values redundantly
We now present the definition of fourth normal form (4NF), which is violated
14This MVD is denoted as A →→ B|C.
540
Basics of Functional Dependencies and Normalization for Relational Databases
Definition. A relation schema R is in 4NF with respect to a set of dependencies
We can state the following points:
■ An all-key relation is always in BCNF since it has no FDs.
■ An all-key relation such as the EMP relation in Figure 15(a), which has no
■ A relation that is not in 4NF due to a nontrivial MVD must be decomposed
■ The decomposition removes the redundancy caused by the MVD.
The process of normalizing a relation involving the nontrivial MVDs that is not in
7 Join Dependencies
In our discussion so far, we have pointed out the problematic functional dependen-
17F+ refers to the cover of functional dependencies F, or all dependencies that are implied by F.
541
Basics of Functional Dependencies and Normalization for Relational Databases
Definition. A join dependency (JD), denoted by JD(R1, R2, …, Rn), specified on
∗ (πR1(r), πR2(r), …, πRn(r)) = r
Notice that an MVD is a special case of a JD where n = 2. That is, a JD denoted as
Definition. A relation schema R is in fifth normal form (5NF) (or project-join
For an example of a JD, consider once again the SUPPLY all-key relation in Figure
Discovering JDs in practical databases with hundreds of attributes is next to impos-
8 Summary
18Again, F+ refers to the cover of functional dependencies F, or all dependencies that are implied by F.
542
Basics of Functional Dependencies and Normalization for Relational Databases
whether a relation schema is good or bad, and provided informal guidelines for a
We defined the concept of functional dependency, which is the basic tool for analyz-
We presented Boyce-Codd normal form (BCNF) and discussed how it is a stronger
Review Questions
tion schema.
543
Basics of Functional Dependencies and Normalization for Relational Databases
2. Discuss insertion, deletion, and modification anomalies. Why are they con-
3. Why should NULLs in a relation be avoided as much as possible? Discuss the
4. State the informal guidelines for relation schema design that we discussed.
5. What is a functional dependency? What are the possible sources of the infor-
6. Why can we not infer a functional dependency automatically from a partic-
7. What does the term unnormalized relation refer to? How did the normal
8. Define first, second, and third normal forms when only primary keys are
9. What undesirable dependencies are avoided when a relation is in 2NF?
10. What undesirable dependencies are avoided when a relation is in 3NF?
11. In what way do the generalized definitions of 2NF and 3NF extend the defi-
12. Define Boyce-Codd normal form. How does it differ from 3NF? Why is it
13. What is multivalued dependency? When does it arise?
14. Does a relation with two or more columns always have an MVD? Show with
15. Define fourth normal form. When is it violated? When is it typically
16. Define join dependency and fifth normal form.
17. Why is 5NF also called project-join normal form (PJNF)?
18. Why do practical database designs typically aim for BCNF and not aim for
Exercises
that is used to keep track of students’ transcripts:
a. The university keeps track of each student’s name (Sname), student num-
544
Basics of Functional Dependencies and Normalization for Relational Databases
phone (Sc_phone), permanent address (Sp_addr) and phone (Sp_phone),
b. Each department is described by a name (Dname), department code
c. Each course has a course name (Cname), description (Cdesc), course
d. Each section has an instructor (Iname), semester (Semester), year (Year),
e. A grade record refers to a student (Ssn), a particular section, and a grade
Design a relational database schema for this database application. First show
20. What update anomalies occur in the EMP_PROJ and EMP_DEPT relations of
21. In what normal form is the LOTS relation schema in Figure 12(a) with
22. Prove that any relation schema with two attributes is in BCNF.
23. Why do spurious tuples occur in the result of joining the EMP_PROJ1 and
24. Consider the universal relation R = {A, B, C, D, E, F, G, H, I, J} and the set of
25. Repeat Exercise 24 for the following different set of functional dependencies
545
Basics of Functional Dependencies and Normalization for Relational Databases
A B C TUPLE#
26. Consider the following relation:
a. Given the previous extension (state), which of the following dependencies
i. A → B, ii. B → C, iii. C → B, iv. B → A, v. C → A
it? If it does not, why not?
27. Consider a relation R(A, B, C, D, E) with the following dependencies:
AB → C, CD → E, DE → B
28. Consider the relation R, which has attributes that hold schedules of courses
{Course_no} → {Offering_dept, Credit_hours, Course_level}
No_of_students, Instructor_ssn}
Sec_no}
Try to determine which sets of attributes form keys of R. How would you
29. Consider the following relations for an order-processing application data-
ORDER (O#, Odate, Cust#, Total_amount)
Assume that each item has a different discount. The Total_price refers to one
546
Basics of Functional Dependencies and Normalization for Relational Databases
30. Consider the following relation:
CAR_SALE(Car#, Date_sold, Salesperson#, Commission%, Discount_amt)
Assume that a car may be sold by multiple salespeople, and hence {Car#,
Date_sold → Discount_amt and
Based on the given primary key, is this relation in 1NF, 2NF, or 3NF? Why or
31. Consider the following relation for published books:
BOOK (Book_title, Author_name, Book_type, List_price, Author_affil,
Author_affil refers to the affiliation of author. Suppose the following depen-
Book_title → Publisher, Book_type
a. What normal form is the relation in? Explain your answer.
b. Apply normalization until you cannot decompose the relations further.
32. This exercise asks you to convert business statements into dependencies.
Write each of the following dependencies as an FD:
a. The manufacturer and serial number uniquely identifies the drive.
b. A model number is registered by a manufacturer and therefore can’t be
c. All disk drives in a particular batch are the same model.
d. All disk drives of a certain model of a particular manufacturer have
33. Consider the following relation:
R (Doctor#, Patient#, Date, Diagnosis, Treat_code, Charge)
547
Basics of Functional Dependencies and Normalization for Relational Databases
In the above relation, a tuple describes a visit of a patient to a doctor along
34. Consider the following relation:
CAR_SALE (Car_id, Option_type, Option_listprice, Sale_date,
This relation refers to options installed in cars (e.g., cruise control) that were
If CarID → Sale_date and Option_type → Option_listprice and CarID,
35. Consider the relation:
BOOK (Book_Name, Author, Edition, Year)
with the data:
a. Based on a common-sense understanding of the above data, what are the
b. Justify that this relation has the MVD { Book } →→ { Author } | { Edition, Year }.
MVD? Evaluate each resulting relation for the highest normal form it
36. Consider the following relation:
TRIP (Trip_id, Start_date, Cities_visited, Cards_used)
This relation refers to business trips made by company salespeople. Suppose
a. Discuss what FDs and/or MVDs exist in this relation.
b. Show how you will go about normalizing it.
Book_Name Author Edition Copyright_Year
DB_fundamentals Navathe 4 2004
548
Basics of Functional Dependencies and Normalization for Relational Databases
Laboratory Exercise
R = [a, b, c, d, e, f, g, h, i, j]
[[a],[d, e]],
Since DBD is implemented in Prolog, use of uppercase terms is reserved for vari-
37. Using the DBD system, verify your answers to the following exercises:
a. 15.24 (3NF only)
b. 15.25
c. 15.27
d. 15.28
Selected Bibliography
549
Relational Database Design
Dependencies
Atop-down relational design technique involvesdesigning an ER or EER conceptual schema, then
In this chapter we use the theory of normal forms and functional, multivalued, and
From Chapter 16 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
550
Relational Database Design Algorithms and Further Dependencies
succinctly, and the minimal cover allows us to do it. Second, we discuss the desirable
In Section 1 we discuss the rules of inference for functional dependencies and use
In Section 5 we discuss the multivalued dependency (MVD) concept further by
It is possible to skip some or all of Sections 3, 4, and 5 in an introductory database
551
Relational Database Design Algorithms and Further Dependencies
1 Further Topics in Functional Dependencies:
In the chapter “Basics of Functional Dependencies and Normalization for Relational
1.1 Inference Rules for Functional Dependencies
In real life, it is impossible to specify all possible functional dependencies for a given
Definition. Formally, the set of all dependencies that include F as well as all
For example, suppose that we specify the following set F of obvious functional
F = {Ssn → {Ename, Bdate, Address, Dnumber}, Dnumber → {Dname, Dmgr_ssn} }
Some of the additional functional dependencies that we can infer from F are the fol-
Ssn → {Dname, Dmgr_ssn}
552
Relational Database Design Algorithms and Further Dependencies
An FD X → Y is inferred from a set of dependencies F specified on R if X → Y
In the following discussion, we use an abbreviated notation when discussing func-
IR1 (reflexive rule)1: If X ⊇ Y, then X →Y.
The reflexive rule (IR1) states that a set of attributes always determines itself or any
One cautionary note regarding the use of these rules. Although X → A and X → B
1The reflexive rule can also be stated as X → X; that is, any set of attributes functionally determines itself.
553
Relational Database Design Algorithms and Further Dependencies
Each of the preceding inference rules can be proved from the definition of func-
Proof of IR1. Suppose that X ⊇ Y and that two tuples t1 and t2 exist in some rela-
Proof of IR3. Assume that (1) X → Y and (2) Y → Z both hold in a relation r.
Using similar proof arguments, we can prove the inference rules IR4 to IR6 and any
Proof of IR4 (Using IR1 through IR3).
1. X → YZ (given).
Proof of IR5 (using IR1 through IR3).
1. X →Y (given).
Proof of IR6 (using IR1 through IR3).
1. X → Y (given).
It has been shown by Armstrong (1974) that inference rules IR1 through IR3 are
554
Relational Database Design Algorithms and Further Dependencies
F specified on a relation schema R, any dependency that we can infer from F by
Typically, database designers first specify the set of functional dependencies F that
Definition. For each such set of attributes X, we determine the set X+ of attrib-
Algorithm 1. Determining X+, the Closure of X under F
Input: A set F of FDs on a relation schema R, and a set of attributes X, which is
X+ := X;
oldX+ := X+;
if X+ ⊇ Y then X+ := X+ ∪ Z;
Algorithm 1 starts by setting X+ to all the attributes in X. By IR1, we know that all
F = {Ssn → Ename,
3They are actually known as Armstrong’s axioms. In the strict mathematical sense, the axioms (given
555
Relational Database Design Algorithms and Further Dependencies
Using Algorithm 1, we calculate the following closure sets with respect to F:
{Ssn} + = {Ssn, Ename}
Intuitively, the set of attributes in the right-hand side in each line above represents
1.2 Equivalence of Sets of Functional Dependencies
Definition. A set of functional dependencies F is said to cover another set of
Definition. Two sets of functional dependencies E and F are equivalent if
We can determine whether F covers E by calculating X+ with respect to F for each FD
F = {A → C, AC → D, E → AD, E → H}
1.3 Minimal Sets of Functional Dependencies
1. Every dependency in F has a single attribute for its right-hand side.
2. We cannot replace any dependency X → A in F with a dependency Y → A,
3. We cannot remove any dependency from F and still have a set of dependen-
We can think of a minimal set of dependencies as being a set of dependencies in a
556
Relational Database Design Algorithms and Further Dependencies
every dependency in a canonical form with a single attribute on the right-hand
Definition. A minimal cover of a set of functional dependencies E is a minimal
If several sets of FDs qualify as minimal covers of E by the definition above, it is cus-
Algorithm 2. Finding a Minimal Cover F for a Set of Functional Dependencies
Input: A set of functional dependencies E.
1. Set F := E.
2. Replace each functional dependency X → {A1, A2, …, An} in F by the n func-
3. For each functional dependency X → A in F
if { {F – {X → A} } ∪ { (X – {B} ) → A} } is equivalent to F
4. For each remaining functional dependency X → A in F
then remove X → A from F.
We illustrate the above algorithm with the following:
Let the given set of FDs be E : {B → A, D → A, AB → D}. We have to find the mini-
■ All above dependencies are in canonical form (that is, they have only one
4This is a standard form to simplify the conditions and algorithms that ensure no redundancy exists in F.
557
Relational Database Design Algorithms and Further Dependencies
■ Since B → A, by augmenting with B on both sides (IR2), we have BB → AB,
■ Hence by the transitive rule (IR3), we get from (i) and (ii), B → D. Thus
■ We now have a set equivalent to original E, say E�: {B → A, D → A, B → D}.
■ In step 3 we look for a redundant FD in E�. By using the transitive rule on
■ Therefore, the minimal cover of E is {B → D, D → A}.
In Section 3 we will see how relations can be synthesized from a given set of depend-
Next, we provide a simple algorithm to determine the key of a relation:
Algorithm 2(a). Finding a Key K for R Given a set F of Functional
Input: A relation R and a set of functional dependencies F on the attributes of
1. Set K := R.
2. For each attribute A in K
{compute (K – A)+ with respect to F;
if (K – A)+ contains all the attributes in R, then set K := K – {A} };
In Algoritm 2(a), we start by setting K to all the attributes of R; we then remove one
2 Properties of Relational Decompositions
558
Relational Database Design Algorithms and Further Dependencies
2.1 Relation Decomposition and Insufficiency
The relational database design algorithms that we present in Section 3 start from a
We must make sure that each attribute in R will appear in at least one relation
This is called the attribute preservation condition of a decomposition.
Another goal is to have each individual relation Ri in the decomposition D be in
2.2 Dependency Preservation Property
It would be useful if each functional dependency X→Y specified in F either
R Ri
m
=
1
5As an exercise, the reader should prove that this statement is true.
559
Relational Database Design Algorithms and Further Dependencies
each dependency in F represents a constraint on the database. If one of the depen-
It is not necessary that the exact dependencies specified in F appear themselves in
Definition. Given a set of dependencies F on R, the projection of F on Ri,
+ = F+.
If a decomposition is not dependency-preserving, some dependency is lost in the
An example of a decomposition that does not preserve dependencies is shown in
Claim 1. It is always possible to find a dependency-preserving decomposition
In Section 3.1, we describe Algorithm 4, which creates a dependency-
2.3 Nonadditive (Lossless) Join Property
Another property that a decomposition D should possess is the nonadditive join
560
Relational Database Design Algorithms and Further Dependencies
Functional Dependencies and Normalization for Relational Databases” with the
Definition. Formally, a decomposition D = {R1, R2, …, Rm} of R has the lossless
The word loss in lossless refers to loss of information, not to loss of tuples. If a decom-
The decomposition of EMP_PROJ(Ssn, Pnumber, Hours, Ename, Pname, Plocation) in
Algorithm 3. Testing for Nonadditive Join Property
Input: A universal relation R, a decomposition D = {R1, R2, …, Rm} of R, and a
Note: Explanatory comments are given at the end of some of the steps. They fol-
1. Create an initial matrix S with one row i for each relation Ri in D, and one
2. Set S(i, j):= bij for all matrix entries. (* each bij is a distinct symbol associated
3. For each row i representing relation schema Ri
{if (relation Ri includes attribute Aj) then set S(i, j):= aj;};}; (* each aj is a
561
Relational Database Design Algorithms and Further Dependencies
4. Repeat the following loop until a complete loop execution results in no
{for all rows in S that have the same symbols in the columns corresponding
{make the symbols in each column that correspond to an attribute in Y
5. If a row is made up entirely of a symbols, then the decomposition has the
Given a relation R that is decomposed into a number of relations R1, R2, …, Rm,
If, on the other hand, no row ends up being all a symbols, D does not satisfy the
Figure 1(a) shows how we apply Algorithm 3 to the decomposition of the
Figure 1(b) shows another decomposition of EMP_PROJ (into EMP, PROJECT, and
562
Relational Database Design Algorithms and Further Dependencies
Pnumber
PROJECT(b)
Ssn
a1
a2
b13
b14
a5
b16
a1
a2
b13
b14
b15
b16
R2
R1
D = {R1, R2 }
(No changes to matrix after applying functional dependencies)
Ename Pnumber Pname HoursPlocation
Ssn
EMP
(a) R = {Ssn, Ename, Pnumber, Pname, Plocation, Hours}
(c)
Ename Ssn
WORKS_ON
Ssn
a1 b32 a3 b34 b35 a6
a1
a2
b13
b14
b15
b16
R1
(Original matrix S at start of algorithm)
Ename Pnumber Pname HoursPlocation
Ssn
(Matrix S after applying the first two functional dependencies;
Ename Pnumber Pname HoursPlocation
F = {Ssn Ename; Pnumber {Pname, Plocation}; {Ssn, Pnumber} Hours}
D = {R1, R2, R3}R = {Ssn, Ename, Pnumber, Pname, Plocation, Hours}
F = {Ssn Ename; Pnumber {Pname, Plocation}; {Ssn, Pnumber} Hours}
b35
Figure 1
563
Relational Database Design Algorithms and Further Dependencies
2.4 Testing Binary Decompositions for the Nonadditive
Algorithm 3 allows us to test whether a particular decomposition D into n relations
Property NJB (Nonadditive Join Test for Binary Decompositions). A
■ The FD ((R1 ∩ R2) → (R1 – R2)) is in F
■ The FD ((R1 ∩ R2) → (R2 – R1)) is in F
You should verify that this property holds with respect to our informal successive
2.5 Successive Nonadditive Join Decompositions
Claim 2 (Preservation of Nonadditivity in Successive Decompositions). If a
3 Algorithms for Relational Database Schema
We now give three algorithms for creating a relational decomposition from a uni-
564
Relational Database Design Algorithms and Further Dependencies
3.1 Dependency-Preserving Decomposition
Algorithm 4 creates a dependency-preserving decomposition D = {R1, R2, …, Rm} of
Algorithm 4. Relational Synthesis into 3NF with Dependency Preservation
1. Find a minimal cover G for F (use Algorithm 2);
2. For each left-hand-side X of a functional dependency that appears in G, cre-
3. Place any remaining attributes (that have not been placed in any relation) in
Example of Algorithm 4. Consider the following universal relation:
U(Emp_ssn, Pno, Esal, Ephone, Dno, Pname, Plocation)
Emp_ssn, Esal, Ephone refer to the Social Security number, salary, and phone number
The following dependencies are present:
FD1: Emp_ssn → {Esal, Ephone, Dno}
By virtue of FD3, the attribute set {Emp_ssn, Pno} represents a key of the universal
By applying the minimal cover Algorithm 2, in step 3 we see that Pno is a redundant
Minimal cover G: {Emp_ssn → Esal, Ephone, Dno; Pno → Pname, Plocation}
565
Relational Database Design Algorithms and Further Dependencies
6See Maier (1983) or Ullman (1982) for a proof.
By applying Algorithm 4 to the above Minimal cover G, we get a 3NF design consist-
R1 (Emp_ssn, Esal, Ephone, Dno)
An observant reader would notice easily that these two relations have lost the original
Claim 3. Every relation schema created by Algorithm 4 is in 3NF. (We will not
It is obvious that all the dependencies in G are preserved by the algorithm because
3.2 Nonadditive Join Decomposition into BCNF Schemas
Algorithm 5. Relational Decomposition into BCNF with Nonadditive
Input: A universal relation R and a set of functional dependencies F on the
1. Set D := {R} ;
2. While there is a relation schema Q in D that is not in BCNF do
choose a relation schema Q in D that is not in BCNF;
} ;
566
Relational Database Design Algorithms and Further Dependencies
Each time through the loop in Algorithm 5, we decompose one relation schema Q
In step 2 of Algorithm 5, it is necessary to determine whether a relation schema Q is
3.3 Dependency-Preserving and Nonadditive (Lossless)
So far, in Algorithm 4 we showed how to achieve a 3NF design with the potential for
■ Preserves dependencies
■ Has the nonadditive join property
■ Is such that each resulting relation schema in the decomposition is in 3NF
Because the Algorithm 6 achieves both the desirable properties, rather than only
Algorithm 6. Relational Synthesis into 3NF with Dependency Preservation
Input: A universal relation R and a set of functional dependencies F on the
567
Relational Database Design Algorithms and Further Dependencies
7Step 3 of Algorithm 4 is not needed in Algorithm 6 to preserve attributes because the key will include
1. Find a minimal cover G for F (use Algorithm 2).
2. For each left-hand-side X of a functional dependency that appears in G, cre-
3. If none of the relation schemas in D contains a key of R, then create one
(Algorithm 2(a) may be used to find a key.)
4. Eliminate redundant relations from the resulting set of relations in the rela-
Step 3 of Algorithm 6 involves identifying a key K of R. Algorithm 2(a) can be used
Example 1 of Algorithm 6. Let us revisit the example given earlier at the end of
R1 (Emp_ssn , Esal, Ephone, Dno)
This design achieves both the desirable properties of dependency preservation and
Example 2 of Algorithm 6 (Case X ). Consider the relation schema LOTS1A
FD1: Property_id → Lot#, County, Area
These were called FD1, FD2, and FD5 in Figure 13(a) of that chapter. For ease of ref-
568
Relational Database Design Algorithms and Further Dependencies
erence, let us abbreviate the above attributes with the first letter for each and repre-
F : { P → LCA, LC → AP, A → C }.
If we apply the minimal cover Algorithm 2 to F, (in step 2) we first represent the set
F : {P → L, P → C, P → A, LC → A, LC → P, A → C}.
In the set F, P → A can be inferred from P → LC and LC → A; hence P → A by tran-
Minimal cover GX: {P → LC, LC → AP, A → C }.
In step 2 of Algorithm 6 we produce design X (before removing redundant rela-
Design X: R1 (P, L, C), R2 (L, C, A, P), and R3 (A, C).
In step 4 of the algorithm, we find that R3 is subsumed by R2 (that is, R3 is always a
Design X: R2 (L, C, A, P).
or, in other words it is identical to the relation LOTS1A (Lot#, County, Area,
Example 2 of Algorithm 6 (Case Y ). Starting with LOTS1A as the universal rela-
F: {P → C, P → A, P → L, LC → A, LC → P, A → C}.
The FD LC → A may be considered redundant because LC → P and P → A implies
Minimal cover GY: { P → LA, LC → P, A → C }.
The alternative design Y produced by the algorithm now is
Design Y: S1 (P, A, L), S2 (L, C, P), and S3 (A, C).
Note that this design has three 3NF relations, none of which can be considered as
It is important to note that the theory of nonadditive join decompositions is based
569
Relational Database Design Algorithms and Further Dependencies
section discusses some of the problems that NULLs may cause in relational decom-
4 About Nulls, Dangling Tuples,
In this section we will discuss a few general issues related to problems that arise
4.1 Problems with NULL Values and Dangling Tuples
In general, whenever a relational database schema is designed in which two or more
A related problem is that of dangling tuples, which may occur if we carry a decom-
9This sometimes happens when we apply vertical fragmentation to a relation in the context of a distrib-
570
Relational Database Design Algorithms and Further Dependencies
(b)
Ename
EMPLOYEE
Ssn Bdate Address Dnum
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
Narayan, Ramesh K.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
987987987
888665555
1969-03-29
1937-11-10
980 Dallas, Houston, TX
450 Stone, Houston, TX
123456789
333445555
999887777
987654321
666884444
453453453
1965-01-09
1955-12-08
1968-07-19
1941-06-20
1962-09-15
1972-07-31
731 Fondren, Houston, TX
638 Voss, Houston, TX
3321 Castle, Spring, TX
291 Berry, Bellaire, TX
975 Fire Oak, Humble, TX
5631 Rice, Houston, TX
5
5
4
4
5
4
1
Berger, Anders C. 999775555 1965-04-26 6530 Braes, Bellaire, TX NULL
Benitez, Carlos M. 888664444 1963-01-09 7654 Beech, Houston, TX NULL
5
Dname
DEPARTMENT
Dnum Dmgr_ssn
Research
Administration
5
4
333445555
987654321
Ename
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
Narayan, Ramesh K.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
999887777
123456789
333445555
453453453
987654321
666884444
987987987
888665555 1937-11-10
Ssn
1968-07-19
1965-01-09
1955-12-08
1972-07-31
1969-03-29
1941-06-20
1962-09-15
Bdate
3321 Castle, Spring, TX
731 Fondren, Houston, TX 5
638 Voss, Houston, TX
5631 Rice, Houston, TX
980 Dallas, Houston, TX
450 Stone, Houston, TX
291 Berry, Bellaire, TX
975 Fire Oak, Humble, TX
Address
4
5
5
4
1
4
5
Administration
Research
Research
Research
Administration
Headquarters
Administration
Research
987654321
333445555
333445555
333445555
987654321
888665555
987654321
333445555
Dnum Dname Dmgr_ssn
(c)
Ename
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
Narayan, Ramesh K.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
999887777
123456789
333445555
453453453
987654321
666884444
987987987
888665555 1937-11-10
1968-07-19
1965-01-09
1955-12-08
1972-07-31
1969-03-29
1941-06-20
1962-09-15
Bdate
3321 Castle, Spring, TX
731 Fondren, Houston, TX 5
638 Voss, Houston, TX
5631 Rice, Houston, TX
980 Dallas, Houston, TX
450 Stone, Houston, TX
291 Berry, Bellaire, TX
975 Fire Oak, Humble, TX
Address
4
5
5
4
1
4
5
Administration
Research
Research
Research
Administration
Headquarters
Administration
Research
987654321
333445555
333445555
333445555
987654321
888665555
Berger, Anders C.
Benitez, Carlos M.
999775555
888665555 1963-01-09
1965-04-26 6530 Braes, Bellaire, TX
7654 Beech, Houston, TX
NULL
NULL
NULL
NULL
NULL
NULL
987654321
333445555
Dnum Dname Dmgr_ssnSsn
Figure 2
571
Ename
EMPLOYEE_1(a)
(b)
Ssn Bdate Address
Smith, John B.
Wong, Franklin T.
Zelaya, Alicia J.
Wallace, Jennifer S.
Narayan, Ramesh K.
English, Joyce A.
Jabbar, Ahmad V.
Borg, James E.
987987987
888665555
1969-03-29
1937-11-10
980 Dallas, Houston, TX
450 Stone, Houston, TX
123456789
333445555
999887777
987654321
666884444
453453453
1965-01-09
1955-12-08
1968-07-19
1941-06-20
1962-09-15
1972-07-31
731 Fondren, Houston, TX
638 Voss, Houston, TX
3321 Castle, Spring, TX
291 Berry, Bellaire, TX
975 Fire Oak, Humble, TX
5631 Rice, Houston, TX
Berger, Anders C.
Benitez, Carlos M.
999775555
888665555
1965-04-26
1963-01-09
6530 Braes, Bellaire, TX
7654 Beech, Houston, TX
EMPLOYEE_2
Ssn
123456789
333445555
999887777
987654321
666884444
453453453
987987987
888665555
999775555
888664444
4
5
5
5
4
5
NULL
4
1
NULL
Dnum
(c) EMPLOYEE_3
Ssn
123456789
333445555
999887777
987654321
666884444
453453453
987987987
888665555
4
5
5
5
4
5
4
1
Dnum
Relational Database Design Algorithms and Further Dependencies
Figure 3
all attributes of EMPLOYEE from
(b) The relation EMPLOYEE_2 (includes
(c) The relation EMPLOYEE_3 (includes
employee has not been assigned a department (instead of including a tuple with
4.2 Discussion of Normalization Algorithms
One of the problems with the normalization algorithms we described is that the
572
Relational Database Design Algorithms and Further Dependencies
of attributes. Failure to specify one or two important dependencies may result in an
It is not always possible to find a decomposition into relation schemas that pre-
To illustrate the above points, let us revisit the LOTS1A relation in Figure 13(a) from
Table 1 summarizes the properties of the algorithms discussed in this chapter so far.
573
Relational Database Design Algorithms and Further Dependencies
Table 1 Summary of the Algorithms Discussed in This Chapter
Algorithm Input Output Properties/Purpose Remarks
1 An attribute or a set
A set of attrbutes in
Determine all the
The closure of a key
2 A set of functional
The minimal cover
To determine the
Multiple minimal
2a Relation schema R
Key K of R To find a key K
The entire relation R
3 A decomposition D
Boolean result: yes or
Testing for nonaddi-
See a simpler test
4 A relation R and a
A set of relations in
Dependency preser-
No guarantee of sat-
5 A relation R and a
A set of relations in
Nonadditive join
No guarantee of
6 A relation R and a
A set of relations in
Nonadditive join
May not achieve
7 A relation R and a
A set of relations in
Nonadditive join
No guarantee of
5 Discussion of Multivalued Dependencies and
Now we visit MVDs (Multivalued Dependencies) to state the rules of inference on
574
5.1 Inference Rules for Functional
As with functional dependencies (FDs), inference rules for multivalued dependen-
IR1 (reflexive rule for FDs): If X ⊇ Y, then X → Y.
IR1 through IR3 are Armstrong’s inference rules for FDs alone. IR4 through IR6 are
5.2 Fourth Normal Form Revisited
Definition. A relation schema R is in 4NF with respect to a set of dependen-
Relational Database Design Algorithms and Further Dependencies
10That is, the set of values of Y determined by a value of X is restricted to being a singleton set with only
575
(a) EMP
Ename
Smith
Smith
Smith
Smith
Brown
Brown
Brown
Brown
Brown
Brown
Brown
Brown
Brown
Brown
Brown
Brown
John
Anna
Anna
John
Jim
Jim
Jim
Jim
Joan
Joan
Joan
Joan
Bob
Bob
Bob
Bob
X
Y
X
Y
Y
Z
W
X
Y
Z
W
X
Y
Z
W
X
Pname Dname
(b) EMP_PROJECTS
Ename
Smith
Smith
Brown
Brown
Brown
Brown
W
X
Y
Z
X
Y
Pname
EMP_DEPENDENTS
Ename
Smith
Brown
Brown
Jim
Bob
Anna
Dname
Figure 4
Relational Database Design Algorithms and Further Dependencies
To illustrate the importance of 4NF, Figure 4(a) shows the EMP relation with an
If the relation has nontrivial MVDs, then insert, delete, and update operations on
576
example, to add the information that ‘Brown’ will be assigned to project ‘P’, only a
The EMP relation in Figure 15(a) from the chapter “Basics of Functional
5.3 Nonadditive Join Decomposition into 4NF Relations
Property NJB�. The relation schemas R1 and R2 form a nonadditive join
(R1 ∩ R2) →→ (R1 – R2)
(R1 ∩ R2) →→ (R2 – R1).
We can use a slight modification of Algorithm 5 to develop Algorithm 7, which cre-
Algorithm 7. Relational Decomposition into 4NF Relations with Nonadditive
Input: A universal relation R and a set of functional and multivalued depend-
1. Set D:= { R };
2. While there is a relation schema Q in D that is not in 4NF, do
find a nontrivial MVD X →→ Y in Q that violates 4NF;
};
Relational Database Design Algorithms and Further Dependencies
577
Relational Database Design Algorithms and Further Dependencies
6 Other Dependencies and Normal Forms
6.1 Inclusion Dependencies
■ The foreign key (or referential integrity) constraint cannot be specified as a
■ The constraint between two relations that represent a class/subclass relation-
Definition. An inclusion dependency R.X < S.Y between two sets of attrib-
utes—X of relation schema R, and Y of relation schema S—specifies the con-
straint that, at any specific time when r is a relation state of R and s a relation
state of S, we must have
πX(r(R)) ⊆ πY(s(S))
The ⊆ (subset) relationship does not necessarily have to be a proper subset.
Obviously, the sets of attributes on which the inclusion dependency is specified—X
of R and Y of S—must have the same number of attributes. In addition, the
domains for each pair of corresponding attributes should be compatible. For exam-
ple, if X = {A1, A2, ..., An} and Y = {B1, B2, ..., Bn}, one possible correspondence is to
have dom(Ai) compatible with dom(Bi) for 1 ≤ i ≤ n. In this case, we say that Ai
corresponds to Bi.
For example, we can specify the following inclusion dependencies on the relational
schema in Figure 1 from the chapter “Basics of Functional Dependencies and
Normalization for Relational Databases”:
DEPARTMENT.Dmgr_ssn < EMPLOYEE.Ssn
WORKS_ON.Ssn < EMPLOYEE.Ssn
EMPLOYEE.Dnumber < DEPARTMENT.Dnumber
PROJECT.Dnum < DEPARTMENT.Dnumber
WORKS_ON.Pnumber < PROJECT.Pnumber
DEPT_LOCATIONS.Dnumber < DEPARTMENT.Dnumber
All the preceding inclusion dependencies represent referential integrity
constraints. We can also use inclusion dependencies to represent class/subclass
578
relationships. For example, in the relational schema of Figure A.1 (in Appendix:
Figures of the end of this chapter), we can specify the following inclusion depend-
encies:
EMPLOYEE.Ssn < PERSON.Ssn
ALUMNUS.Ssn < PERSON.Ssn
STUDENT.Ssn < PERSON.Ssn
As with other types of dependencies, there are inclusion dependency inference rules
(IDIRs). The following are three examples:
IDIR1 (reflexivity): R.X < R.X.
IDIR2 (attribute correspondence): If R.X < S.Y, where X = {A1, A2, ..., An} and
Y = {B1, B2, ..., Bn} and Ai corresponds to Bi, then R.Ai < S.Bi for 1 ≤ i ≤ n.
IDIR3 (transitivity): If R.X < S.Y and S.Y < T.Z, then R.X < T.Z.
The preceding inference rules were shown to be sound and complete for inclusion
dependencies. So far, no normal forms have been developed based on inclusion
dependencies.
6.2 Template Dependencies
Template dependencies provide a technique for representing constraints in relations
that typically have no easy and formal definitions. No matter how many types of
dependencies we develop, some peculiar constraint may come up based on the
semantics of attributes within relations that cannot be represented by any of them.
The idea behind template dependencies is to specify a template—or example—that
defines each constraint or dependency.
There are two types of templates: tuple-generating templates and constraint-
generating templates. A template consists of a number of hypothesis tuples that are
meant to show an example of the tuples that may appear in one or more relations.
The other part of the template is the template conclusion. For tuple-generating
templates, the conclusion is a set of tuples that must also exist in the relations if the
hypothesis tuples are there. For constraint-generating templates, the template con-
clusion is a condition that must hold on the hypothesis tuples. Using constraint-
generating templates, we are able to define semantic constraints—those that are
beyond the scope of the relational model in terms of its data definition language
and notation.
Figure 5 shows how we may define functional, multivalued, and inclusion depend-
encies by templates. Figure 6 shows how we may specify the constraint that an
employee’s salary cannot be higher than the salary of his or her direct supervisor on the
relation schema EMPLOYEE in Figure A.2.
Relational Database Design Algorithms and Further Dependencies
579
(a)
X = {A, B}
Y = {C, D}
Hypothesis
Conclusion
a1 b1 c1
c1 = c2 and d1 = d2
a1 b1 c2
(b) R = {A, B, C, D}
R = {A, B, C, D}
X = {A, B}
Y = {C}
X = {C, D}
Y = {E, F}
Hypothesis
Conclusion
(c)
Hypothesis
S = {E, F, G}
Conclusion
a1 b1 c1 d1
a1 b1 c1 d1
c1 d1 g
a1 b1 c2 d2
a1 b1
c1
d1
a1 b1
c2
d2
R = {A, B, C, D}
d1
d2
Figure 5
Templates for some common type of dependencies.
(a) Template for functional dependency X → Y.
(b) Template for the multivalued dependency X →→ Y.
(c) Template for the inclusion dependency R.X < S.Y.
EMPLOYEE = {Name, Ssn, . . . , Salary, Supervisor_ssn}
Hypothesis
Conclusion
a b c d
e f g
c < f
d
Figure 6
Templates for the constraint that an employee’s salary must
be less than the supervisor’s salary.
Relational Database Design Algorithms and Further Dependencies
580
Relational Database Design Algorithms and Further Dependencies
6.3 Functional Dependencies Based on Arithmetic Functions
and Procedures
Sometimes some attributes in a relation may be related via some arithmetic func-
tion or a more complicated functional relationship. As long as a unique value of Y is
associated with every X, we can still consider that the FD X → Y exists. For example,
in the relation
ORDER_LINE (Order#, Item#, Quantity, Unit_price, Extended_price,
Discounted_price)
each tuple represents an item from an order with a particular quantity, and the price
per unit for that item. In this relation, (Quantity, Unit_price ) → Extended_price by the
formula
Extended_price = Unit_price * Quantity.
Hence, there is a unique value for Extended_price for every pair (Quantity, Unit_price ),
and thus it conforms to the definition of functional dependency.
Moreover, there may be a procedure that takes into account the quantity discounts,
the type of item, and so on and computes a discounted price for the total quantity
ordered for that item. Therefore, we can say
(Item#, Quantity, Unit_price ) → Discounted_price, or
(Item#, Quantity, Extended_price) → Discounted_price.
To check the above FD, a more complex procedure COMPUTE_TOTAL_PRICE may
have to be called into play. Although the above kinds of FDs are technically present
in most relations, they are not given particular attention during normalization.
6.4 Domain-Key Normal Form
There is no hard-and-fast rule about defining normal forms only up to 5NF.
Historically, the process of normalization and the process of discovering undesir-
able dependencies were carried through 5NF, but it has been possible to define
stricter normal forms that take into account additional types of dependencies and
constraints. The idea behind domain-key normal form (DKNF) is to specify (theo-
retically, at least) the ultimate normal form that takes into account all possible types
of dependencies and constraints. A relation schema is said to be in DKNF if all con-
straints and dependencies that should hold on the valid relation states can be
enforced simply by enforcing the domain constraints and key constraints on the
relation. For a relation in DKNF, it becomes very straightforward to enforce all data-
base constraints by simply checking that each attribute value in a tuple is of the
appropriate domain and that every key constraint is enforced.
However, because of the difficulty of including complex constraints in a DKNF rela-
tion, its practical utility is limited, since it may be quite difficult to specify general
integrity constraints. For example, consider a relation CAR(Make, Vin#) (where Vin#
is the vehicle identification number) and another relation MANUFACTURE(Vin#,
581
Relational Database Design Algorithms and Further Dependencies
Country) (where Country is the country of manufacture). A general constraint may be
of the following form: If the Make is either ‘Toyota’ or ‘Lexus,’ then the first character
of the Vin# is a ‘J’ if the country of manufacture is ‘Japan’; if the Make is ‘Honda’ or
‘Acura,’ the second character of the Vin# is a ‘J’ if the country of manufacture is ‘Japan.’
There is no simplified way to represent such constraints short of writing a proce-
dure (or general assertions) to test them. The procedure COMPUTE_TOTAL_PRICE
above is an example of such procedures needed to enforce an appropriate integrity
constraint.
7 Summary
In this chapter we presented a further set of topics related to dependencies, a discus-
sion of decomposition, and several algorithms related to them as well as to normal-
ization. In Section 1 we presented inference rules for functional dependencies
(FDs), the notion of closure of an attribute, closure of a set of functional dependen-
cies, equivalence among sets of functional dependencies, and algorithms for finding
the closure of an attribute (Algorithm 1) and the minimal cover of a set of FDs
(Algorithm 2). We then discussed two important properties of decompositions: the
nonadditive join property and the dependency-preserving property. An algorithm
to test for nonadditive decomposition (Algorithm 3), and a simpler test for check-
ing the losslessness of binary decompositions (Property NJB) were described. We
then discussed relational design by synthesis, based on a set of given functional
dependencies. The relational synthesis algorithms (such as Algorithms 4 and 6) cre-
ate 3NF relations from a universal relation schema based on a given set of functional
dependencies that has been specified by the database designer. The relational decom-
position algorithms (such as Algorithms 5 and 7) create BCNF (or 4NF) relations by
successive nonadditive decomposition of unnormalized relations into two compo-
nent relations at a time. We saw that it is possible to synthesize 3NF relation schemas
that meet both of the above properties; however, in the case of BCNF, it is possible
to aim only for the nonadditiveness of joins—dependency preservation cannot be
necessarily guaranteed. If the designer has to aim for one of these two, the nonaddi-
tive join condition is an absolute must. In Section 4 we showed how certain difficul-
ties arise in a collection of relations due to null values that may exist in relations in
spite of the relations being individually in 3NF or BCNF. Sometimes when decom-
position is improperly carried too far, certain “dangling tuples” may result that do
not participate in results of joins and hence may become invisible. We also showed
how it is possible to have alternative designs that meet a given desired normal form.
Then we revisited multivalued dependencies (MVDs) in Section 5, which arise from
an improper combination of two or more independent multivalued attributes in
the same relation, and that result in a combinational expansion of the tuples used to
define fourth normal form (4NF). We discussed inference rules applicable to MVDs
and discussed the importance of 4NF. Finally, in Section 6 we discussed inclusion
dependencies, which are used to specify referential integrity and class/subclass con-
straints, and template dependencies, which can be used to specify arbitrary types of
582
Relational Database Design Algorithms and Further Dependencies
constraints. We pointed out the need for arithmetic functions or more complex
procedures to enforce certain functional dependency constraints. We concluded
with a brief discussion of the domain-key normal form (DKNF).
Review Questions
1. What is the role of Armstrong’s inference rules (inference rules IR1 through
IR3) in the development of the theory of relational design?
2. What is meant by the completeness and soundness of Armstrong’s inference
rules?
3. What is meant by the closure of a set of functional dependencies? Illustrate
with an example.
4. When are two sets of functional dependencies equivalent? How can we
determine their equivalence?
5. What is a minimal set of functional dependencies? Does every set of depen-
dencies have a minimal equivalent set? Is it always unique?
6. What is meant by the attribute preservation condition on a decomposition?
7. Why are normal forms alone insufficient as a condition for a good schema
design?
8. What is the dependency preservation property for a decomposition? Why is
it important?
9. Why can we not guarantee that BCNF relation schemas will be produced by
dependency-preserving decompositions of non-BCNF relation schemas?
Give a counterexample to illustrate this point.
10. What is the lossless (or nonadditive) join property of a decomposition? Why
is it important?
11. Between the properties of dependency preservation and losslessness, which
one must definitely be satisfied? Why?
12. Discuss the NULL value and dangling tuple problems.
13. Illustrate how the process of creating first normal form relations may lead to
multivalued dependencies. How should the first normalization be done
properly so that MVDs are avoided?
14. What types of constraints are inclusion dependencies meant to represent?
15. How do template dependencies differ from the other types of dependencies
we discussed?
16. Why is the domain-key normal form (DKNF) known as the ultimate normal
form?
583
Relational Database Design Algorithms and Further Dependencies
Exercises
17. Show that the relation schemas produced by Algorithm 4 are in 3NF.
18. Show that, if the matrix S resulting from Algorithm 3 does not have a row
that is all a symbols, projecting S on the decomposition and joining it back
will always produce at least one spurious tuple.
19. Show that the relation schemas produced by Algorithm 5 are in BCNF.
20. Show that the relation schemas produced by Algorithm 6 are in 3NF.
21. Specify a template dependency for join dependencies.
22. Specify all the inclusion dependencies for the relational schema in Figure A.2.
23. Prove that a functional dependency satisfies the formal definition of multi-
valued dependency.
24. Consider the example of normalizing the LOTS relation in Sections 4 and 5
from the chapter “Basics of Functional Dependencies and Normalization for
Relational Databases.” Determine whether the decomposition of LOTS into
{LOTS1AX, LOTS1AY, LOTS1B, LOTS2} has the lossless join property, by
applying Algorithm 3 and also by using the test under Property NJB.
25. Show how the MVDs Ename →→ Pname and Ename →→ Dname in Figure
15(a) from the chapter “Basics of Functional Dependencies and
Normalization for Relational Databases” may arise during normalization into
1NF of a relation, where the attributes Pname and Dname are multivalued.
26. Apply Algorithm 2(a) to the relation in Exercise 24 from the chapter “Basics
of Functional Dependencies and Normalization for Relational Databases” to
determine a key for R. Create a minimal set of dependencies G that is equiv-
alent to F, and apply the synthesis algorithm (Algorithm 6) to decompose R
into 3NF relations.
27. Repeat Exercise 26 for the functional dependencies in Exercise 25 from the
chapter “Basics of Functional Dependencies and Normalization for
Relational Databases.”
28. Apply the decomposition algorithm (Algorithm 5) to the relation R and the
set of dependencies F in Exercise 24 from the chapter “Basics of Functional
Dependencies and Normalization for Relational Databases.” Repeat for the
dependencies G in Exercise 25 from the same chapter.
29. Apply Algorithm 2(a) to the relations in Exercises 27 and 28 from the chap-
ter “Basics of Functional Dependencies and Normalization for Relational
Databases” to determine a key for R. Apply the synthesis algorithm
(Algorithm 6) to decompose R into 3NF relations and the decomposition
algorithm (Algorithm 5) to decompose R into BCNF relations.
30. Write programs that implement Algorithms 5 and 6.
31. Consider the following decompositions for the relation schema R of Exercise
24 from the chapter “Basics of Functional Dependencies and Normalization
for Relational Databases.” Determine whether each decomposition has (1)
the dependency preservation property, and (2) the lossless join property,
with respect to F. Also determine which normal form each relation in the
decomposition is in.
584
Relational Database Design Algorithms and Further Dependencies
a. D1 = {R1, R2, R3, R4, R5} ; R1 = {A, B, C} , R2 = {A, D, E} , R3 = {B, F} , R4 =
{F, G, H} , R5 = {D, I, J}
b. D2 = {R1, R2, R3} ; R1 = {A, B, C, D, E} , R2 = {B, F, G, H} , R3 = {D, I, J}
c. D3 = {R1, R2, R3, R4, R5} ; R1 = {A, B, C, D} , R2 = {D, E} , R3 = {B, F} , R4 =
{F, G, H} , R5 = {D, I, J}
32. Consider the relation REFRIG(Model#, Year, Price, Manuf_plant, Color), which
is abbreviated as REFRIG(M, Y, P, MP, C), and the following set F of func-
tional dependencies: F = {M → MP, {M, Y} → P, MP → C}
a. Evaluate each of the following as a candidate key for REFRIG, giving rea-
sons why it can or cannot be a key: {M}, {M, Y}, {M, C}.
b. Based on the above key determination, state whether the relation REFRIG
is in 3NF and in BCNF, giving proper reasons.
c. Consider the decomposition of REFRIG into D = {R1(M, Y, P), R2(M, MP,
C)}. Is this decomposition lossless? Show why. (You may consult the test
under Property NJB in Section 2.4.)
Laboratory Exercises
Note: These exercises use the DBD (Data Base Designer) system that is described in
the laboratory manual. The relational schema R and set of functional dependencies
F need to be coded as lists. As an example, R and F for problem 24 from the chapter
“Basics of Functional Dependencies and Normalization for Relational Databases”
are coded as:
R = [a, b, c, d, e, f, g, h, i, j]
F = [[[a, b],[c]],
[[a],[d, e]],
[[b],[f]],
[[f],[g, h]],
[[d],[i, j]]]
Since DBD is implemented in Prolog, use of uppercase terms is reserved for vari-
ables in the language and therefore lowercase constants are used to code the attrib-
utes. For further details on using the DBD system, please refer to the laboratory
manual.
33. Using the DBD system, verify your answers to the following exercises:
a. 24
b. 26
c. 27
d. 28
e. 29
f. 31 (a) and (b)
g. 32 (a) and (c)
585
Relational Database Design Algorithms and Further Dependencies
Selected Bibliography
The books by Maier (1983) and Atzeni and De Antonellis (1993) include a compre-
hensive discussion of relational dependency theory. The decomposition algorithm
(Algorithm 5) is due to Bernstein (1976). Algorithm 6 is based on the normalization
algorithm presented in Biskup et al. (1979). Tsou and Fischer (1982) give a polyno-
mial-time algorithm for BCNF decomposition.
The theory of dependency preservation and lossless joins is given in Ullman (1988),
where proofs of some of the algorithms discussed here appear. The lossless join
property is analyzed in Aho et al. (1979). Algorithms to determine the keys of a rela-
tion from functional dependencies are given in Osborn (1977); testing for BCNF is
discussed in Osborn (1979). Testing for 3NF is discussed in Tsou and Fischer
(1982). Algorithms for designing BCNF relations are given in Wang (1990) and
Hernandez and Chan (1991).
Multivalued dependencies and fourth normal form are defined in Zaniolo (1976)
and Nicolas (1978). Many of the advanced normal forms are due to Fagin: the
fourth normal form in Fagin (1977), PJNF in Fagin (1979), and DKNF in Fagin
(1981). The set of sound and complete rules for functional and multivalued
dependencies was given by Beeri et al. (1977). Join dependencies are discussed by
Rissanen (1977) and Aho et al. (1979). Inference rules for join dependencies are
given by Sciore (1982). Inclusion dependencies are discussed by Casanova et al.
(1981) and analyzed further in Cosmadakis et al. (1990). Their use in optimizing
relational schemas is discussed in Casanova et al. (1989). Template dependencies are
discussed by Sadri and Ullman (1982). Other dependencies are discussed in Nicolas
(1978), Furtado (1978), and Mendelzon and Maier (1979). Abiteboul et al. (1995)
provides a theoretical treatment of many of the ideas presented in this chapter.
586
EMPLOYEE
Salary Employee_type Position Rank Percent_time Ra_flag Ta_flag Project Course
STUDENT
Major_dept Grad_flag Undergrad_flag Degree_program Class Student_assist_flag
Name Birth_date Sex Address
PERSON
Ssn
ALUMNUS ALUMNUS_DEGREES
Year MajorSsn
Ssn
Ssn
Ssn Degree
Figure A.1
Mapping an EER specialization lattice using multiple options.
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.2
Schema diagram for the
COMPANY relational
database schema.
587
Disk Storage, Basic File
Structures, and Hashing
Databases are stored physically as files of records,which are typically stored on magnetic disks. This
chapter deals with the organization of databases in storage and the techniques for
accessing them efficiently using various algorithms, some of which require auxiliary
data structures called indexes. These structures are often referred to as physical
database file structures, and are at the physical level of three-schema architecture.
We start in Section 1 by introducing the concepts of computer storage hierarchies
and how they are used in database systems. Section 2 is devoted to a description of
magnetic disk storage devices and their characteristics, and we also briefly describe
magnetic tape storage devices. After discussing different storage technologies, we
turn our attention to the methods for physically organizing data on disks. Section 3
covers the technique of double buffering, which is used to speed retrieval of multi-
ple disk blocks. In Section 4 we discuss various ways of formatting and storing file
records on disk. Section 5 discusses the various types of operations that are typically
applied to file records. We present three primary methods for organizing file records
on disk: unordered records, in Section 6; ordered records, in Section 7; and hashed
records, in Section 8.
Section 9 briefly introduces files of mixed records and other primary methods for
organizing records, such as B-trees. These are particularly relevant for storage of
object-oriented databases. Section 10 describes RAID (Redundant Arrays of
Inexpensive (or Independent) Disks)—a data storage system architecture that is
commonly used in large organizations for better reliability and performance.
Finally, in Section 11 we describe three developments in the storage systems area:
storage area networks (SAN), network-attached storage (NAS), and iSCSI (Internet
From Chapter 17 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
Shamkant B. Navathe. Copyright © 2011 by Pearson Education, Inc. Published by Addison-
Wesley. All rights reserved.
588
Disk Storage, Basic File Structures, and Hashing
SCSI—Small Computer System Interface), the latest technology, which makes stor-
age area networks more affordable without the use of the Fiber Channel infrastruc-
ture and hence is getting very wide acceptance in industry. Section 12 summarizes
the chapter.
This chapter may be browsed through or even omitted by readers who have already
studied file organizations and indexing in a separate course. The material covered
here, in particular Sections 1 through 8, is necessary for understanding query pro-
cessing and optimization, and database tuning for improving performance of
queries.
1 Introduction
The collection of data that makes up a computerized database must be stored phys-
ically on some computer storage medium. The DBMS software can then retrieve,
update, and process this data as needed. Computer storage media form a storage
hierarchy that includes two main categories:
■ Primary storage. This category includes storage media that can be operated
on directly by the computer’s central processing unit (CPU), such as the com-
puter’s main memory and smaller but faster cache memories. Primary stor-
age usually provides fast access to data but is of limited storage capacity.
Although main memory capacities have been growing rapidly in recent
years, they are still more expensive and have less storage capacity than sec-
ondary and tertiary storage devices.
■ Secondary and tertiary storage. This category includes magnetic disks,
optical disks (CD-ROMs, DVDs, and other similar storage media), and
tapes. Hard-disk drives are classified as secondary storage, whereas remov-
able media such as optical disks and tapes are considered tertiary storage.
These devices usually have a larger capacity, cost less, and provide slower
access to data than do primary storage devices. Data in secondary or tertiary
storage cannot be processed directly by the CPU; first it must be copied into
primary storage and then processed by the CPU.
We first give an overview of the various storage devices used for primary and sec-
ondary storage in Section 1.1 and then discuss how databases are typically handled
in the storage hierarchy in Section 1.2.
1.1 Memory Hierarchies and Storage Devices
In a modern computer system, data resides and is transported throughout a hierar-
chy of storage media. The highest-speed memory is the most expensive and is there-
fore available with the least capacity. The lowest-speed memory is offline tape
storage, which is essentially available in indefinite storage capacity.
589
Disk Storage, Basic File Structures, and Hashing
At the primary storage level, the memory hierarchy includes at the most expensive
end, cache memory, which is a static RAM (Random Access Memory). Cache mem-
ory is typically used by the CPU to speed up execution of program instructions
using techniques such as prefetching and pipelining. The next level of primary stor-
age is DRAM (Dynamic RAM), which provides the main work area for the CPU for
keeping program instructions and data. It is popularly called main memory. The
advantage of DRAM is its low cost, which continues to decrease; the drawback is its
volatility1 and lower speed compared with static RAM. At the secondary and tertiary
storage level, the hierarchy includes magnetic disks, as well as mass storage in the
form of CD-ROM (Compact Disk–Read-Only Memory) and DVD (Digital Video
Disk or Digital Versatile Disk) devices, and finally tapes at the least expensive end of
the hierarchy. The storage capacity is measured in kilobytes (Kbyte or 1000 bytes),
megabytes (MB or 1 million bytes), gigabytes (GB or 1 billion bytes), and even ter-
abytes (1000 GB). The word petabyte (1000 terabytes or 10**15 bytes) is now
becoming relevant in the context of very large repositories of data in physics,
astronomy, earth sciences, and other scientific applications.
Programs reside and execute in DRAM. Generally, large permanent databases reside
on secondary storage, (magnetic disks), and portions of the database are read into
and written from buffers in main memory as needed. Nowadays, personal comput-
ers and workstations have large main memories of hundreds of megabytes of RAM
and DRAM, so it is becoming possible to load a large part of the database into main
memory. Eight to 16 GB of main memory on a single server is becoming common-
place. In some cases, entire databases can be kept in main memory (with a backup
copy on magnetic disk), leading to main memory databases; these are particularly
useful in real-time applications that require extremely fast response times. An
example is telephone switching applications, which store databases that contain
routing and line information in main memory.
Between DRAM and magnetic disk storage, another form of memory, flash mem-
ory, is becoming common, particularly because it is nonvolatile. Flash memories are
high-density, high-performance memories using EEPROM (Electrically Erasable
Programmable Read-Only Memory) technology. The advantage of flash memory is
the fast access speed; the disadvantage is that an entire block must be erased and
written over simultaneously. Flash memory cards are appearing as the data storage
medium in appliances with capacities ranging from a few megabytes to a few giga-
bytes. These are appearing in cameras, MP3 players, cell phones, PDAs, and so on.
USB (Universal Serial Bus) flash drives have become the most portable medium for
carrying data between personal computers; they have a flash memory storage device
integrated with a USB interface.
CD-ROM (Compact Disk – Read Only Memory) disks store data optically and are
read by a laser. CD-ROMs contain prerecorded data that cannot be overwritten.
WORM (Write-Once-Read-Many) disks are a form of optical storage used for
1Volatile memory typically loses its contents in case of a power outage, whereas nonvolatile memory
does not.
590
Disk Storage, Basic File Structures, and Hashing
archiving data; they allow data to be written once and read any number of times
without the possibility of erasing. They hold about half a gigabyte of data per disk
and last much longer than magnetic disks.2 Optical jukebox memories use an array
of CD-ROM platters, which are loaded onto drives on demand. Although optical
jukeboxes have capacities in the hundreds of gigabytes, their retrieval times are in
the hundreds of milliseconds, quite a bit slower than magnetic disks. This type of
storage is continuing to decline because of the rapid decrease in cost and increase in
capacities of magnetic disks. The DVD is another standard for optical disks allowing
4.5 to 15 GB of storage per disk. Most personal computer disk drives now read CD-
ROM and DVD disks. Typically, drives are CD-R (Compact Disk Recordable) that
can create CD-ROMs and audio CDs (Compact Disks), as well as record on DVDs.
Finally, magnetic tapes are used for archiving and backup storage of data. Tape
jukeboxes—which contain a bank of tapes that are catalogued and can be automat-
ically loaded onto tape drives—are becoming popular as tertiary storage to hold
terabytes of data. For example, NASA’s EOS (Earth Observation Satellite) system
stores archived databases in this fashion.
Many large organizations are already finding it normal to have terabyte-sized data-
bases. The term very large database can no longer be precisely defined because disk
storage capacities are on the rise and costs are declining. Very soon the term may be
reserved for databases containing tens of terabytes.
1.2 Storage of Databases
Databases typically store large amounts of data that must persist over long periods
of time, and hence is often referred to as persistent data. Parts of this data are
accessed and processed repeatedly during this period. This contrasts with the notion
of transient data that persist for only a limited time during program execution.
Most databases are stored permanently (or persistently) on magnetic disk secondary
storage, for the following reasons:
■ Generally, databases are too large to fit entirely in main memory.
■ The circumstances that cause permanent loss of stored data arise less fre-
quently for disk secondary storage than for primary storage. Hence, we refer
to disk—and other secondary storage devices—as nonvolatile storage,
whereas main memory is often called volatile storage.
■ The cost of storage per unit of data is an order of magnitude less for disk sec-
ondary storage than for primary storage.
Some of the newer technologies—such as optical disks, DVDs, and tape juke-
boxes—are likely to provide viable alternatives to the use of magnetic disks. In the
future, databases may therefore reside at different levels of the memory hierarchy
from those described in Section 1.1. However, it is anticipated that magnetic disks
2Their rotational speeds are lower (around 400 rpm), giving higher latency delays and low transfer rates
(around 100 to 200 KB/second).
591
Disk Storage, Basic File Structures, and Hashing
will continue to be the primary medium of choice for large databases for years to
come. Hence, it is important to study and understand the properties and character-
istics of magnetic disks and the way data files can be organized on disk in order to
design effective databases with acceptable performance.
Magnetic tapes are frequently used as a storage medium for backing up databases
because storage on tape costs even less than storage on disk. However, access to data
on tape is quite slow. Data stored on tapes is offline; that is, some intervention by an
operator—or an automatic loading device—to load a tape is needed before the data
becomes available. In contrast, disks are online devices that can be accessed directly
at any time.
The techniques used to store large amounts of structured data on disk are impor-
tant for database designers, the DBA, and implementers of a DBMS. Database
designers and the DBA must know the advantages and disadvantages of each stor-
age technique when they design, implement, and operate a database on a specific
DBMS. Usually, the DBMS has several options available for organizing the data. The
process of physical database design involves choosing the particular data organiza-
tion techniques that best suit the given application requirements from among the
options. DBMS system implementers must study data organization techniques so
that they can implement them efficiently and thus provide the DBA and users of the
DBMS with sufficient options.
Typical database applications need only a small portion of the database at a time for
processing. Whenever a certain portion of the data is needed, it must be located on
disk, copied to main memory for processing, and then rewritten to the disk if the
data is changed. The data stored on disk is organized as files of records. Each record
is a collection of data values that can be interpreted as facts about entities, their
attributes, and their relationships. Records should be stored on disk in a manner
that makes it possible to locate them efficiently when they are needed.
There are several primary file organizations, which determine how the file records
are physically placed on the disk, and hence how the records can be accessed. A heap file
(or unordered file) places the records on disk in no particular order by appending
new records at the end of the file, whereas a sorted file (or sequential file) keeps the
records ordered by the value of a particular field (called the sort key). A hashed file
uses a hash function applied to a particular field (called the hash key) to determine
a record’s placement on disk. Other primary file organizations, such as B-trees, use
tree structures. We discuss primary file organizations in Sections 6 through 9. A
secondary organization or auxiliary access structure allows efficient access to file
records based on alternate fields than those that have been used for the primary file
organization. Most of these exist as indexes.
2 Secondary Storage Devices
In this section we describe some characteristics of magnetic disk and magnetic tape
storage devices. Readers who have already studied these devices may simply browse
through this section.
592
Disk Storage, Basic File Structures, and Hashing
Actuator movement
Track
ArmActuator
Read/write
head Spindle Disk rotation
Cylinder
of tracks
(imaginary)
(a)
(b)
Figure 1
(a) A single-sided disk with read/write hardware.
(b) A disk pack with read/write hardware.
2.1 Hardware Description of Disk Devices
Magnetic disks are used for storing large amounts of data. The most basic unit of
data on the disk is a single bit of information. By magnetizing an area on disk in cer-
tain ways, one can make it represent a bit value of either 0 (zero) or 1 (one). To code
information, bits are grouped into bytes (or characters). Byte sizes are typically 4 to
8 bits, depending on the computer and the device. We assume that one character is
stored in a single byte, and we use the terms byte and character interchangeably. The
capacity of a disk is the number of bytes it can store, which is usually very large.
Small floppy disks used with microcomputers typically hold from 400 KB to 1.5
MB; they are rapidly going out of circulation. Hard disks for personal computers
typically hold from several hundred MB up to tens of GB; and large disk packs used
with servers and mainframes have capacities of hundreds of GB. Disk capacities
continue to grow as technology improves.
Whatever their capacity, all disks are made of magnetic material shaped as a thin
circular disk, as shown in Figure 1(a), and protected by a plastic or acrylic cover. A
593
Disk Storage, Basic File Structures, and Hashing
Track(a) Sector (arc of track)
(b)
Three sectors
Two sectors
One sector
Figure 2
Different sector organ-
izations on disk. (a)
Sectors subtending a
fixed angle. (b) Sectors
maintaining a uniform
recording
density.
disk is single-sided if it stores information on one of its surfaces only and double-
sided if both surfaces are used. To increase storage capacity, disks are assembled into
a disk pack, as shown in Figure 1(b), which may include many disks and therefore
many surfaces. Information is stored on a disk surface in concentric circles of small
width,3 each having a distinct diameter. Each circle is called a track. In disk packs,
tracks with the same diameter on the various surfaces are called a cylinder because
of the shape they would form if connected in space. The concept of a cylinder is
important because data stored on one cylinder can be retrieved much faster than if
it were distributed among different cylinders.
The number of tracks on a disk ranges from a few hundred to a few thousand, and
the capacity of each track typically ranges from tens of Kbytes to 150 Kbytes.
Because a track usually contains a large amount of information, it is divided into
smaller blocks or sectors. The division of a track into sectors is hard-coded on the
disk surface and cannot be changed. One type of sector organization, as shown in
Figure 2(a), calls a portion of a track that subtends a fixed angle at the center a sec-
tor. Several other sector organizations are possible, one of which is to have the sec-
tors subtend smaller angles at the center as one moves away, thus maintaining a
uniform density of recording, as shown in Figure 2(b). A technique called ZBR
(Zone Bit Recording) allows a range of cylinders to have the same number of sectors
per arc. For example, cylinders 0–99 may have one sector per track, 100–199 may
have two per track, and so on. Not all disks have their tracks divided into sectors.
The division of a track into equal-sized disk blocks (or pages) is set by the operat-
ing system during disk formatting (or initialization). Block size is fixed during ini-
tialization and cannot be changed dynamically. Typical disk block sizes range from
512 to 8192 bytes. A disk with hard-coded sectors often has the sectors subdivided
into blocks during initialization. Blocks are separated by fixed-size interblock gaps,
which include specially coded control information written during disk initializa-
tion. This information is used to determine which block on the track follows each
3In some disks, the circles are now connected into a kind of continuous spiral.
594
Disk Storage, Basic File Structures, and Hashing
Table 1 Specifications of Typical High-End Cheetah Disks from Seagate
Description Cheetah 15K.6 Cheetah NS 10K
Model Number ST3450856SS/FC ST3400755FC
Height 25.4 mm 26.11 mm
Width 101.6 mm 101.85 mm
Length 146.05 mm 147 mm
Weight 0.709 kg 0.771 kg
Capacity
Formatted Capacity 450 Gbytes 400 Gbytes
Configuration
Number of disks (physical) 4 4
Number of heads (physical) 8 8
Performance
Transfer Rates
Internal Transfer Rate (min) 1051 Mb/sec
Internal Transfer Rate (max) 2225 Mb/sec 1211 Mb/sec
Mean Time Between Failure (MTBF) 1.4 M hours
Seek Times
Avg. Seek Time (Read) 3.4 ms (typical) 3.9 ms (typical)
Avg. Seek Time (Write) 3.9 ms (typical) 4.2 ms (typical)
Track-to-track, Seek, Read 0.2 ms (typical) 0.35 ms (typical)
Track-to-track, Seek, Write 0.4 ms (typical) 0.35 ms (typical)
Average Latency 2 ms 2.98 msec
Courtesy Seagate Technology
interblock gap. Table 1 illustrates the specifications of typical disks used on large
servers in industry. The 10K and 15K prefixes on disk names refer to the rotational
speeds in rpm (revolutions per minute).
There is continuous improvement in the storage capacity and transfer rates associ-
ated with disks; they are also progressively getting cheaper—currently costing only a
fraction of a dollar per megabyte of disk storage. Costs are going down so rapidly
that costs as low 0.025 cent/MB—which translates to $0.25/GB and $250/TB—are
already here.
A disk is a random access addressable device. Transfer of data between main memory
and disk takes place in units of disk blocks. The hardware address of a block—a
combination of a cylinder number, track number (surface number within the cylin-
der on which the track is located), and block number (within the track) is supplied
to the disk I/O (input/output) hardware. In many modern disk drives, a single num-
ber called LBA (Logical Block Address), which is a number between 0 and n (assum-
ing the total capacity of the disk is n + 1 blocks), is mapped automatically to the
right block by the disk drive controller. The address of a buffer—a contiguous
595
reserved area in main storage that holds one disk block—is also provided. For a
read command, the disk block is copied into the buffer; whereas for a write com-
mand, the contents of the buffer are copied into the disk block. Sometimes several
contiguous blocks, called a cluster, may be transferred as a unit. In this case, the
buffer size is adjusted to match the number of bytes in the cluster.
The actual hardware mechanism that reads or writes a block is the disk read/write
head, which is part of a system called a disk drive. A disk or disk pack is mounted in
the disk drive, which includes a motor that rotates the disks. A read/write head
includes an electronic component attached to a mechanical arm. Disk packs with
multiple surfaces are controlled by several read/write heads—one for each surface,
as shown in Figure 1(b). All arms are connected to an actuator attached to another
electrical motor, which moves the read/write heads in unison and positions them
precisely over the cylinder of tracks specified in a block address.
Disk drives for hard disks rotate the disk pack continuously at a constant speed
(typically ranging between 5,400 and 15,000 rpm). Once the read/write head is
positioned on the right track and the block specified in the block address moves
under the read/write head, the electronic component of the read/write head is acti-
vated to transfer the data. Some disk units have fixed read/write heads, with as many
heads as there are tracks. These are called fixed-head disks, whereas disk units with
an actuator are called movable-head disks. For fixed-head disks, a track or cylinder
is selected by electronically switching to the appropriate read/write head rather than
by actual mechanical movement; consequently, it is much faster. However, the cost
of the additional read/write heads is quite high, so fixed-head disks are not com-
monly used.
A disk controller, typically embedded in the disk drive, controls the disk drive and
interfaces it to the computer system. One of the standard interfaces used today for
disk drives on PCs and workstations is called SCSI (Small Computer System
Interface). The controller accepts high-level I/O commands and takes appropriate
action to position the arm and causes the read/write action to take place. To transfer
a disk block, given its address, the disk controller must first mechanically position
the read/write head on the correct track. The time required to do this is called the
seek time. Typical seek times are 5 to 10 msec on desktops and 3 to 8 msecs on
servers. Following that, there is another delay—called the rotational delay or
latency—while the beginning of the desired block rotates into position under the
read/write head. It depends on the rpm of the disk. For example, at 15,000 rpm, the
time per rotation is 4 msec and the average rotational delay is the time per half rev-
olution, or 2 msec. At 10,000 rpm the average rotational delay increases to 3 msec.
Finally, some additional time is needed to transfer the data; this is called the block
transfer time. Hence, the total time needed to locate and transfer an arbitrary
block, given its address, is the sum of the seek time, rotational delay, and block
transfer time. The seek time and rotational delay are usually much larger than the
block transfer time. To make the transfer of multiple blocks more efficient, it is
common to transfer several consecutive blocks on the same track or cylinder. This
eliminates the seek time and rotational delay for all but the first block and can result
Disk Storage, Basic File Structures, and Hashing
596
Disk Storage, Basic File Structures, and Hashing
in a substantial saving of time when numerous contiguous blocks are transferred.
Usually, the disk manufacturer provides a bulk transfer rate for calculating the time
required to transfer consecutive blocks.
The time needed to locate and transfer a disk block is in the order of milliseconds,
usually ranging from 9 to 60 msec. For contiguous blocks, locating the first block
takes from 9 to 60 msec, but transferring subsequent blocks may take only 0.4 to 2
msec each. Many search techniques take advantage of consecutive retrieval of blocks
when searching for data on disk. In any case, a transfer time in the order of millisec-
onds is considered quite high compared with the time required to process data in
main memory by current CPUs. Hence, locating data on disk is a major bottleneck in
database applications. The file structures we discuss here attempt to minimize the
number of block transfers needed to locate and transfer the required data from disk
to main memory. Placing “related information” on contiguous blocks is the basic
goal of any storage organization on disk.
2.2 Magnetic Tape Storage Devices
Disks are random access secondary storage devices because an arbitrary disk block
may be accessed at random once we specify its address. Magnetic tapes are sequen-
tial access devices; to access the nth block on tape, first we must scan the preceding
n – 1 blocks. Data is stored on reels of high-capacity magnetic tape, somewhat sim-
ilar to audiotapes or videotapes. A tape drive is required to read the data from or
write the data to a tape reel. Usually, each group of bits that forms a byte is stored
across the tape, and the bytes themselves are stored consecutively on the tape.
A read/write head is used to read or write data on tape. Data records on tape are also
stored in blocks—although the blocks may be substantially larger than those for
disks, and interblock gaps are also quite large. With typical tape densities of 1600 to
6250 bytes per inch, a typical interblock gap4 of 0.6 inch corresponds to 960 to 3750
bytes of wasted storage space. It is customary to group many records together in one
block for better space utilization.
The main characteristic of a tape is its requirement that we access the data blocks in
sequential order. To get to a block in the middle of a reel of tape, the tape is
mounted and then scanned until the required block gets under the read/write head.
For this reason, tape access can be slow and tapes are not used to store online data,
except for some specialized applications. However, tapes serve a very important
function—backing up the database. One reason for backup is to keep copies of disk
files in case the data is lost due to a disk crash, which can happen if the disk
read/write head touches the disk surface because of mechanical malfunction. For
this reason, disk files are copied periodically to tape. For many online critical appli-
cations, such as airline reservation systems, to avoid any downtime, mirrored sys-
tems are used to keep three sets of identical disks—two in online operation and one
4Called interrecord gaps in tape terminology.
597
Disk Storage, Basic File Structures, and Hashing
as backup. Here, offline disks become a backup device. The three are rotated so that
they can be switched in case there is a failure on one of the live disk drives. Tapes can
also be used to store excessively large database files. Database files that are seldom
used or are outdated but required for historical record keeping can be archived on
tape. Originally, half-inch reel tape drives were used for data storage employing the
so-called 9 track tapes. Later, smaller 8-mm magnetic tapes (similar to those used in
camcorders) that can store up to 50 GB, as well as 4-mm helical scan data cartridges
and writable CDs and DVDs, became popular media for backing up data files from
PCs and workstations. They are also used for storing images and system libraries.
Backing up enterprise databases so that no transaction information is lost is a major
undertaking. Currently, tape libraries with slots for several hundred cartridges are
used with Digital and Superdigital Linear Tapes (DLTs and SDLTs) having capacities
in hundreds of gigabytes that record data on linear tracks. Robotic arms are used to
write on multiple cartridges in parallel using multiple tape drives with automatic
labeling software to identify the backup cartridges. An example of a giant library is
the SL8500 model of Sun Storage Technology that can store up to 70 petabytes
(petabyte = 1000 TB) of data using up to 448 drives with a maximum throughput
rate of 193.2 TB/hour. We defer the discussion of disk storage technology called
RAID, and of storage area networks, network-attached storage, and iSCSI storage
systems to the end of the chapter.
3 Buffering of Blocks
When several blocks need to be transferred from disk to main memory and all the
block addresses are known, several buffers can be reserved in main memory to
speed up the transfer. While one buffer is being read or written, the CPU can
process data in the other buffer because an independent disk I/O processor (con-
troller) exists that, once started, can proceed to transfer a data block between mem-
ory and disk independent of and in parallel to CPU processing.
Figure 3 illustrates how two processes can proceed in parallel. Processes A and B are
running concurrently in an interleaved fashion, whereas processes C and D are
running concurrently in a parallel fashion. When a single CPU controls multiple
processes, parallel execution is not possible. However, the processes can still run
concurrently in an interleaved way. Buffering is most useful when processes can run
concurrently in a parallel fashion, either because a separate disk I/O processor is
available or because multiple CPU processors exist.
Figure 4 illustrates how reading and processing can proceed in parallel when the
time required to process a disk block in memory is less than the time required to
read the next block and fill a buffer. The CPU can start processing a block once its
transfer to main memory is completed; at the same time, the disk I/O processor can
be reading and transferring the next block into a different buffer. This technique is
called double buffering and can also be used to read a continuous stream of blocks
from disk to memory. Double buffering permits continuous reading or writing of
data on consecutive disk blocks, which eliminates the seek time and rotational delay
598
Disk Storage, Basic File Structures, and Hashing
Interleaved concurrency
of operations A and B
Parallel execution of
operations C and D
t1
A A
B B
t2 t3 t4
Time
Figure 3
Interleaved concurrency
versus parallel execution.
i + 1
Process B
i + 2
Fill A
Time
i
Process A
i + 1
Fill B
Disk Block:
I/O:
Disk Block:
PROCESSING:
i
Fill A
i + 2
Process A
i + 3
Fill A
i + 4
Process A
i + 3
Process B
i + 4
Fill A
Figure 4
Use of two buffers, A and B, for reading from disk.
for all but the first block transfer. Moreover, data is kept ready for processing, thus
reducing the waiting time in the programs.
4 Placing File Records on Disk
In this section, we define the concepts of records, record types, and files. Then we
discuss techniques for placing file records on disk.
4.1 Records and Record Types
Data is usually stored in the form of records. Each record consists of a collection of
related data values or items, where each value is formed of one or more bytes and
corresponds to a particular field of the record. Records usually describe entities and
their attributes. For example, an EMPLOYEE record represents an employee entity,
and each field value in the record specifies some attribute of that employee, such as
Name, Birth_date, Salary, or Supervisor. A collection of field names and their corre-
599
Disk Storage, Basic File Structures, and Hashing
sponding data types constitutes a record type or record format definition. A data
type, associated with each field, specifies the types of values a field can take.
The data type of a field is usually one of the standard data types used in program-
ming. These include numeric (integer, long integer, or floating point), string of
characters (fixed-length or varying), Boolean (having 0 and 1 or TRUE and FALSE
values only), and sometimes specially coded date and time data types. The number
of bytes required for each data type is fixed for a given computer system. An integer
may require 4 bytes, a long integer 8 bytes, a real number 4 bytes, a Boolean 1 byte,
a date 10 bytes (assuming a format of YYYY-MM-DD), and a fixed-length string of
k characters k bytes. Variable-length strings may require as many bytes as there are
characters in each field value. For example, an EMPLOYEE record type may be
defined—using the C programming language notation—as the following structure:
struct employee{
char name[30];
char ssn[9];
int salary;
int job_code;
char department[20];
} ;
In some database applications, the need may arise for storing data items that consist
of large unstructured objects, which represent images, digitized video or audio
streams, or free text. These are referred to as BLOBs (binary large objects). A BLOB
data item is typically stored separately from its record in a pool of disk blocks, and a
pointer to the BLOB is included in the record.
4.2 Files, Fixed-Length Records,
and Variable-Length Records
A file is a sequence of records. In many cases, all records in a file are of the same
record type. If every record in the file has exactly the same size (in bytes), the file is
said to be made up of fixed-length records. If different records in the file have dif-
ferent sizes, the file is said to be made up of variable-length records. A file may have
variable-length records for several reasons:
■ The file records are of the same record type, but one or more of the fields are
of varying size (variable-length fields). For example, the Name field of
EMPLOYEE can be a variable-length field.
■ The file records are of the same record type, but one or more of the fields
may have multiple values for individual records; such a field is called a
repeating field and a group of values for the field is often called a repeating
group.
■ The file records are of the same record type, but one or more of the fields are
optional; that is, they may have values for some but not all of the file records
(optional fields).
600
Name = Smith, John Ssn = 123456789 DEPARTMENT = Computer
Smith, John
Name
1
(a)
(b)
(c)
1 12 21 25 29
Name Ssn Salary Job_code Department Hire_date
31 40 44 48 68
Ssn Salary Job_code Department
Separator Characters123456789 XXXX XXXX Computer
Separator Characters
Separates field name
from field value
Separates fields
Terminates record
=
Disk Storage, Basic File Structures, and Hashing
Figure 5
Three record storage formats. (a) A fixed-length record with six
fields and size of 71 bytes. (b) A record with two variable-length
fields and three fixed-length fields. (c) A variable-field record with
three types of separator characters.
■ The file contains records of different record types and hence of varying size
(mixed file). This would occur if related records of different types were
clustered (placed together) on disk blocks; for example, the GRADE_REPORT
records of a particular student may be placed following that STUDENT’s
record.
The fixed-length EMPLOYEE records in Figure 5(a) have a record size of 71 bytes.
Every record has the same fields, and field lengths are fixed, so the system can iden-
tify the starting byte position of each field relative to the starting position of the
record. This facilitates locating field values by programs that access such files. Notice
that it is possible to represent a file that logically should have variable-length records
as a fixed-length records file. For example, in the case of optional fields, we could
have every field included in every file record but store a special NULL value if no value
exists for that field. For a repeating field, we could allocate as many spaces in each
record as the maximum possible number of occurrences of the field. In either case,
space is wasted when certain records do not have values for all the physical spaces
provided in each record. Now we consider other options for formatting records of a
file of variable-length records.
601
Disk Storage, Basic File Structures, and Hashing
For variable-length fields, each record has a value for each field, but we do not know
the exact length of some field values. To determine the bytes within a particular
record that represent each field, we can use special separator characters (such as ? or
% or $)—which do not appear in any field value—to terminate variable-length
fields, as shown in Figure 5(b), or we can store the length in bytes of the field in the
record, preceding the field value.
A file of records with optional fields can be formatted in different ways. If the total
number of fields for the record type is large, but the number of fields that actually
appear in a typical record is small, we can include in each record a sequence of
pairs rather than just the field values. Three types of sep- pairs rather than pairs.
A repeating field needs one separator character to separate the repeating values of
4.3 Record Blocking and Spanned
The records of a file must be allocated to disk blocks because a block is the unit of
B − (bfr * R) bytes
5Other schemes are also possible for representing variable-length records.
602
Disk Storage, Basic File Structures, and Hashing
Record 1Block i Record 2 Record 3 Record 4 P
Record 4 (rest)Block i + 1 Record 5 Record 6 Record 7 P
Record 1Block i
(b)
(a) Record 2 Record 3
Record 4Block i + 1 Record 5 Record 6
Figure 6
each record start at a known location in the block, simplifying record processing. For
For variable-length records using spanned organization, each block may store a dif-
b = ⎡(r/bfr)⎤ blocks
4.4 Allocating File Blocks on Disk
4.5 File Headers
To search for a record on disk, one or more blocks are copied into main memory
603
Disk Storage, Basic File Structures, and Hashing
the file blocks. Each file block is copied into a buffer and searched until the record is
5 Operations on Files
Consider an EMPLOYEE file with fields Name, Ssn, Salary, Job_code, and Department.
Search operations on files are generally based on simple selection conditions. A
When several file records satisfy a search condition, the first record—with respect to
Actual operations for locating and accessing file records vary from system to system.
■ Open. Prepares the file for reading or writing. Allocates appropriate buffers
■ Reset. Sets the file pointer of an open file to the beginning of the file.
tion. Transfers the block containing that record into a main memory buffer
604
Disk Storage, Basic File Structures, and Hashing
and it becomes the current record. Sometimes, different verbs are used to
■ Read (or Get). Copies the current record from the buffer to a program vari-
■ FindNext. Searches for the next record in the file that satisfies the search
■ Delete. Deletes the current record and (eventually) updates the file on disk
■ Modify. Modifies some field values for the current record and (eventually)
■ Insert. Inserts a new record in the file by locating the block where the record
■ Close. Completes the file access by releasing the buffers and performing any
The preceding (except for Open and Close) are called record-at-a-time operations
■ Scan. If the file has just been opened or reset, Scan returns the first record;
In database systems, additional set-at-a-time higher-level operations may be
■ FindAll. Locates all the records in the file that satisfy a search condition.
tion and then continues to locate the next n – 1 records satisfying the same
■ FindOrdered. Retrieves all the records in the file in some specified order.
organizations require periodic reorganization. An example is to reorder the
605
Disk Storage, Basic File Structures, and Hashing
At this point, it is worthwhile to note the difference between the terms file organiza-
Usually, we expect to use some search conditions more than others. Some files may
If users expect mainly to apply a search condition based on Ssn, the designer must
In the following sections, we discuss methods for organizing records of a file on
6 Files of Unordered Records (Heap Files)
606
Disk Storage, Basic File Structures, and Hashing
file. Such an organization is called a heap or pile file.6 This organization is often
Inserting a new record is very efficient. The last disk block of the file is copied into a
To delete a record, a program must first find its block, copy the block into a buffer,
We can use either spanned or unspanned organization for an unordered file, and it
To read all records in order of the values of some field, we create a sorted copy of the
For a file of unordered fixed-length records using unspanned blocks and contiguous
6Sometimes this organization is called a sequential file.
607
Disk Storage, Basic File Structures, and Hashing
7 Files of Ordered Records (Sorted Files)
If the ordering field is also a key field of the file—a field guaranteed to have a
Ordered records have some advantages over unordered files. First, reading the records
A binary search for disk files can be done on the blocks rather than on the records.
Algorithm 1. Binary Search on an Ordering Key of a Disk File
begin i ← (l + u) div 2;
then l ← i + 1
then goto found
goto notfound;
A search criterion involving the conditions >, <, ≥, and ≤ on the ordering field
is quite efficient, since the physical ordering of records means that all records
7The term sequential file has also been used to refer to unordered files, although it is more appropriate
for ordered files.
608
Name
Aaron, Ed
Abbott, Diane
Block 1
Acosta, Marc
Ssn Birth_date
...
Job Salary Sex
...
Adams, John
Adams, Robin
Block 2
Akers, Jan
...
Alexander, Ed
Alfred, Bob
Block 3
Allen, Sam
...
Allen, Troy
Anders, Keith
Block 4
Anderson, Rob
...
Anderson, Zach
Angeli, Joe
Block 5
Archer, Sue
...
Arnold, Mack
Arnold, Steven
Block 6
Atkins, Timothy
Wong, James
Wood, Donald
Block n–1
Woods, Manny
...
Wright, Pam
Wyatt, Charles
Block n
Zimmer, Byron
...
Figure 7
Some blocks of an ordered
(sequential) file of EMPLOYEE
records with Name as the
ordering key field.
Disk Storage, Basic File Structures, and Hashing
satisfying the condition are contiguous in the file. For example, referring to Figure 7,
if the search criterion is (Name < ‘G’)—where < means alphabetically before—the
records satisfying the search criterion are those from the beginning of the file up to
the first record that has a Name value starting with the letter ‘G’.
609
Ordering does not provide any advantages for random or ordered access of the
records based on values of the other nonordering fields of the file. In these cases, we
do a linear search for random access. To access the records in order based on a
nonordering field, it is necessary to create another sorted copy—in a different
order—of the file.
Inserting and deleting records are expensive operations for an ordered file because
the records must remain physically ordered. To insert a record, we must find its cor-
rect position in the file, based on its ordering field value, and then make space in the
file to insert the record in that position. For a large file this can be very time-
consuming because, on the average, half the records of the file must be moved to
make space for the new record. This means that half the file blocks must be read and
rewritten after records are moved among them. For record deletion, the problem is
less severe if deletion markers and periodic reorganization are used.
One option for making insertion more efficient is to keep some unused space in each
block for new records. However, once this space is used up, the original problem
resurfaces. Another frequently used method is to create a temporary unordered file
called an overflow or transaction file. With this technique, the actual ordered file is
called the main or master file. New records are inserted at the end of the overflow file
rather than in their correct position in the main file. Periodically, the overflow file is
sorted and merged with the master file during file reorganization. Insertion becomes
very efficient, but at the cost of increased complexity in the search algorithm. The
overflow file must be searched using a linear search if, after the binary search, the
record is not found in the main file. For applications that do not require the most up-
to-date information, overflow records can be ignored during a search.
Modifying a field value of a record depends on two factors: the search condition to
locate the record and the field to be modified. If the search condition involves the
ordering key field, we can locate the record using a binary search; otherwise we must
do a linear search. A nonordering field can be modified by changing the record and
rewriting it in the same physical location on disk—assuming fixed-length records.
Modifying the ordering field means that the record can change its position in the
file. This requires deletion of the old record followed by insertion of the modified
record.
Reading the file records in order of the ordering field is quite efficient if we ignore
the records in overflow, since the blocks can be read consecutively using double
buffering. To include the records in overflow, we must merge them in their correct
positions; in this case, first we can reorganize the file, and then read its blocks
sequentially. To reorganize the file, first we sort the records in the overflow file, and
then merge them with the master file. The records marked for deletion are removed
during the reorganization.
Table 2 summarizes the average access time in block accesses to find a specific
record in a file with b blocks.
Ordered files are rarely used in database applications unless an additional access
path, called a primary index, is used; this results in an indexed-sequential file. This
Disk Storage, Basic File Structures, and Hashing
610
Disk Storage, Basic File Structures, and Hashing
Table 2 Average Access Times for a File of b Blocks under Basic File Organizations
Average Blocks to Access
Type of Organization Access/Search Method a Specific Record
Heap (unordered) Sequential scan (linear search) b/2
Ordered Sequential scan b/2
Ordered Binary search log2 b
further improves the random access time on the ordering key field. If the ordering
attribute is not a key, the file is called a clustered file.
8 Hashing Techniques
Another type of primary file organization is based on hashing, which provides very
fast access to records under certain search conditions. This organization is usually
called a hash file.8 The search condition must be an equality condition on a single
field, called the hash field. In most cases, the hash field is also a key field of the file,
in which case it is called the hash key. The idea behind hashing is to provide a func-
tion h, called a hash function or randomizing function, which is applied to the
hash field value of a record and yields the address of the disk block in which the
record is stored. A search for the record within the block can be carried out in a
main memory buffer. For most records, we need only a single-block access to
retrieve that record.
Hashing is also used as an internal search structure within a program whenever a
group of records is accessed exclusively by using the value of one field. We describe
the use of hashing for internal files in Section 8.1; then we show how it is modified
to store external files on disk in Section 8.2. In Section 8.3 we discuss techniques for
extending hashing to dynamically growing files.
8.1 Internal Hashing
For internal files, hashing is typically implemented as a hash table through the use
of an array of records. Suppose that the array index range is from 0 to M – 1, as
shown in Figure 8(a); then we have M slots whose addresses correspond to the array
indexes. We choose a hash function that transforms the hash field value into an inte-
ger between 0 and M − 1. One common hash function is the h(K) = K mod M func-
tion, which returns the remainder of an integer hash field value K after division by
M; this value is then used for the record address.
8A hash file has also been called a direct file.
611
Disk Storage, Basic File Structures, and Hashing
Noninteger hash field values can be transformed into integers before the mod
function is applied. For character strings, the numeric (ASCII) codes associated
with characters can be used in the transformation—for example, by multiplying
those code values. For a hash field whose data type is a string of 20 characters,
Algorithm 2(a) can be used to calculate the hash address. We assume that the code
function returns the numeric code of a character and that we are given a hash field
value K of type K: array [1..20] of char (in Pascal) or char K[20] (in C).
(a)
–1
–1
–1
M + 2
M
0
1
2
3
M – 2
M – 1
Data fields Overflow pointer
Address space
Overflow space
M + 1
M + 5
–1
M + 4
–1
M + 0 – 2
M + 0 – 1
null pointer = –1
overflow pointer refers to position of next record in linked list
M – 2
M
M + 1
M + 2
M – 1
Name Ssn Job Salary
(b) 0
1
2
3
4
...
Figure 8
Internal hashing data structures. (a) Array
of M positions for use in internal hashing.
(b) Collision resolution by chaining records.
612
Disk Storage, Basic File Structures, and Hashing
Algorithm 2. Two simple hashing algorithms: (a) Applying the mod hash
function to a character string K. (b) Collision resolution by open addressing.
(a) temp ← 1;
for i ← 1 to 20 do temp ← temp * code(K[i ] ) mod M ;
hash_address ← temp mod M;
(b) i ← hash_address(K); a ← i;
if location i is occupied
then begin i ← (i + 1) mod M;
while (i ≠ a) and location i is occupied
do i ← (i + 1) mod M;
if (i = a) then all positions are full
else new_hash_address ← i;
end;
Other hashing functions can be used. One technique, called folding, involves apply-
ing an arithmetic function such as addition or a logical function such as exclusive or
to different portions of the hash field value to calculate the hash address (for exam-
ple, with an address space from 0 to 999 to store 1,000 keys, a 6-digit key 235469
may be folded and stored at the address: (235+964) mod 1000 = 199). Another tech-
nique involves picking some digits of the hash field value—for instance, the third,
fifth, and eighth digits—to form the hash address (for example, storing 1,000
employees with Social Security numbers of 10 digits into a hash file with 1,000 posi-
tions would give the Social Security number 301-67-8923 a hash value of 172 by this
hash function).9 The problem with most hashing functions is that they do not guar-
antee that distinct values will hash to distinct addresses, because the hash field
space—the number of possible values a hash field can take—is usually much larger
than the address space—the number of available addresses for records. The hashing
function maps the hash field space to the address space.
A collision occurs when the hash field value of a record that is being inserted hashes
to an address that already contains a different record. In this situation, we must
insert the new record in some other position, since its hash address is occupied. The
process of finding another position is called collision resolution. There are numer-
ous methods for collision resolution, including the following:
■ Open addressing. Proceeding from the occupied position specified by the
hash address, the program checks the subsequent positions in order until an
unused (empty) position is found. Algorithm 2(b) may be used for this pur-
pose.
■ Chaining. For this method, various overflow locations are kept, usually by
extending the array with a number of overflow positions. Additionally, a
pointer field is added to each record location. A collision is resolved by plac-
ing the new record in an unused overflow location and setting the pointer of
the occupied hash address location to the address of that overflow location.
9A detailed discussion of hashing functions is outside the scope of our presentation.
613
Disk Storage, Basic File Structures, and Hashing
A linked list of overflow records for each hash address is thus maintained, as
shown in Figure 8(b).
■ Multiple hashing. The program applies a second hash function if the first
results in a collision. If another collision results, the program uses open
addressing or applies a third hash function and then uses open addressing if
necessary.
Each collision resolution method requires its own algorithms for insertion,
retrieval, and deletion of records. The algorithms for chaining are the simplest.
Deletion algorithms for open addressing are rather tricky. Data structures textbooks
discuss internal hashing algorithms in more detail.
The goal of a good hashing function is to distribute the records uniformly over the
address space so as to minimize collisions while not leaving many unused locations.
Simulation and analysis studies have shown that it is usually best to keep a hash
table between 70 and 90 percent full so that the number of collisions remains low
and we do not waste too much space. Hence, if we expect to have r records to store
in the table, we should choose M locations for the address space such that (r/M) is
between 0.7 and 0.9. It may also be useful to choose a prime number for M, since it
has been demonstrated that this distributes the hash addresses better over the
address space when the mod hashing function is used. Other hash functions may
require M to be a power of 2.
8.2 External Hashing for Disk Files
Hashing for disk files is called external hashing. To suit the characteristics of disk
storage, the target address space is made of buckets, each of which holds multiple
records. A bucket is either one disk block or a cluster of contiguous disk blocks. The
hashing function maps a key into a relative bucket number, rather than assigning an
absolute block address to the bucket. A table maintained in the file header converts
the bucket number into the corresponding disk block address, as illustrated in
Figure 9.
The collision problem is less severe with buckets, because as many records as will fit
in a bucket can hash to the same bucket without causing problems. However, we
must make provisions for the case where a bucket is filled to capacity and a new
record being inserted hashes to that bucket. We can use a variation of chaining in
which a pointer is maintained in each bucket to a linked list of overflow records for
the bucket, as shown in Figure 10. The pointers in the linked list should be record
pointers, which include both a block address and a relative record position within
the block.
Hashing provides the fastest possible access for retrieving an arbitrary record given
the value of its hash field. Although most good hash functions do not maintain
records in order of hash field values, some functions—called order preserving—
do. A simple example of an order preserving hash function is to take the leftmost
three digits of an invoice number field that yields a bucket address as the hash
address and keep the records sorted by invoice number within each bucket. Another
614
Disk Storage, Basic File Structures, and Hashing
0
1
2
M – 2
M – 1
Bucket
Number Block address on disk
Figure 9
Matching bucket numbers to disk
block addresses.
example is to use an integer hash key directly as an index to a relative file, if the hash
key values fill up a particular interval; for example, if employee numbers in a com-
pany are assigned as 1, 2, 3, ... up to the total number of employees, we can use the
identity hash function that maintains order. Unfortunately, this only works if keys
are generated in order by some application.
The hashing scheme described so far is called static hashing because a fixed number
of buckets M is allocated. This can be a serious drawback for dynamic files. Suppose
that we allocate M buckets for the address space and let m be the maximum number
of records that can fit in one bucket; then at most (m * M) records will fit in the allo-
cated space. If the number of records turns out to be substantially fewer than
(m * M), we are left with a lot of unused space. On the other hand, if the number of
records increases to substantially more than (m * M), numerous collisions will
result and retrieval will be slowed down because of the long lists of overflow
records. In either case, we may have to change the number of blocks M allocated and
then use a new hashing function (based on the new value of M) to redistribute the
records. These reorganizations can be quite time-consuming for large files. Newer
dynamic file organizations based on hashing allow the number of buckets to vary
dynamically with only localized reorganization (see Section 8.3).
When using external hashing, searching for a record given a value of some field
other than the hash field is as expensive as in the case of an unordered file. Record
deletion can be implemented by removing the record from its bucket. If the bucket
has an overflow chain, we can move one of the overflow records into the bucket to
replace the deleted record. If the record to be deleted is already in overflow, we sim-
ply remove it from the linked list. Notice that removing an overflow record implies
that we should keep track of empty positions in overflow. This is done easily by
maintaining a linked list of unused overflow locations.
615
Disk Storage, Basic File Structures, and Hashing
Bucket 0
Main buckets
Overflow buckets
340
460
Record pointer
NULL
NULL
NULL
Bucket 1 321
761
91
Record pointer
981
182
Record pointer
(Pointers are to records within the overflow blocks)
Record pointer
Record pointer
652 Record pointer
Record pointer
Record pointer
Bucket 2 22
72
522
Record pointer
Bucket 9 399
89
Record pointer
NULL
...
Figure 10
Handling overflow for buckets
by chaining.
Modifying a specific record’s field value depends on two factors: the search condi-
tion to locate that specific record and the field to be modified. If the search condi-
tion is an equality comparison on the hash field, we can locate the record efficiently
by using the hashing function; otherwise, we must do a linear search. A nonhash
field can be modified by changing the record and rewriting it in the same bucket.
Modifying the hash field means that the record can move to another bucket, which
requires deletion of the old record followed by insertion of the modified record.
8.3 Hashing Techniques That Allow Dynamic File Expansion
A major drawback of the static hashing scheme just discussed is that the hash
address space is fixed. Hence, it is difficult to expand or shrink the file dynamically.
The schemes described in this section attempt to remedy this situation. The first
scheme—extendible hashing—stores an access structure in addition to the file, and
616
Disk Storage, Basic File Structures, and Hashing
hence is somewhat similar to indexing. The main difference is that the access struc-
ture is based on the values that result after application of the hash function to the
search field. In indexing, the access structure is based on the values of the search
field itself. The second technique, called linear hashing, does not require additional
access structures. Another scheme, called dynamic hashing, uses an access structure
based on binary tree data structures..
These hashing schemes take advantage of the fact that the result of applying a hash-
ing function is a nonnegative integer and hence can be represented as a binary num-
ber. The access structure is built on the binary representation of the hashing
function result, which is a string of bits. We call this the hash value of a record.
Records are distributed among buckets based on the values of the leading bits in
their hash values.
Extendible Hashing. In extendible hashing, a type of directory—an array of 2d
bucket addresses—is maintained, where d is called the global depth of the direc-
tory. The integer value corresponding to the first (high-order) d bits of a hash value
is used as an index to the array to determine a directory entry, and the address in
that entry determines the bucket in which the corresponding records are stored.
However, there does not have to be a distinct bucket for each of the 2d directory
locations. Several directory locations with the same first d� bits for their hash values
may contain the same bucket address if all the records that hash to these locations fit
in a single bucket. A local depth d��—stored with each bucket—specifies the number
of bits on which the bucket contents are based. Figure 11 shows a directory with
global depth d = 3.
The value of d can be increased or decreased by one at a time, thus doubling or halv-
ing the number of entries in the directory array. Doubling is needed if a bucket,
whose local depth d� is equal to the global depth d, overflows. Halving occurs if d >
To illustrate bucket splitting, suppose that a new inserted record causes overflow in
If a bucket that overflows and is split used to have a local depth d�equal to the global
617
Disk Storage, Basic File Structures, and Hashing
Global depth
000
001
010
011
100
101
110
111
d´ = 3 Bucket for records
Directory Data file buckets
d´ = 3 Bucket for records
d´ = 2 Bucket for records
d´ = 2 Bucket for records
d´ = 3 Bucket for records
d´ = 3 Bucket for records
Figure 11
split into two locations, both of which have the same pointer value as did the origi-
The main advantage of extendible hashing that makes it attractive is that the per-
618
Disk Storage, Basic File Structures, and Hashing
increases the average number of accesses per key. Additionally, no space is allocated
Dynamic Hashing. A precursor to extendible hashing was dynamic hashing, in
■ Internal nodes that have two pointers—the left pointer corresponding to the
■ Leaf nodes—these hold a pointer to the actual bucket with records.
An example of the dynamic hashing appears in Figure 12. Four buckets are shown
Linear Hashing. The idea behind linear hashing is to allow a hash file to expand
619
Disk Storage, Basic File Structures, and Hashing
Data File Buckets
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Directory
1
0
1
0
1
0
1
0
1
internal directory node
leaf directory node
Figure 12
functions hi and hi+1 is that any records that hashed to bucket 0 based on hi will hash
As further collisions lead to overflow records, additional buckets are split in the.
To create a primary index on the ordered file shown in Figure A.1 (at the end of this
Figure 1 illustrates this primary index. The total number of entries in the index is
Indexes can also be characterized as dense or sparse. A dense index has an index
The index file for a primary index occupies a much smaller space than does the data
2We can use a scheme similar to the one described here, with the last record in each block (rather than
638
Indexing Structures for Files
Index file entries)
Block anchor
value
pointer
(Primary
Name
Aaron, Ed
…
…
…
…
…
Acosta, Marc
Adams, John
Adams, Robin
Akers, Jan
Alexander, Ed
Alfred, Bob
Allen, Sam
Allen, Troy
Anders, Keith
Anderson, Rob
Anderson, Zach
Angel, Joe
Archer, Sue
Arnold, Mack
Arnold, Steven
Atkins, Timothy
Wong, James
Wood, Donald
Woods, Manny
Wright, Pam
Wyatt, Charles
Zimmer, Byron
Aaron, Ed
Adams, John
Alexander, Ed
Allen, Troy
Anderson, Zach
Arnold, Mack
Wong, James
Wright, Pam
…
…
…
. .
. .
Ssn Birth_date Job Salary Sex
Figure 1
639
Indexing Structures for Files
value requires a binary search of that index and access to the block containing that
A record whose primary key value is K lies in the block whose address is P(i), where entries)
Clustering
Block
Figure 2
Indexing Structures for Files
Again we refer to the two field values of index entry i as . The entries are
642
Indexing Structures for Files
Data file
Block pointer
(Clustering
Dept_number
1
1
Name Ssn Birth_date SalaryJob
Block pointer
2
Block pointer
3
3
Block pointer
3
Block pointer
4
Block pointer
5
Block pointer
6
Block pointer
6
Block pointer
8
8
NULL pointer
NULL pointer
NULL pointer
NULL pointer
NULL pointer
NULL pointer
Index file entries)
Clustering
Block
Figure 3
643
Indexing Structures for Files
Data file
Indexing field
6
3
17
9
13
21
11
16
24
10
20
4
23
18
12
7
19
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20
21
22
24
Index file entries)
Index
Block
Figure 4
file, rather than for each block, as in the case of a primary index. Figure 4 illustrates a
644
Indexing Structures for Files
A secondary index usually needs more storage space and longer search time than
Example 2. Consider the file of Example 1 with r = 30,000 fixed-length records of
A binary search on this secondary index needs ⎡(log2bi)⎤ = ⎡(log2442)⎤ = 9 block
We can also create a secondary index on a nonkey, nonordering field of a file. In this
■ Option 1 is to include duplicate index entries with the same K(i) value—one
■ Option 2 is to have variable-length records for the index entries, with a
■ Option 3, which is more commonly used, is to keep the index entries them- points to a
645
Indexing Structures for Files
Data file
(Indexing field)
Dept_number
3
5
1
Name Ssn Birth_date SalaryJob
2
3
4
8
6
4
1
6
5
2
5
5
1
6
3
6
3
8
3
1
2
3
4
5
6
Blocks of
pointers
Index file
( entries)
Field
Block
Figure 5
used. This technique is illustrated in Figure 5. Retrieval via the index requires
646
Notice that a secondary index provides a logical ordering on the records by the
1.4 Summary
Indexing Structures for Files
Table 1 Types of Indexes Based on the Properties of the Indexing Field
Index Field Used
Index Field Not Used
Indexing field is key Primary index Secondary index (Key)
Indexing field is nonkey Clustering index Secondary index (NonKey)
Table 2 Properties of Index Types
Type of Index
Dense or Nondense
Block Anchoring
Primary Number of blocks in
Nondense Yes
Clustering Number of distinct
Nondense Yes/noa
Secondary (key) Number of records in
Dense No
Secondary (nonkey) Number of recordsb or
Dense or Nondense No
aYes if every distinct value of the ordering field starts a new block; no otherwise.
647
Indexing Structures for Files
2 Multilevel Indexes
A multilevel index considers the index file, which we will now refer to as the first (or
We can repeat this process for the second level. The third level, which is a primary
t)) to calculate t. Hence, a multilevel index with r1 first-level
4The numbering scheme for index levels used here is the reverse of the way levels are commonly
648
Indexing Structures for Files
index, a single disk block is retrieved at each level. Hence, t disk blocks are accessed
The multilevel scheme described here can be used on any type of index—whether it
Example 3. Suppose that the dense secondary index of Example 2 is converted into
Notice that we could also have a multilevel primary index, which would be non-
A common file organization used in business data processing is an ordered file with
Algorithm 1 outlines the search procedure for a record in a data file that uses a non-, and we search for a record whose primary key value is K. We, where q ≤ p. Each Pi is a
652
Indexing Structures for Files
P1
P1
K1 Ki–1
Kq–1 < X
X
Ki–1 < X < Ki
X
X < K1
X
Pi Ki Kq–1 Pq. . . . . .Figure 8
A node in a search
tree with pointers to
subtrees below it.
5
3
Tree node pointer
Null tree pointer
6 9
7 8 121
Figure 9
A search tree of
order p = 3.
ordered set of values. All search values are assumed to be unique.6 Figure 8 illus-
trates a node in a search tree. Two constraints must hold at all times on the search
tree:
1. Within each node, K1 < K2 < ... < Kq−1.
2. For all values X in the subtree pointed at by Pi, we have Ki−1 < X < Ki for 1 <
i < q; X < Ki for i = 1; and Ki−1 < X for i = q (see Figure 8).
Whenever we search for a value X, we follow the appropriate pointer Pi according to
the formulas in condition 2 above. Figure 9 illustrates a search tree of order p = 3
and integer search values. Notice that some of the pointers Pi in a node may be NULL
pointers.
We can use a search tree as a mechanism to search for records stored in a disk file.
The values in the tree can be the values of one of the fields of the file, called
the search field (which is the same as the index field if a multilevel index guides the
search). Each key value in the tree is associated with a pointer to the record in the
data file having that value. Alternatively, the pointer could be to the disk block con-
taining that record. The search tree itself can be stored on disk by assigning each tree
node to a disk block. When a new record is inserted in the file, we must update the
search tree by inserting an entry in the tree containing the search field value of the
new record and a pointer to the new record.
6This restriction can be relaxed. If the index is on a nonkey field, duplicate search values may exist and
the node structure and the navigation rules for the tree may be modified.
653
Indexing Structures for Files
Algorithms are necessary for inserting and deleting search values into and from the
search tree while maintaining the preceding two constraints. In general, these algo-
rithms do not guarantee that a search tree is balanced, meaning that all of its leaf
nodes are at the same level.7 The tree in Figure 7 is not balanced because it has leaf
nodes at levels 1, 2, and 3. The goals for balancing a search tree are as follows:
■ To guarantee that nodes are evenly distributed, so that the depth of the tree
is minimized for the given set of keys and that the tree does not get skewed
with some nodes being at very deep levels
■ To make the search speed uniform, so that the average time to find any ran-
dom key is roughly the same
While minimizing the number of levels in the tree is one goal, another implicit goal
is to make sure that the index tree does not need too much restructuring as records
are inserted into and deleted from the main file. Thus we want the nodes to be as full
as possible and do not want any nodes to be empty if there are too many deletions.
Record deletion may leave some nodes in the tree nearly empty, thus wasting storage
space and increasing the number of levels. The B-tree addresses both of these prob-
lems by specifying additional constraints on the search tree.
B-Trees. The B-tree has additional constraints that ensure that the tree is always
balanced and that the space wasted by deletion, if any, never becomes excessive. The
algorithms for insertion and deletion, though, become more complex in order to
maintain these constraints. Nonetheless, most insertions and deletions are simple
processes; they become complicated only under special circumstances—namely,
whenever we attempt an insertion into a node that is already full or a deletion from
a node that makes it less than half full. More formally, a B-tree of order p, when
used as an access structure on a key field to search for records in a data file, can be
defined as follows:
1. Each internal node in the B-tree (Figure 10(a)) is of the form
, P2, , …, , Pq>
where q ≤ p. Each Pi is a tree pointer—a pointer to another node in the B-
8—a pointer to the record whose search key
2. Within each node, K1 < K2 < ... < Kq−1.
3. For all search key field values X in the subtree pointed at by Pi (the ith sub-
tree, see Figure 10(a)), we have:
Ki–1 < X < Ki for 1 < i < q; X < Ki for i = 1; and Ki–1 < X for i = q.
4. Each node has at most p tree pointers.
7The definition of balanced is different for binary trees. Balanced binary trees are known as AVL trees.
8A data pointer is either a block address or a record address; the latter is essentially a block address and
a record offset within the block.
654
Indexing Structures for Files
X Tree
pointer
Tree
pointer
Tree
pointer
(a)
(b)
P2
X
Data
pointer
Data
pointer
Data
pointer
5 o 8 Tree node pointero
6 o 7 o 9 o 12 o
Data pointer
Null tree pointer
1 o 3 o
Data
pointer
P1 Pr1K1 Ki–1 Prq–1Kq–1
X
Kq–1 < XKi–1 < X < Ki X < K1
Pi Pq. . . . . .Pri–1 Ki Pri
Tree
pointer
o
Figure 10
B-tree structures. (a) A node in a B-tree with q – 1 search values. (b) A B-tree
of order p = 3.The values were inserted in the order 8, 5, 1, 7, 3, 12, 9, 6.
5. Each node, except the root and leaf nodes, has at least ⎡(p/2)⎤ tree pointers.
The root node has at least two tree pointers unless it is the only node in the
tree.
6. A node with q tree pointers, q ≤ p, has q – 1 search key field values (and hence
has q – 1 data pointers).
7. All leaf nodes are at the same level. Leaf nodes have the same structure as
internal nodes except that all of their tree pointers Pi are NULL.
Figure 10(b) illustrates a B-tree of order p = 3. Notice that all search values K in the
B-tree are unique because we assumed that the tree is used as an access structure on
a key field. If we use a B-tree on a nonkey field, we must change the definition of the
file pointers Pri to point to a block—or a cluster of blocks—that contain the point-
ers to the file records. This extra level of indirection is similar to option 3, discussed
in Section 1.3, for secondary indexes.
A B-tree starts with a single root node (which is also a leaf node) at level 0 (zero).
Once the root node is full with p – 1 search key values and we attempt to insert
another entry in the tree, the root node splits into two nodes at level 1. Only the
middle value is kept in the root node, and the rest of the values are split evenly
655
Indexing Structures for Files
between the other two nodes. When a nonroot node is full and a new entry is
inserted into it, that node is split into two nodes at the same level, and the middle
entry is moved to the parent node along with two pointers to the new split nodes. If
the parent node is full, it is also split. Splitting can propagate all the way to the root
node, creating a new level if the root is split. We do not discuss algorithms for B-
trees in detail in this book,9 but we outline search and insertion procedures for
B+-trees in the next section.
If deletion of a value causes a node to be less than half full, it is combined with its
neighboring nodes, and this can also propagate all the way to the root. Hence, dele-
tion can reduce the number of tree levels. It has been shown by analysis and simula-
tion that, after numerous random insertions and deletions on a B-tree, the nodes
are approximately 69 percent full when the number of values in the tree stabilizes.
This is also true of B+-trees. If this happens, node splitting and combining will
occur only rarely, so insertion and deletion become quite efficient. If the number of
values grows, the tree will expand without a problem—although splitting of nodes
may occur, so some insertions will take more time. Each B-tree node can have at
most p tree pointers, p – 1 data pointers, and p – 1 search key field values (see Figure
10(a)).
In general, a B-tree node may contain additional information needed by the algo-
rithms that manipulate the tree, such as the number of entries q in the node and a
pointer to the parent node. Next, we illustrate how to calculate the number of blocks
and levels for a B-tree.
Example 4. Suppose that the search field is a nonordering key field, and we con-
struct a B-tree on this field with p = 23. Assume that each node of the B-tree is 69
percent full. Each node, on the average, will have p * 0.69 = 23 * 0.69 or approxi-
mately 16 pointers and, hence, 15 search key field values. The average fan-out fo =
16. We can start at the root and see how many values and pointers can exist, on the
average, at each subsequent level:
Root: 1 node 15 key entries 16 pointers
Level 1: 16 nodes 240 key entries 256 pointers
Level 2: 256 nodes 3840 key entries 4096 pointers
Level 3: 4096 nodes 61,440 key entries
At each level, we calculated the number of key entries by multiplying the total num-
ber of pointers at the previous level by 15, the average number of entries in each
node. Hence, for the given block size, pointer size, and search key field size, a two-
level B-tree holds 3840 + 240 + 15 = 4095 entries on the average; a three-level B-tree
holds 65,535 entries on the average.
B-trees are sometimes used as primary file organizations. In this case, whole records
are stored within the B-tree nodes rather than just the
9For details on insertion and deletion algorithms for B-trees, consult Ramakrishnan and Gehrke [2003].
656
Indexing Structures for Files
record size. Otherwise, the fan-out and the number of levels become too great to
In summary, B-trees provide a multilevel access structure that is a balanced tree
3.2 B+-Trees
The leaf nodes of the B+-tree are usually linked to provide ordered access on the
1. Each internal node is of the form
where q ≤ p and each Pi is a tree pointer., , …, , Pnext>
where q ≤ p, each Pri is a data pointer, and Pnext points to the next leaf node of
10Our definition follows Knuth (1998). One can define a B+-tree differently by exchanging the < and �
symbols (K
i−1 � X < Ki; Kq−1 ≤ X), but the principles remain the same.
657
Indexing Structures for Files
(b) Pointer to
next leaf
node in
tree
Data
pointer
Data
pointer
Data
pointer
Data
pointer
Pr1K1 Pr2K2 Pri Prq–1 PnextKi Kq–1. . . . . .
(a) P1 K1 Ki–1
Kq–1 < X
XX
X < K1
X
Pi Ki Kq–1 Pq. . . . . .
Tree
pointer
Tree
pointer
Tree
pointer
Ki–1 < X < Ki
Figure 11
The nodes of a B+-tree. (a) Internal node of a B+-tree with q – 1 search values.
(b) Leaf node of a B+-tree with q – 1 search values and q – 1 data pointers.
2. Within each leaf node, K1 ≤ K2 ... , Kq−1, q ≤ p.
3. Each Pri is a data pointer that points to the record whose search field value is
Ki or to a file block containing the record (or to a block of record pointers
that point to records whose search field value is Ki if the search field is not a
key).
4. Each leaf node has at least ⎡(p/2)⎤ values.
5. All leaf nodes are at the same level.
The pointers in internal nodes are tree pointers to blocks that are tree nodes, whereas
the pointers in leaf nodes are data pointers to the data file records or blocks—except
for the Pnext pointer, which is a tree pointer to the next leaf node. By starting at the
leftmost leaf node, it is possible to traverse leaf nodes as a linked list, using the Pnext
pointers. This provides ordered access to the data records on the indexing field. A
Pprevious pointer can also be included. For a B
+-tree on a nonkey field, an extra level
of indirection is needed similar to the one shown in Figure 5, so the Pr pointers are
block pointers to blocks that contain a set of record pointers to the actual records in
the data file, as discussed in option 3 of Section 1.3.
Because entries in the internal nodes of a B+-tree include search values and tree
pointers without any data pointers, more entries can be packed into an internal node
of a B+-tree than for a similar B-tree. Thus, for the same block (node) size, the order
p will be larger for the B+-tree than for the B-tree, as we illustrate in Example 5. This
can lead to fewer B+-tree levels, improving search time. Because the structures for
internal and for leaf nodes of a B+-tree are different, the order p can be different. We
658
Indexing Structures for Files
will use p to denote the order for internal nodes and pleaf to denote the order for leaf
nodes, which we define as being the maximum number of data pointers in a leaf
node.
Example 5. To calculate the order p of a B+-tree, suppose that the search key field
is V = 9 bytes long, the block size is B = 512 bytes, a record pointer is Pr = 7 bytes,
and a block pointer is P = 6 bytes. An internal node of the B+-tree can have up to p
tree pointers and p – 1 search field values; these must fit into a single block. Hence,
we have:
(p * P) + ((p – 1) * V) ≤ B
(P * 6) + ((P − 1) * 9) ≤ 512
(15 * p) ≤ 521
We can choose p to be the largest value satisfying the above inequality, which gives
p = 34. This is larger than the value of 23 for the B-tree (it is left to the reader to
compute the order of the B-tree assuming same size pointers), resulting in a larger
fan-out and more entries in each internal node of a B+-tree than in the correspon-
ding B-tree. The leaf nodes of the B+-tree will have the same number of values and
pointers, except that the pointers are data pointers and a next pointer. Hence, the
order pleaf for the leaf nodes can be calculated as follows:
(pleaf * (Pr + V)) + P ≤ B
(pleaf * (7 + 9)) + 6 ≤ 512
(16 * pleaf) ≤ 506
It follows that each leaf node can hold up to pleaf = 31 key value/data pointer combi-
nations, assuming that the data pointers are record pointers.
As with the B-tree, we may need additional information—to implement the inser-
tion and deletion algorithms—in each node. This information can include the type
of node (internal or leaf), the number of current entries q in the node, and pointers
to the parent and sibling nodes. Hence, before we do the above calculations for p
and pleaf, we should reduce the block size by the amount of space needed for all such
information. The next example illustrates how we can calculate the number of
entries in a B+-tree.
Example 6. Suppose that we construct a B+-tree on the field in Example 5. To cal-
culate the approximate number of entries in the B+-tree, we assume that each node
is 69 percent full. On the average, each internal node will have 34 * 0.69 or approxi-
mately 23 pointers, and hence 22 values. Each leaf node, on the average, will hold
0.69 * pleaf = 0.69 * 31 or approximately 21 data record pointers. A B
+-tree will have
the following average number of entries at each level:
Root: 1 node 22 key entries 23 pointers
Level 1: 23 nodes 506 key entries 529 pointers
Level 2: 529 nodes 11,638 key entries 12,167 pointers
Leaf level: 12,167 nodes 255,507 data record pointers
659
Indexing Structures for Files
For the block size, pointer size, and search field size given above, a three-level B+-
tree holds up to 255,507 record pointers, with the average 69 percent occupancy of
nodes. Compare this to the 65,535 entries for the corresponding B-tree in Example
4. This is the main reason that B+-trees are preferred to B-trees as indexes to data-
base files.
Search, Insertion, and Deletion with B+-Trees. Algorithm 2 outlines the pro-
cedure using the B+-tree as the access structure to search for a record. Algorithm 3
illustrates the procedure for inserting a record in a file with a B+-tree access struc-
ture. These algorithms assume the existence of a key search field, and they must be
modified appropriately for the case of a B+-tree on a nonkey field. We illustrate
insertion and deletion with an example.
Algorithm 2. Searching for a Record with Search Key Field Value K, Using a
B+-tree
n ← block containing root node of B+-tree;
read block n;
while (n is not a leaf node of the B+-tree) do
begin
q ← number of tree pointers in node n;
if K ≤ n.K1 (*n.Ki refers to the ith search field value in node n*)
then n ← n.P1 (*n.Pi refers to the ith tree pointer in node n*)
else if K > n.Kq−1
then n ← n.Pq
search node n for an entry i such that n.Ki−1 < K ≤n.Ki;
n ← n.Pi
end;
read block n
end;
search block n for entry (Ki, Pri) with K = Ki; (* search leaf node *)
if found
then read data file block with address Pri and retrieve record
else the record with search field value K is not in the data file;
Algorithm 3. Inserting a Record with Search Key Field Value K in a B+-tree of
Order p
n ← block containing root node of B+-tree;
read block n; set stack S to empty;
while (n is not a leaf node of the B+-tree) do
begin
push address of n on stack S;
(*stack S holds parent nodes that are needed in case of split*)
q ← number of tree pointers in node n;
if K ≤n.K1 (*n.Ki refers to the ith search field value in node n*)
660
then n ← n.P1 (*n.Pi refers to the ith tree pointer in node n*)
else if K > n.Kq−1
then n ← n.Pq
search node n for an entry i such that n.Ki−1 < K ≤n.Ki;
n ← n.Pi
end;
read block n
end;
search block n for entry (Ki,Pri) with K = Ki; (*search leaf node n*)
if found
then record already in file; cannot insert
else (*insert entry in B+-tree to point to record*)
begin
create entry (K, Pr) where Pr points to the new record;
if leaf node n is not full
then insert entry (K, Pr) in correct position in leaf node n
else begin (*leaf node n is full with pleaf record pointers; is split*)
copy n to temp (*temp is an oversize leaf node to hold extra
entries*);
insert entry (K, Pr) in temp in correct position;
(*temp now holds pleaf + 1 entries of the form (Ki, Pri)*)
new ← a new empty leaf node for the tree; new.Pnext ← n.Pnext ;
j ← ⎡(pleaf + 1)/2 ⎤ ;
n ← first j entries in temp (up to entry (Kj, Prj)); n.Pnext ← new;
new ← remaining entries in temp; K ← Kj ;
(*now we must move (K, new) and insert in parent internal node;
however, if parent is full, split may propagate*)
finished ← false;
repeat
if stack S is empty
then (*no parent node; new root node is created for the tree*)
begin
root ← a new empty internal node for the tree;
root ← ; finished ← true;
else begin
then
else begin (*internal node n is full with p tree pointers;
Indexing Structures for Files
661
Indexing Structures for Files
copy n to temp (*temp is an oversize internal node*);*)
internal node*)
end
end;
Figure 12 illustrates insertion of records in a B+-tree of order p = 3 and pleaf = 2.
When a leaf node is full and a new entry is inserted there, the node overflows and
Figure 13 illustrates deletion from a B+-tree. When an entry is deleted, it is always
662
Indexing Structures for Files
5 0 8
0
5
1 0 5 0 8 0
5
3 5
5
3
3
8
3 7 8
8
Tree node pointer
Data pointer
Null tree pointerInsert 7
Insert 9
Insert 6: overflow (split, propagates)
Insert 3: overflow
Insert 12: overflow (split, propagates,
1 0 5 0 7 0 8 0
1 0 5 0
5 0 12 0
3 0
1 0 3 0
5 01 0 3 0
7 0 8 0
7 0 8 0
12 09 07 0 8 0
5
5
5 01 0 3 0 8 0 12 09 06 0 7 0
Insertion sequence: 8, 5, 1, 7, 3, 12, 9, 6
0
Figure 12
663
Indexing Structures for Files
7
1 6
7
1 6 9
Deletion sequence: 5, 12, 9
1 o 5 o 6
Delete 5
o 8 o 9 o 12 o7 o
1 o 6 o 8 o 9 o 12 o7 o
9
7
1 6 8
1 o 6 o 8 o 9 o7 o
Delete 12: underflow
Delete 9: underflow
1 7
1 o 6 o 8 o7 o
Figure 13
and redistribute the entries among the node and its sibling so that both are at least
664
Indexing Structures for Files
made. If this is also not possible, the three nodes are merged into two leaf nodes. In
Notice that implementing the insertion and deletion algorithms may require parent
Variations of B-Trees and B+-Trees. To conclude this section, we briefly men-
4 Indexes on Multiple Keys
For example, consider an EMPLOYEE file containing attributes Dno (department
1. Assuming Dno has an index, but Age does not, access the records having
11For more details on insertion and deletion algorithms for B+ trees, consult Ramakrishnan and Gehrke
665
Indexing Structures for Files
2. Alternately, if Age is indexed but Dno is not, access the records having Age =
3. If indexes have been created on both Dno and Age, both indexes may be used;
All of these alternatives eventually give the correct result. However, if the set of
4.1 Ordered Index on Multiple Attributes. The search key is a pair of values <4,
59> in the above example. In general, if an index is created on attributes , the search key values are tuples with n values: .
A lexicographic ordering of these tuple values establishes an order on this compos-
4.2 Partitioned Hashing
For example, consider the composite search key . If Dno and Age are
666
Indexing Structures for Files
Linear Scale for Age
EMPLOYEE file Bucket pool
Bucket pool
4
5
3
2
0 1 2 3 4 5
< 20 21–25 26–30 31–40 41–50 > 50
0 1 2 3 4 5
Dno
Linear scale
0 1, 2
3, 4
5
6, 7
8
9, 10
1
2
3
4
5
Figure 14
so on. An advantage of partitioned hashing is that it can be easily extended to any
4.3 Grid Files
Thus our request for Dno = 4 and Age = 59 maps into the cell (1, 5) corresponding
667
Indexing Structures for Files
and Age > 40 refers to the data in the top bucket shown in Figure 14. The grid file
5 Other Types of Indexes
5.1 Hash Indexes or , where Pr is a pointer to the record
5.2 Bitmap Indexes
12Insertion/deletion algorithms for grid files may be found in Nievergelt et al. (1984).
668
Indexing Structures for Files
Bucket 0 Emp_id
12676 Marcus M . .
. . . . . . . . . .
13646 Hanson M . .
. . . . . . . . . .
21124 Dunhill M . .
. . . . . . . . . .
23402 Clarke F . .
. . . . . . . . . .
34723 Ferragamo F . .
. . . . . . . . . .
41301 Zara F . .
. . . . . . . . . .
51024 Bass M . .
. . . . . . . . . .
62104 England M . .
. . . . . . . . . .
71221 Abercombe F . .
. . . . . . . . . .
81165 Gucci F . .
. . . . . . . . . .
13646
21124
. . . . .
Lastname Sex . . . . .
Bucket 1
23402
81165
. . . . .
Bucket 2
51024
12676
. . . . .
Bucket 3
62104
71221
. . . . .
Bucket 9
34723
41301
. . . . .
Figure 15
A bitmap index is built on one particular value of a particular field (the column in a with m distinct values, then m bitmap indexes
■ For the query C1 = V1 , the corresponding bitmap for value V1 returns the
669
Indexing Structures for Files
EMPLOYEE
Row_id Emp_id Lname Sex Zipcode Salary_grade
Bitmap index for Sex
M F
Bitmap index for Zipcode
Zipcode 19046 Zipcode 30022 Zipcode 94040
Figure 16
■ For the query C1= V1 and C2 = V2 (a multikey search request), the two cor-
■ To retrieve a count of rows that qualify for the condition C1 = V1, the “1”
■ Queries with negation, such as C1 ¬ = V1, can be handled by applying the
Consider the example in Figure 16. To find employees with Sex = F and
In general, bitmap indexes are efficient in terms of the storage space that they need.
670
Indexing Structures for Files
When records are deleted, renumbering rows and shifting bits in bitmaps becomes
Large bitvectors are handled by treating them as a series of 32-bit or 64-bit vectors,
Bitmaps for B+-Tree Leaf Nodes. Bitmaps can be used on the leaf nodes of
5.3 Function-Based Indexing
The idea behind function-based indexing is to create an index such that the value
Example 1. The following statement creates a function-based index on the
CREATE INDEX upper_ix ON Employee (UPPER(Lname));
13Rafi Ahmed contributed most of this section.
671
Indexing Structures for Files
This statement will create an index based on the function UPPER(Lname), which
Function-based indexes ensure that Oracle Database system will use the index
SELECT First_name, Lname
Without the function-based index, an Oracle Database might perform a full table
Example 2. In this example, the EMPLOYEE table is supposed to contain two
CREATE INDEX income_ix
The following query uses the income_ix index even though the fields salary and
SELECT First_name, Lname
Example 3. This is a more advanced example of using function-based indexing to
CREATE UNIQUE INDEX promo_ix ON Orders
Note that by using the CASE statement, the objective is to remove from the index any
672
Indexing Structures for Files
6 Some General Issues
6.1 Logical versus Physical Indexes (or ) always include a physical pointer Pr (or P) that specifies the physical record
To remedy this situation, we can use a structure called a logical index, whose index. Each entry has one value K for the secondary index-
6.2 Discussion
The main advantage of secondary indexes is that—theoretically, at least—they can
It is common to use an index to enforce a key constraint on an attribute. While
673
Indexing Structures for Files
time whether another record in the file—and hence in the index tree—has the same
If an index is created on a nonkey field, duplicates occur; handling of these dupli- combination as the de facto key for the index, turning
In actual DBMS products, deletion from B+-tree indexes is also handled in various
A file that has a secondary index on every one of its fields is often called a fully
We referred to the popular IBM file organization called ISAM in Section 2. Another
6.3 Column-Based Storage of Relations
■ Vertically partitioning the table column by column, so that a two-column
■ Use of column-wise indexes (similar to the bitmap indexes discussed in
674
Indexing Structures for Files
■ Use of materialized views to support queries on multiple columns
Column-wise storage of data affords additional freedom in the creation of indexes,
7 Summary
Three types of ordered single-level indexes were introduced: primary, clustering, and
Next we showed how multilevel indexes can be implemented as B-trees and B+-
We gave an overview of multiple key access methods, and showed how an index can
675
Indexing Structures for Files
We introduced the concept of a logical index and compared it with the physical
Review Questions
secondary key field, block anchor, dense index, and nondense (sparse) index.
2. What are the differences among primary, secondary, and clustering indexes?
3. Why can we have at most one primary or clustering index on a file, but sev-
4. How does multilevel indexing improve the efficiency of searching an index
5. What is the order p of a B-tree? Describe the structure of B-tree nodes.
6. What is the order p of a B+-tree? Describe the structure of both internal and
7. How does a B-tree differ from a B+-tree? Why is a B+-tree usually preferred
8. Explain what alternative choices exist for accessing a file based on multiple
9. What is partitioned hashing? How does it work? What are its limitations?
10. What is a grid file? What are its advantages and disadvantages?
11. Show an example of constructing a grid array on two attributes on some file.
12. What is a fully inverted file? What is an indexed sequential file?
13. How can hashing be used to construct an index?
14. What is bitmap indexing? Create a relation with two columns and sixteen
15. What is the concept of function-based indexing? What additional purpose
16. What is the difference between a logical index and a physical index?
17. What is column-based storage of a relational database?
676
Indexing Structures for Files
Exercises
long, and a record pointer is PR = 7 bytes long. A file has r = 30,000
a. Calculate the record size R in bytes.
b. Calculate the blocking factor bfr and the number of file blocks b, assum-
c. Suppose that the file is ordered by the key field Ssn and we want to con-
d. Suppose that the file is not ordered by the key field Ssn and we want to
e. Suppose that the file is not ordered by the nonkey field Department_code
f. Suppose that the file is ordered by the nonkey field Department_code and
677
Indexing Structures for Files
required by the multilevel index; and (v) the number of block accesses
g. Suppose that the file is not ordered by the key field Ssn and we want to
+-tree; (ii) the number of leaf-level blocks
h. Repeat part g, but for a B-tree rather than for a B+-tree. Compare your
19. A PARTS file with Part# as the key field includes records with the following
20. Repeat Exercise 19, but use a B-tree of order p = 4 instead of a B+-tree.
21. Suppose that the following search field values are deleted, in the given order,
22. Repeat Exercise 21, but for the B-tree of Exercise 20.
23. Algorithm 1 outlines the procedure for searching a nondense multilevel pri-
a. A multilevel secondary index on a nonkey nonordering field of a file.
b. A multilevel secondary index on a nonordering key field of a file.
c. A multilevel clustering index on a nonkey ordering field of a file.
24. Suppose that several secondary indexes exist on nonkey fields of a file,
678
Indexing Structures for Files
25. Adapt Algorithms 2 and 3, which outline search and insertion procedures for
26. It is possible to modify the B+-tree insertion algorithm to delay the case
27. Outline an algorithm for deletion from a B+-tree.
28. Repeat Exercise 27 for a B-tree.
Selected Bibliography
New techniques and applications of indexes and B+-trees are discussed in Lanka
679
Indexing Structures for Files
1 0 3 0 5 0 7 0 8 0
3 5
Insert 12: overflow (left
Insert 9: overflow (new level)
Insert 6: overflow (split)
1 0 3 0 5 0 7 0 8 0 12 0
1 0 3 0 5 0 7 0 8 0 9 0 12 0
12 0
3 7
3 9
7
1 0 3 0 5 0 6 0 7 0 8 0 9 0
3 6 9
7
Figure 17
680
Name
Aaron, Ed
Abbott, Diane
Block 1
Acosta, Marc
Ssn Birth_date
…
Job Salary Sex
…
Adams, John
Adams, Robin
Block 2
Akers, Jan
…
Alexander, Ed
Alfred, Bob
Block 3
Allen, Sam
…
Allen, Troy
Anders, Keith
Block 4
Anderson, Rob
…
Anderson, Zach
Angeli, Joe
Block 5
Archer, Sue
…
Arnold, Mack
Arnold, Steven
Block 6
Atkins, Timothy
Wong, James
Wood, Donald
Block n–1
Woods, Manny
…
Wright, Pam
Wyatt, Charles
Block n
Zimmer, Byron
…
Figure A.1
681
Indexing Structures for Files
Table A.1 Average Access Times for a File of b Blocks under Basic File Organizations
Average Blocks to Access
Heap (unordered) Sequential scan (linear search) b/2
Global depth
000
001
010
011
100
101
110
111
d´ = 3 Bucket for records
Directory Data file buckets
d´ = 3 Bucket for records
d´ = 2 Bucket for records
d´ = 2 Bucket for records
d´ = 3 Bucket for records
d´ = 3 Bucket for records
Figure A.2
682
Indexing Structures for Files
Data File Buckets
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Bucket for records
Directory
1
0
1
0
1
0
1
0
1
internal directory node
leaf directory node
Figure A.3
683
Algorithms for Query
In this chapter we discuss the techniques used inter-nally by a DBMS to process, optimize, and execute
Figure 1 shows the different steps of processing a high-level query. The query opti-
1We will not discuss the parsing and syntax-checking phase of query processing here; this material is
From Chapter 19 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
684
Algorithms for Query Processing and Optimization
The term optimization is actually a misnomer because in some cases the chosen exe-
For lower-level navigational database languages in legacy systems—such as the
We will concentrate on describing query optimization in the context of an RDBMS
Query in a high-level language
Scanning, parsing, and validating
Immediate form of query
Query optimizer
Execution plan
Query code generator
Code to execute the query
Runtime database processor
Code can be:
Executed directly (interpreted mode)
Stored and executed later whenever
Result of query
Figure 1
685
Algorithms for Query Processing and Optimization
2There are some query optimization problems and techniques that are pertinent only to ODBMSs.
of database management systems, such as ODBMSs.2 A relational DBMS must sys-
This chapter starts with a general discussion of how SQL queries are typically trans-
The topics covered in this chapter require that the reader be familiar with SQL, rela-
1 Translating SQL Queries into Relational
In practice, SQL is the query language that is used in most commercial RDBMSs. An
686
Algorithms for Query Processing and Optimization
separate query blocks. Because SQL includes aggregate operators—such as MAX,
Consider the following SQL query on the EMPLOYEE relation in Figure A.1 (in
SELECT Lname, Fname
FROM EMPLOYEE
This query retrieves the names of employees (from any department in the com-
( SELECT MAX (Salary)
This retrieves the highest salary in department 5. The outer query block is:
SELECT Lname, Fname
where c represents the result returned from the inner block. The inner block could
ℑMAX Salary(σDno=5(EMPLOYEE))
and the outer block into the expression:
πLname,Fname(σSalary>c(EMPLOYEE))
The query optimizer would then choose an execution plan for each query block.
2 Algorithms for External Sorting
687
Algorithms for Query Processing and Optimization
option in the SELECT clause). We will discuss one of these algorithms in this sec-
External sorting refers to sorting algorithms that are suitable for large files of
In the sorting phase, runs (portions or pieces) of the file that can fit in the available
In the merging phase, the sorted runs are merged during one or more merge
3Internal sorting algorithms are suitable for sorting data structures, such as tables and lists, that can fit
688
Algorithms for Query Processing and Optimization
set i ← 1;
{Sorting Phase}
read next k blocks of the file into the buffer or if there are less than k blocks
sort the records in the buffer and write as a temporary subfile;
}
{Merging Phase: merge subfiles until only 1 remains}
p ← ⎡logk–1m⎤ {p is the number of passes for the merging phase}
while (i ≤ p)
n ← 1;
read next k–1 subfiles or remaining subfiles (from previous pass)
merge and write as new subfile one block at a time;
}
}
Figure 2
The performance of the sort-merge algorithm can be measured in the number of
(2 * b) + (2 * b * (logdM nR))
The first term (2 * b) represents the number of block accesses for the sorting phase,
689
Algorithms for Query Processing and Optimization
The minimum number of main memory buffers needed is nB = 3, which gives a dM
(2 * b) + (2 * (b * (log2 nR))).
The following sections discuss the various algorithms for the operations of the rela-
3 Algorithms for SELECT and JOIN Operations
3.1 Implementing the SELECT Operation
OP1: σSsn = ‘123456789’ (EMPLOYEE)
OP2: σDnumber > 5 (DEPARTMENT)
OP3: σDno = 5 (EMPLOYEE)
OP4: σDno = 5 AND Salary > 30000 AND Sex = ‘F’ (EMPLOYEE)
OP5: σEssn=‘123456789’ AND Pno =10(WORKS_ON)
Search Methods for Simple Selection. A number of search algorithms are
■ S1—Linear search (brute force algorithm). Retrieve every record in the file,
4A selection operation is sometimes called a filter, since it filters out the records in the file that do not
690
Algorithms for Query Processing and Optimization
■ S2—Binary search. If the selection condition involves an equality compari-
■ S3a—Using a primary index. If the selection condition involves an equality
■ S3b—Using a hash key. If the selection condition involves an equality com-
■ S4—Using a primary index to retrieve multiple records. If the comparison
With such a condition, little optimization can be done, because the records satisfy-
A DBMS will have available many of the methods discussed above, and typically
8In more sophisticated optimizers, histograms representing the distribution of the records among the dif-
693
Algorithms for Query Processing and Optimization
3.2 Implementing the JOIN Operation
There are many possible ways to implement a two-way join, which is a join on two
R A=B S
where A and B are the join attributes, which should be domain-compatible attrib-
OP6: EMPLOYEE Dno=Dnumber DEPARTMENT
Methods for Implementing Joins.
■ J1—Nested-loop join (or nested-block join). This is the default (brute
■ J2—Single-loop join (using an access structure to retrieve the matching
■ J3—Sort-merge join. If the records of R and S are physically sorted (ordered)
9For disk files, it is obvious that the loops will be over disk blocks, so this technique has also been called
694
matching with the other file—unless both A and B are nonkey attributes, in
■ J4—Partition-hash join. The records of files R and S are partitioned into
How Buffer Space and Choice of Outer-Loop File Affect Performance of
Algorithms for Query Processing and Optimization
695
(a) sort the tuples in R on attribute A; (* assume R has n tuples (records) *)
then set j ← j + 1 to T;
(* output other tuples that match R(i), if any *) to T;
set l ← l + 1
(* output other tuples that match S(j), if any *) to T;
set k ← k + 1
}
(b) create a tuple t[] in T� for each tuple t in R;
if includes a key of R
else { sort the tuples in T �;
while T�[ i ] = T�[ j ] and j ≤ n do j ← j + 1; (* eliminate duplicates *)
}
Figure 3
A=B(R).
Algorithms for Query Processing and Optimization
696
(c) sort the tuples in R and S using the same unique sort attributes;
then { output S( j ) to T;
}
then set j ← j + 1
then set j ← j + 1
13If the hash function used for partitioning is used again, all records in a partition will hash to the same
700
Algorithms for Query Processing and Optimization
For illustration, assume we are performing the join operation OP6, repeated below:
OP6: EMPLOYEE Dno=Dnumber DEPARTMENT
In this example, the smaller file is the DEPARTMENT file; hence, if the number of
Hybrid Hash-Join. The hybrid hash-join algorithm is a variation of partition(R) is straightforward to implement if includes a key of relation R, because in this case the result of the operation will
701
Algorithms for Query Processing and Optimization
have the same number of tuples as R, but with only the values for the attributes in in each tuple. If does not include a key of R, duplicate
Set operations—UNION, INTERSECTION, SET DIFFERENCE, and CARTESIAN
The other three set operations—UNION, INTERSECTION, and SET
Hashing can also be used to implement UNION, INTERSECTION, and SET DIFFER-
14SET DIFFERENCE is called EXCEPT in SQL.
702
Algorithms for Query Processing and Optimization
In SQL, there are two variations of these set operations. The operations UNION,
5 Implementing Aggregate Operations
5.1 Implementing Aggregate Operations
SELECT MAX(Salary)
If an (ascending) B+-tree index on Salary exists for the EMPLOYEE relation, then the
The index could also be used for the AVERAGE and SUM aggregate functions, but
When a GROUP BY clause is used in a query, the aggregate operator must be applied
SELECT Dno, AVG(Salary)
703
Algorithms for Query Processing and Optimization
The usual technique for such queries is to first use either sorting or hashing on the
Notice that if a clustering index exists on the grouping attribute(s), then the
5.2 Implementing OUTER JOINs
SELECT Lname, Fname, Dname
The result of this query is a table of employee names and their associated depart-
Outer join can be computed by modifying one of the join algorithms, such as
Theoretically, outer join can also be computed by executing a combination of rela-
1. Compute the (inner) JOIN of the EMPLOYEE and DEPARTMENT tables.
TEMP1 ← πLname, Fname, Dname (EMPLOYEE Dno=Dnumber DEPARTMENT)
2. Find the EMPLOYEE tuples that do not appear in the (inner) JOIN result.
TEMP2 ← πLname, Fname (EMPLOYEE) – πLname, Fname (TEMP1)
3. Pad each tuple in TEMP2 with a NULL Dname field.
TEMP2 ← TEMP2 × NULL
704
Algorithms for Query Processing and Optimization
4. Apply the UNION operation to TEMP1, TEMP2 to produce the LEFT OUTER
RESULT ← TEMP1 ∪ TEMP2
The cost of the outer join as computed above would be the sum of the costs of the
6 Combining Operations Using Pipelining
For example, rather than being implemented separately, a JOIN can be combined
It is common to create the query execution code dynamically to implement multiple
7 Using Heuristics in Query Optimization
705
Algorithms for Query Processing and Optimization
to execute groups of operations based on the access paths available on the files
One of the main heuristic rules is to apply SELECT and PROJECT operations before
In Section 7.1 we discuss query tree and query graph notations in the context of
7.1 Notation for Query Trees and Query Graphs
Figure 4a shows a query tree: For every project located in ‘Stafford’, retrieve the proj-
πPnumber, Dnum, Lname, Address, Bdate (((σPlocation=‘Stafford’(PROJECT))
Dnum=Dnumber(DEPARTMENT)) Mgr_ssn=Ssn(EMPLOYEE))
This corresponds to the following SQL query:
Q2: SELECT P.Pnumber, P.Dnum, E.Lname, E.Address, E.Bdate
P.Plocation= ‘Stafford’;
706
Algorithms for Query Processing and Optimization
(b)
(a)
E
DP
P.Pnumber, P.Dnum, E.Lname, E.Address, E.Bdate
P.Dnum=D.Dnumber AND D.Mgr_ssn=E.Ssn AND P.Plocation=‘Stafford’
(c)
EDP
[P.Pnumber, P.Dnum] [E.Lname, E.Address, E.Bdate]
P.Dnum=D.Dnumber
P.Plocation=‘Stafford’
D.Mgr_ssn=E.Ssn
‘Stafford’
XX
XX
(1)
(2)
(3)
P.Pnumber,P.Dnum,E.Lname,E.Address,E.Bdateπ
D.Mgr_ssn=E.Ssn
P.Dnum=D.Dnumber
σP.Plocation= ‘Stafford’
E
D
P
EMPLOYEE
DEPARTMENT
PROJECT
Figure 4
In Figure 4a, the leaf nodes P, D, and E represent the three relations PROJECT,
As we can see, the query tree represents a specific order of operations for executing
707
Algorithms for Query Processing and Optimization
query are represented by relation nodes, which are displayed as single circles.
The query graph representation does not indicate an order on which operations to
7.2 Heuristic Optimization of Query Trees
The query parser will typically generate a standard initial query tree to correspond
The optimizer must include rules for equivalence among relational algebra expres-
Example of Transforming a Query. Consider the following query Q on the
15Hence, a query graph corresponds to a relational calculus expression.
708
Algorithms for Query Processing and Optimization
Q: SELECT Lname
AND Bdate > ‘1957-12-31’;
The initial query tree for Q is shown in Figure 5(a). Executing this tree directly first
(a) Lname
Pname=‘Aquarius’ AND Pnumber=Pno AND Essn=Ssn AND Bdate>‘1957-12-31’
PROJECT
WORKS_ONEMPLOYEE
(b) Lname
Pnumber=Pno
Bdate>‘1957-12-31’
Pname=‘Aquarius’Essn=Ssn
π
π
σ
σ
σσ
σ
EMPLOYEE
PROJECT
WORKS_ON
X
X
X
X
Figure 5
709
Algorithms for Query Processing and Optimization
(e) π Lname
σBdate>‘1957-12-31’
σPname=‘Aquarius’
πPnumber πEssn,Pno
π Essn
EMPLOYEE
WORKS_ON
PROJECT
(d)
σBdate>‘1957-12-31’
σPname=‘Aquarius’ EMPLOYEEWORKS_ON
PROJECT
Essn=Ssn
Pnumber=Pno
Pnumber=Pno
Essn=Ssn
(c)
σ Essn=Ssn
π Lname
σ Pnumber=Pno σBdate>‘1957-12-31’
σ
EMPLOYEE
WORKS_ON
PROJECT
X
X
710
execute. This particular query needs only one record from the PROJECT relation—
A further improvement is achieved by switching the positions of the EMPLOYEE and
As the preceding example demonstrates, a query tree can be transformed step by
General Transformation Rules for Relational Algebra Operations. There
1. Cascade of σ A conjunctive selection condition can be broken up into a cas-
(R)� σc1
(…(σcn
2. Commutativity of σσ. The σ operation is commutative:
(σc2
(σc1
3. Cascade of ππ. In a cascade (sequence) of π operations, all but the last one can
πList1
(…(πListn
(R)
4. Commuting σ with π. If the selection condition c involves only those attrib-
πA1, A2, …, An
(R))
Algorithms for Query Processing and Optimization
711
Algorithms for Query Processing and Optimization
5. Commutativity of (and ××). The join operation is commutative, as is the
Notice that although the order of attributes may not be the same in the rela-
6. Commuting σσ with (or ××). If all the attributes in the selection condition
σc (R S) ≡ (σc (R)) S
σc (R S) � (σc1
(S))
The same rules apply if the is replaced by a × operation.
7. Commuting ππ with (or ××). Suppose that the projection list is L = {A1, …,
πL (R c S) � (πA1, …, An
(S))
If the join condition c contains additional attributes not in L, these must be
πL (R c S) � πL ((πA1, …, An, An+1, …, An+k
(S)))
For ×, there is no condition c, so the first transformation rule always applies
8. Commutativity of set operations. The set operations ∪ and ∩ are commu-
9. Associativity of , ××, ∪∪, and ∩∩. These four operations are individually
(R θ S) θ T ≡ R θ (S θ T)
10. Commuting σ with set operations. The σ operation commutes with ∪, ∩,
σc (R θ S) ≡ (σc (R)) θ (σc (S))
712
Algorithms for Query Processing and Optimization
11. The π operation commutes with ∪∪.
12. Converting a (σσ, ××) sequence into . If the condition c of a σ that follows a
(σc (R × S)) ≡ (R c S)
There are other possible transformations. For example, a selection or join condition
NOT (c1 AND c2) ≡ (NOT c1) OR (NOT c2)
We discuss next how transformations can be used in heuristic optimization.
Outline of a Heuristic Algebraic Optimization Algorithm. We can now out-
1. Using Rule 1, break up any SELECT operations with conjunctive conditions
2. Using Rules 2, 4, 6, and 10 concerning the commutativity of SELECT with
3. Using Rules 5 and 9 concerning commutativity and associativity of binary
17Either definition can be used, since these rules are heuristic.
713
Algorithms for Query Processing and Optimization
the two relations with the most restrictive SELECT do not have a direct join
4. Using Rule 12, combine a CARTESIAN PRODUCT operation with a subse-
5. Using Rules 3, 4, 7, and 11 concerning the cascading of PROJECT and the
6. Identify subtrees that represent groups of operations that can be executed by
In our example, Figure 5(b) shows the tree in Figure 5(a) after applying steps 1 and
Summary of Heuristics for Algebraic Optimization. The main heuristic is to
7.3 Converting Query Trees into Query Execution Plans
πFname, Lname, Address(σDname=‘Research’(DEPARTMENT) Dnumber=Dno EMPLOYEE)
18Note that a CARTESIAN PRODUCT is acceptable in some cases—for example, if each relation has
714
Algorithms for Query Processing and Optimization
π Fname, Lname, Address
σ Dname=‘Research’
DEPARTMENT
EMPLOYEE
Dnumber=Dno
Figure 6
The query tree is shown in Figure 6. To convert this into an execution plan, the opti-
With materialized evaluation, the result of an operation is stored as a temporary
8 Using Selectivity and Cost Estimates
A query optimizer does not depend solely on heuristic rules; it also estimates and
715
Algorithms for Query Processing and Optimization
Figure 1 occurs at runtime, a full-scale optimization may slow down the response
This approach is generally referred to as cost-based query optimization.19 It uses
8.1 Cost Components for Query Execution
1. Access cost to secondary storage. This is the cost of transferring (reading
2. Disk storage cost. This is the cost of storing on disk any intermediate files
3. Computation cost. This is the cost of performing in-memory operations on
4. Memory usage cost. This is the cost pertaining to the number of main mem-
5. Communication cost. This is the cost of shipping the query and its results
For large databases, the main emphasis is often on minimizing the access cost to sec-
19This approach was first used in the optimizer for the SYSTEM R in an experimental DBMS developed
716
Algorithms for Query Processing and Optimization
and main memory buffers. For smaller databases, where most of the data in the files
8.2 Catalog Information Used in Cost Functions
Another important parameter is the number of distinct values (d) of an attribute
Information such as the number of index levels is easy to maintain because it does
For a nonkey attribute with d distinct values, it is often the case that the records are
20More accurate optimizers store histograms of the distribution of records over the data values for an
717
Algorithms for Query Processing and Optimization
uted among the departments as follows: (1, 5), (2, 25), (3, 70), (4, 40), (5, 60). In
In the next two sections we examine how some of these parameters are used in cost
8.3 Examples of Cost Functions for SELECT
■ S1—Linear search (brute force) approach. We search all the file blocks to
■ S2—Binary search. This search accesses approximately CS2 = log2b +
■ S3a—Using a primary index to retrieve a single record. For a primary
■ S3b—Using a hash key to retrieve a single record. For hashing, only one
■ S4—Using an ordering index to retrieve multiple records. If the compari-
OP3: σDno=5(EMPLOYEE)
OP4: σDno=5 AND SALARY>30000 AND Sex=‘F’(EMPLOYEE)
The cost of the brute force (linear search or file scan) option S1 will be estimated as
Finally, consider OP4, which has a conjunctive selection condition. We need to esti-
8.4 Examples of Cost Functions for JOIN
js = |(R c S)| / |(R × S)| = |(R c S)| / (|R| * |S|)
If there is no join condition c, then js = 1 and the join is the same as the CARTESIAN
720
Algorithms for Query Processing and Optimization
general, 0 ≤ js ≤ 1. For a join where the condition c is an equality comparison R.A =
1. If A is a key of R, then |(R c S)| ≤ |S|, so js ≤ (1/|R|). This is because each
2. If B is a key of S, then |(R c S)| ≤ |R|, so js ≤ (1/|S|).
Having an estimate of the join selectivity for commonly occurring join conditions
* |R| * |S|. We can now give some sample approximate cost functions for estimating
R A=B S
where A and B are domain-compatible attributes of R and S, respectively. Assume
■ J1—Nested-loop join. Suppose that we use R for the outer loop; then we get
CJ1 = bR + (bR * bS) + (( js * |R| * |S|)/bfrRS)
The last part of the formula is the cost of writing the resulting file to disk.
CJ1 = bR + ( ⎡bR/(nB – 2)⎤ * bS) + ((js * |R| * |S|)/bfrRS)
record(s)). If an index exists for the join attribute B of S with index levels xB,
CJ2a = bR + (|R| * (xB + 1 + sB)) + (( js * |R| * |S|)/bfrRS)
For a clustering index where sB is the selection cardinality of B, we get
CJ2b = bR + (|R| * (xB + (sB/bfrB))) + (( js * |R| * |S|)/bfrRS)
For a primary index, we get
21Selection cardinality was defined as the average number of records that satisfy an equality condition on
721
Algorithms for Query Processing and Optimization
CJ2c = bR + (|R| * (xB + 1)) + (( j s * |R| * |S|)/bfrRS)
If a hash key exists for one of the two join attributes—say, B of S—we get
CJ2d = bR + (|R| * h) + (( j s * |R| * |S|)/bfrRS)
where h ≥ 1 is the average number of block accesses to retrieve a record,
■ J3—Sort-merge join. If the files are already sorted on the join attributes, the
CJ3a = bR + bS + (( j s * |R| * |S|)/bfrRS)
If we must sort the files, the cost of sorting must be added. We can use the
Example of Using the Cost Functions. Suppose that we have the EMPLOYEE
OP6: EMPLOYEE Dno=Dnumber DEPARTMENT
Suppose that we have a primary index on Dnumber of DEPARTMENT with xDnumber= 1
1. Using method J1 with EMPLOYEE as outer loop:
CJ1 = bE + (bE * bD) + (( jsOP6 * rE * rD)/bfrED)
= 2000 + (2000 * 13) + (((1/125) * 10,000 * 125)/4) = 30,500
2. Using method J1 with DEPARTMENT as outer loop:
CJ1 = bD + (bE * bD) + (( jsOP6 * rE * rD)/bfrED)
= 13 + (13 * 2000) + (((1/125) * 10,000 * 125/4) = 28,513
3. Using method J2 with EMPLOYEE as outer loop:
CJ2c = bE + (rE * (xDnumber+ 1)) + (( jsOP6 * rE * rD)/bfrED
4. Using method J2 with DEPARTMENT as outer loop:
CJ2a = bD + (rD * (xDno + sDno)) + (( jsOP6 * rE * rD)/bfrED)
= 13 + (125 * (2 + 80)) + (((1/125) * 10,000 * 125/4) = 12,763
Case 4 has the lowest cost estimate and will be chosen. Notice that in case 2 above, if
722
Algorithms for Query Processing and Optimization
R1 R2
R3
R4
R4 R3
R2
R1
Figure 7
relation) in memory, one could be used as buffer for the result, and one would be
CJ1 = bD + ( ⎡bD/(nB – 2)⎤ * bE) + ((js * |R| * |S|)/bfrRS)
As an exercise, the reader should perform a similar analysis for OP7.
8.5 Multiple Relation Queries and JOIN Ordering
With left-deep trees, the right child is considered to be the inner relation when exe-
723
Algorithms for Query Processing and Optimization
matching tuples. As resulting tuples (records) are produced from the join of R1 and
If materialization is used instead of pipelining (see Sections 6 and 7.3), the join
8.6 Example to Illustrate Cost-Based Query Optimization
Q2: SELECT Pnumber, Dnum, Lname, Address, Bdate
Plocation=‘Stafford’;
Suppose we have the information about the relations shown in Figure 8. The
The first cost-based optimization to consider is join ordering. As previously men-
1. PROJECT DEPARTMENT EMPLOYEE
2. DEPARTMENT PROJECT EMPLOYEE
3. DEPARTMENT EMPLOYEE PROJECT
4. EMPLOYEE DEPARTMENT PROJECT
Assume that the selection operation has already been applied to the PROJECT rela-
724
Algorithms for Query Processing and Optimization
(a) Table_name
PROJECT
PROJECT
DEPARTMENT
DEPARTMENT
EMPLOYEE
EMPLOYEE
200
50
50
50
50
500
1
1
1
1
1
1
200
50
50
50
50
500
Dnum
Dnumber
Plocation
Pnumber
Dno
Salary
Mgr_ssn
Ssn
Column_name Num_distinct Low_value High_value
(c) Index_name
*Blevel is the number of levels without the leaf level.
PROJ_PLOC
EMP_SAL
1
1
4
50
200
500NONUNIQUE
NONUNIQUE
UNIQUE
Uniqueness Blevel* Leaf_blocks Distinct_keys
(b) Table_name
PROJECT
EMPLOYEE
100
200010000
2000
50
Num_rows Blocks
Figure 8
statistical information on the PROJ_PLOC index (see Figure 8) shows the number
In the materialized approach, a temporary file TEMP1 of size 1 block is created to
725
Algorithms for Query Processing and Optimization
into a single block. Now we can compute the estimated cost of the first join. We will
Finally, the cost of the last join needs to be estimated. We can use a single-loop join
For the final projection, assume pipelining is used to produce the final result, which
9 Overview of Query Optimization
The Oracle DBMS22 provides two different approaches to query optimization: rule-
22The discussion in this section is primarily based on version 7 of Oracle. More optimization techniques
726
Algorithms for Query Processing and Optimization
The Oracle optimizer calculates this cost based on the estimated usage of resources,
An interesting addition to the Oracle query optimizer is the capability for an appli-
■ The optimization approach for an SQL statement
■ The access path for a table accessed by the statement
■ The join order for a join statement
■ A particular join operation in a join statement
The cost-based optimization of Oracle 8 and later versions is a good example of the
10 Semantic Query Optimization
SELECT E.Lname, M.Lname
This query retrieves the names of employees who earn more than their supervisors.
23Such hints have also been called query annotations.
727
Algorithms for Query Processing and Optimization
query and that may semantically optimize it can also be quite time-consuming.
11 Summary
Following that, we described heuristic approaches to query optimization, which use
We discussed the cost-based approach to query optimization. We showed how cost
Review Questions
queries before optimization is done.
2. Discuss the different algorithms for implementing each of the following
3. What is a query execution plan?
4. What is meant by the term heuristic optimization? Discuss the main heuris-
5. How does a query tree represent a relational algebra expression? What is
728
Algorithms for Query Processing and Optimization
6. How many different join orders are there for a query that joins 10 relations?
7. What is meant by cost-based query optimization?
8. What is the difference between pipelining and materialization?
9. Discuss the cost components for a cost function that is used to estimate
10. Discuss the different types of parameters that are used in cost functions.
11. List the cost functions for the SELECT and JOIN methods discussed in
12. What is meant by semantic query optimization? How does it differ from
Exercises
Q27 in the chapter More SQL: Complex Queries, Triggers, Views, and
a. Draw at least two query trees that can represent each of these queries.
b. Draw the initial query tree for each of these queries, and then show how
c. For each query, compare your own query trees of part (a) and the initial
14. A file of 4096 blocks is to be sorted with an available buffer space of 64
15. Develop cost functions for the PROJECT, UNION, INTERSECTION, SET DIF-
16. Develop cost functions for an algorithm that consists of two SELECTs, a
17. Can a nondense index be used in the implementation of an aggregate opera-
18. Calculate the cost functions for different options of executing the JOIN oper-
19. Develop formulas for the hybrid hash-join algorithm for calculating the size
729
Algorithms for Query Processing and Optimization
20. Estimate the cost of operations OP6 and OP7, using the formulas developed
21. Extend the sort-merge join algorithm to implement the LEFT OUTER JOIN
22. Compare the cost of two different query plans for the following query:
σSalary > 40000(EMPLOYEE Dno=DnumberDEPARTMENT)
Use the database statistics in Figure 8.
Selected Bibliography
Whang (1985) discusses query optimization in OBE (Office-By-Example), which is
Kim et al. (1985) discuss advanced topics in query optimization. Semantic query
730
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.1
731
Algorithms for Query Processing and Optimization
DEPT_LOCATIONS
Dnumber
Houston
Stafford
Bellaire
Sugarland
Dlocation
DEPARTMENT
Dname
Research
Administration
Headquarters 1
5
4
888665555
333445555
987654321
1981-06-19
1988-05-22
1995-01-01
Dnumber Mgr_ssn Mgr_start_date
WORKS_ON
Essn
123456789
123456789
666884444
453453453
453453453
333445555
333445555
333445555
333445555
999887777
999887777
987987987
987987987
987654321
987654321
888665555
3
1
2
2
1
2
30
30
30
10
10
3
10
20
20
20
40.0
32.5
7.5
10.0
10.0
10.0
10.0
20.0
20.0
30.0
5.0
10.0
35.0
20.0
15.0
NULL
Pno Hours
PROJECT
Pname
ProductX
ProductY
ProductZ
Computerization
Reorganization
Newbenefits
3
1
2
30
10
20
5
5
5
4
4
1
Houston
Bellaire
Sugarland
Stafford
Stafford
Houston
Pnumber Plocation Dnum
DEPENDENT
333445555
333445555
333445555
987654321
123456789
123456789
123456789
Joy
Alice F
M
F
M
M
F
F
1986-04-05
1983-10-25
1958-05-03
1942-02-28
1988-01-04
1988-12-30
1967-05-05
Theodore
Alice
Elizabeth
Abner
Michael
Spouse
Daughter
Son
Daughter
Spouse
Spouse
Son
Dependent_name Sex Bdate Relationship
EMPLOYEE
Fname
John
Franklin
Jennifer
Alicia
Ramesh
Joyce
James
Ahmad
Narayan
English
Borg
Jabbar
666884444
453453453
888665555
987987987
F
F
M
M
M
M
M
F
4
4
5
5
4
1
5
5
25000
43000
30000
40000
25000
55000
38000
25000
987654321
888665555
333445555
888665555
987654321
NULL
333445555
333445555
Zelaya
Wallace
Smith
Wong
3321 Castle, Spring, TX
291 Berry, Bellaire, TX
731 Fondren, Houston, TX
638 Voss, Houston, TX
1968-01-19
1941-06-20
1965-01-09
1955-12-08
1969-03-29
1937-11-10
1962-09-15
1972-07-31
980 Dallas, Houston, TX
450 Stone, Houston, TX
975 Fire Oak, Humble, TX
5631 Rice, Houston, TX
999887777
987654321
123456789
333445555
Minit Lname Ssn Bdate Address Sex DnoSalary Super_ssn
B
T
J
S
K
A
V
E
Houston
1
4
5
5
Essn
5
Figure A.2
732
Physical Database
Various techniques by which queries can beprocessed efficiently by the DBMS are mostly
1 Physical Database Design
In this section, we begin by discussing the physical design factors that affect the per-
1.1 Factors That Influence Physical Database Design
From Chapter 20 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
733
Physical Database Design and Tuning
decisions and performance analyses until the database designer knows the mix of
A. Analyzing the Database Queries and Transactions. Before undertaking
1. The files that will be accessed by the query.1
2. The attributes on which any selection conditions for the query are specified.
3. Whether the selection condition is an equality, inequality, or a range condi-
4. The attributes on which any join conditions or conditions to link multiple
5. The attributes whose values will be retrieved by the query.
The attributes listed in items 2 and 4 above are candidates for the definition of
For each update operation or update transaction, the following information
1. The files that will be updated.
2. The type of operation on each file (insert, update, or delete).
3. The attributes on which selection conditions for a delete or update are spec-
4. The attributes whose values will be changed by an update operation.
Again, the attributes listed in item 3 are candidates for access structures on the files,
B. Analyzing the Expected Frequency of Invocation of Queries and
1For simplicity we use the term files here, but this can also mean tables or relations.
734
Physical Database Design and Tuning
over all the queries and transactions. Generally, for large volumes of processing, the
C. Analyzing the Time Constraints of Queries and Transactions. Some
D. Analyzing the Expected Frequencies of Update Operations. A minimum
E. Analyzing the Uniqueness Constraints on Attributes. Access paths should
Once the preceding information is compiled, it is possible to address the physical
1.2 Physical Database Design Decisions
2The reader should review the various types of indexes and be familiar with the algorithms for query
735
Physical Database Design and Tuning
Design Decisions about Indexing. The attributes whose values are required in
The performance of queries largely depends upon what indexes or hashing schemes
The physical design decisions for indexing fall into the following categories:
1. Whether to index an attribute. The general rules for creating an index on an
2. What attribute or attributes to index on. An index can be constructed on a
3. Whether to set up a clustered index. At most, one index per table can be a
4. Whether to use a hash index over a tree index. In general, RDBMSs use B+-
736
Physical Database Design and Tuning
conditions, particularly during joins to find a matching record(s), but they
5. Whether to use dynamic hashing for the file. For files that are very
How to Create an Index. Many RDBMSs have a similar type of command for
CREATE [ UNIQUE ] INDEX
( [ ] { , [ ] } )The keywords UNIQUE and CLUSTER are optional. The keyword CLUSTER is used can be either ASC (ascending) or DESC (descending), and specifies
CREATE INDEX DnoIndex
Denormalization as a Design Decision for Speeding Up Queries. The ulti-
The above ideals are sometimes sacrificed in favor of faster execution of frequently
737
Physical Database Design and Tuning
versus the effort needed to perform a join to incorporate the additional attributes
ASSIGN (Emp_id, Proj_id, Emp_name, Emp_job_title, Percent_assigned, Proj_name,
which corresponds exactly to the headers in a report called The Employee
This relation is only in 1NF because of the following functional dependencies:
Proj_id → Proj_name, Proj_mgr_id
This relation may be preferred over the design in 2NF (and 3NF) consisting of the
EMP (Emp_id, Emp_name, Emp_job_title)
This is because to produce the The Employee Assignment Roster report (with all
((EMP_PROJ * EMP) * PROJ) PROJ.Proj_mgr_id = EMP.Emp_id EMP
It is also possible to create a view for the ASSIGN table. This does not mean that
738
Physical Database Design and Tuning
2 An Overview of Database Tuning
After a database is deployed and is in operation, actual use of the applications, trans-
■ To make applications run faster.
■ To improve (lower) the response time of queries and transactions.
■ To improve the overall throughput of transactions.
The dividing line between physical design and tuning is very thin. The same design
■ Sizes of individual tables.
■ Number of distinct values in a column.
■ The number of times a particular query or transaction is submitted and exe-
■ The times required for different phases of query and transaction processing
These and other statistics create a profile of the contents and use of the database.
■ Storage statistics. Data about allocation of storage into tablespaces, index-
■ I/O and device performance statistics. Total read/write activity (paging) on
■ Query/transaction processing statistics. Execution times of queries and
3Interested readers should consult Shasha and Bonnet (2002) for a detailed discussion of tuning.
739
Physical Database Design and Tuning
■ Locking/logging related statistics. Rates of issuing different types of locks,
■ Index statistics. Number of levels in an index, number of noncontiguous
Some of the above statistics relate to transactions, concurrency control, and recov-
■ How to avoid excessive lock contention, thereby increasing concurrency
■ How to minimize the overhead of logging and unnecessary dumping of data.
■ How to optimize the buffer size and scheduling of processes.
■ How to allocate resources such as disks, RAM, and processes for most effi-
Most of the previously mentioned problems can be solved by the DBA by setting
2.1 Tuning Indexes
■ Certain queries may take too long to run for lack of an index.
■ Certain indexes may not get utilized at all.
■ Certain indexes may undergo too much updating because the index is on an
Most DBMSs have a command or trace facility, which can be used by the DBA to ask
The goal of tuning is to dynamically evaluate the requirements, which sometimes
740
Physical Database Design and Tuning
index is dropped or created; this loss of service must be accounted for. Besides drop-
The available options for indexing and the way they are defined, created, and reor-
2.2 Tuning the Database Design
As discussed, the entire database design has to be driven by the processing require-
■ Existing tables may be joined (denormalized) because certain attributes
■ For the given set of tables, there may be alternative design choices, all of
■ A relation of the form R(K,A, B, C, D, …)—with K as a set of key attributes—
4Note that 3NF and 2NF address different types of problem dependencies that are independent of each
741
Physical Database Design and Tuning
sets of attributes that are accessed together. For example, the table
■ Attribute(s) from one table may be repeated in another even though this cre-
■ Just as vertical partitioning splits a table vertically into multiple tables,
These types of adjustments designed to meet the high volume of queries or transac-
2.3 Tuning Queries
1. A query issues too many disk accesses (for example, an exact match query
2. The query plan shows that relevant indexes are not being used.
Some typical instances of situations prompting query tuning include the following:
1. Many query optimizers do not use indexes in the presence of arithmetic
2. Indexes are often not used for nested queries using IN; for example, the fol-
SELECT Ssn FROM EMPLOYEE
WHERE Mgr_ssn = ‘333445555’ );
742
Physical Database Design and Tuning
may not use the index on Dno in EMPLOYEE, whereas using Dno = Dnumber
3. Some DISTINCTs may be redundant and can be avoided without changing
4. Unnecessary use of temporary result tables can be avoided by collapsing
5. In some situations involving the use of correlated queries, temporaries are
SELECT Ssn
FROM EMPLOYEE AS M
This has the potential danger of searching all of the inner EMPLOYEE table M
SELECT MAX (Salary) AS High_salary, Dno INTO TEMP
AND EMPLOYEE.Dno = TEMP.Dno;
6. If multiple options for a join condition are possible, choose one that uses a
7. One idiosyncrasy with some query optimizers is that the order of tables in
8. Some query optimizers perform worse on nested queries compared to their
■ Uncorrelated subqueries with aggregates in an inner query.
■ Uncorrelated subqueries without aggregates.
■ Correlated subqueries with aggregates in an inner query.
743
Physical Database Design and Tuning
■ Correlated subqueries without aggregates.
Of the four types above, the first one typically presents no problem, since
9. Finally, many applications are based on views that define the data of interest
2.4 Additional Query Tuning Guidelines
1. A query with multiple selection conditions that are connected via OR may
SELECT Fname, Lname, Salary, Age6
FROM EMPLOYEE, where the value for can be READ UNCOMMITTED, READ
1. Dirty read. A transaction T1 may read the update of a transaction T2, which
13These are similar to the isolation levels discussed briefly at the end of Section 3.
774
Introduction to Transaction Processing Concepts and Theory
Table 1 Possible Violations Based on Isolation Levels as Defined in SQL
Type of Violation
Isolation Level Dirty Read Nonrepeatable Read Phantom
READ UNCOMMITTED Yes Yes Yes
2. Nonrepeatable read. A transaction T1 may read a given value from a table. If
3. Phantoms. A transaction T1 may read a set of rows from a table, perhaps
Table 1 summarizes the possible violations for the different isolation levels. An entry
A sample SQL transaction might look like the following:
EXEC SQL WHENEVER SQLERROR GOTO UNDO;
READ WRITE
EXEC SQL INSERT INTO EMPLOYEE (Fname, Lname, Ssn, Dno, Salary)
EXEC SQL UPDATE EMPLOYEE
EXEC SQL COMMIT;
The above transaction consists of first inserting a new row in the EMPLOYEE table
As we have seen, SQL provides a number of transaction-oriented features. The DBA
775
Introduction to Transaction Processing Concepts and Theory
transaction performance by relaxing serializability if that is acceptable for their
7 Summary
Next we introduced the typical states that a transaction passes through during execu-
Then we defined a schedule (or history) as an execution sequence of the operations
We defined equivalence of schedules and saw that a serializable schedule is equiva-
Review Questions
multiuser system? Discuss why concurrency control is needed, and give
776
Introduction to Transaction Processing Concepts and Theory
2. Discuss the different types of failures. What is meant by catastrophic failure?
3. Discuss the actions taken by the read_item and write_item operations on a
4. Draw a state diagram and discuss the typical states that a transaction goes
5. What is the system log used for? What are the typical kinds of records in a
6. Discuss the atomicity, durability, isolation, and consistency preservation
7. What is a schedule (history)? Define the concepts of recoverable, cascadeless,
8. Discuss the different measures of transaction equivalence. What is the differ-
9. What is a serial schedule? What is a serializable schedule? Why is a serial
10. What is the difference between the constrained write and the unconstrained
11. Discuss how serializability is used to enforce concurrency control in a data-
12. Describe the four levels of isolation in SQL.
13. Define the violations caused by each of the following: dirty read, nonrepeat-
Exercises
read_item(X);
Discuss the final result of the different schedules in Figure 3(a) and (b),
15. Repeat Exercise 14, adding a check in T1 so that Y does not exceed 90.
777
Introduction to Transaction Processing Concepts and Theory
16. Add the operation commit at the end of each of the transactions T1 and T2 in
17. List all possible schedules for transactions T1 and T2 in Figure 2, and deter-
18. How many serial schedules exist for the three transactions in Figure 8(a)?
19. Write a program to create all possible schedules for the three transactions in
20. Why is an explicit transaction end statement needed in SQL but not an
21. Describe situations where each of the different isolation levels would be use-
22. Which of the following schedules is (conflict) serializable? For each serializ-
a. r1(X); r3(X); w1(X); r2(X); w3(X);
b. r1(X); r3(X); w3(X); w1(X); r2(X);
c. r3(X); r2(X); w3(X); r1(X); w1(X);
d. r3(X); r2(X); r1(X); w3(X); w1(X);
23. Consider the three transactions T1, T2, and T3, and the schedules S1 and S2
T1: r1 (X); r1 (Z); w1 (X);
24. Consider schedules S3, S4, and S5 below. Determine whether each schedule is
S3: r1 (X); r2 (Z); r1 (Z); r3 (X); r3 (Y); w1 (X); c1; w3 (Y); c3; r2 (Y); w2 (Z);
S4: r1 (X); r2 (Z); r1 (Z); r3 (X); r3 (Y); w1 (X); w3 (Y); r2 (Y); w2 (Z); w2 (Y); c1;
S5: r1 (X); r2 (Z); r3 (X); r1 (Z); r2 (Y); r3 (Y); w1 (X); c1; w2 (Z); w3 (Y); w2 (Y);
778
Selected Bibliography
Introduction to Transaction Processing Concepts and Theory
779
Concurrency Control
In this chapter we discuss a number of concurrencycontrol techniques that are used to ensure the nonin-
Another factor that affects concurrency control is the granularity of the data
From Chapter 22 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
780
Concurrency Control Techniques
indexes are used to process transactions, and in Section 7 we discuss some addi-
It is sufficient to read Sections 1, 5, 6, and 7, and possibly 3.2, if your main interest is
1 Two-Phase Locking Techniques
Some of the main techniques used to control concurrent execution of transactions
1.1 Types of Locks and System Lock Tables
Binary Locks. A binary lock can have two states or values: locked and unlocked (or
Two operations, lock_item and unlock_item, are used with binary locking. A transaction
781
Concurrency Control Techniques
lock_item(X):
then LOCK(X) ←1 (* lock the item *)
begin
and the lock manager wakes up the transaction);
unlock_item(X):
then wakeup one of the waiting transactions;
Figure 1
Notice that the lock_item and unlock_item operations must be implemented as indi-
It is quite simple to implement a binary lock; all that is needed is a binary-valued plus a queue for transactions that are waiting to access the item.
If the simple binary locking scheme described here is used, every transaction must
1. A transaction T must issue the operation lock_item(X) before any
2. A transaction T must issue the operation unlock_item(X) after all read_item(X)
3. A transaction T will not issue a lock_item(X) operation if it already holds the
4. A transaction T will not issue an unlock_item(X) operation unless it already
1This rule may be removed if we modify the lock_item (X) operation in Figure 1 so that if the item is cur-
782
Concurrency Control Techniques
These rules can be enforced by the lock manager module of the DBMS. Between the
Shared/Exclusive (or Read/Write) Locks. The preceding binary locking
One method for implementing the preceding operations on a read/write lock is to. Again, to save space, the system needs to
When we use the shared/exclusive locking scheme, the system must enforce the fol-
1. A transaction T must issue the operation read_lock(X) or write_lock(X) before
2. A transaction T must issue the operation write_lock(X) before any
2These algorithms do not allow upgrading or downgrading of locks, as described later in this section. The
783
read_lock(X):
then begin LOCK(X) ← “read-locked”;
else if LOCK(X) = “read-locked”
else begin
and the lock manager wakes up the transaction);
write_lock(X):
then LOCK(X) ← “write-locked”
wait (until LOCK(X) = “unlocked”
go to B
unlock (X):
then begin LOCK(X) ← “unlocked”;
else it LOCK(X) = “read-locked”
no_of_reads(X) ← no_of_reads(X) −1;
then begin LOCK(X) = “unlocked”;
end;
Concurrency Control Techniques
Figure 2
3. A transaction T must issue the operation unlock(X) after all read_item(X) and
4. A transaction T will not issue a read_lock(X) operation if it already holds a
3This rule may be relaxed to allow a transaction to unlock an item, then lock it again later.
784
Concurrency Control Techniques
5. A transaction T will not issue a write_lock(X) operation if it already holds a
6. A transaction T will not issue an unlock(X) operation unless it already holds
Conversion of Locks. Sometimes it is desirable to relax conditions 4 and 5 in the
Using binary locks or read/write locks in transactions, as described earlier, does not
1.2 Guaranteeing Serializability by Two-Phase Locking
Such a transaction can be divided into two phases: an expanding or growing (first)
4This is unrelated to the two-phase commit protocol for recovery in distributed databases.
785
Concurrency Control Techniques
(a) T1 Initial values: X=20, Y=30
Result serial schedule T1
Result of serial schedule T2
read_lock(Y );
write_lock(X );
read_lock(X );
read_lock(X );
(b)
(c)
Time
read_lock(Y );
Result of schedule S:
T2
T1 T2
Figure 3
shrinking phase. Hence, a read_lock(X) operation that downgrades an already held
Transactions T1 and T2 in Figure 3(a) do not follow the two-phase locking protocol
786
Concurrency Control Techniques
read_lock(Y );
read_lock(X );
T1� T2�
Figure 4
It can be proved that, if every transaction in a schedule follows the two-phase lock-
Two-phase locking may limit the amount of concurrency that can occur in a sched-
Although the two-phase locking protocol guarantees serializability (that is, every
Basic, Conservative, Strict, and Rigorous Two-Phase Locking. There are a
In practice, the most popular variation of 2PL is strict 2PL, which guarantees strict
787
Concurrency Control Techniques
(write) locks until after it commits or aborts. Hence, no other transaction can read
In many cases, the concurrency control subsystem itself is responsible for generat-
The use of locks can cause two additional problems: deadlock and starvation. We
1.3 Dealing with Deadlock and Starvation
Deadlock Prevention Protocols. One way to prevent deadlock is to use a
5These protocols are not generally used in practice, either because of unrealistic assumptions or
788
Concurrency Control Techniques
(a) T1� (b)
read_lock(Y );
Time
write_lock(X );
read_lock(X );
write_lock(Y );
T2�
T2�T1�
X
Y
Figure 5
in conservative two-phase locking, requires that every transaction lock all the items
A number of other deadlock prevention schemes have been proposed that make a
b. If the condition in part (a) does not occur, then execute the write_item(X)
2. Whenever a transaction T issues a read_item(X) operation, the following is
a. If write_TS(X) > TS(T), then abort and roll back T and reject the opera-
b. If write_TS(X) ≤ TS(T), then execute the read_item(X) operation of T and
Whenever the basic TO algorithm detects two conflicting operations that occur in the
Strict Timestamp Ordering (TO). A variation of basic TO called strict TO
793
Concurrency Control Techniques
locking of an item X that has been written by transaction T� until T� is either com-
Thomas’s Write Rule. A modification of the basic TO algorithm, known as
1. If read_TS(X) > TS(T), then abort and roll back T and reject the operation.
2. If write_TS(X) > TS(T), then do not execute the write operation but continue
3. If neither the condition in part (1) nor the condition in part (2) occurs, then
3 Multiversion Concurrency
Other protocols for concurrency control keep the old values of a data item when the
An obvious drawback of multiversion techniques is that more storage is needed to
Several multiversion concurrency control schemes have been proposed. We discuss
794
Concurrency Control Techniques
3.1 Multiversion Technique Based on Timestamp Ordering
1. read_TS(Xi). The read timestamp of Xi is the largest of all the timestamps of
2. write_TS(Xi). The write timestamp of Xi is the timestamp of the transac-
Whenever a transaction T is allowed to execute a write_item(X) operation, a new ver-
To ensure serializability, the following rules are used:
1. If transaction T issues a write_item(X) operation, and version i of X has the
2. If transaction T issues a read_item(X) operation, find the version i of X that
As we can see in case 2, a read_item(X) is always successful, since it finds the appro-
3.2 Multiversion Two-Phase Locking Using Certify Locks
795
Concurrency Control Techniques
(b) Read Write
Read
Write
Certify
Yes No No
No No No
Yes Yes No
Certify
(a) Read Write
Read
Write No No
Yes No
Figure 6
transaction T�requests the type of lock specified in the row header on the same item
In the standard locking scheme, once a transaction obtains a write lock on an item,
In this multiversion 2PL scheme, reads can proceed concurrently with a single write
796
Concurrency Control Techniques
4 Validation (Optimistic) Concurrency Control
In all the concurrency control techniques we have discussed so far, a certain degree
In optimistic concurrency control techniques, also known as validation or
There are three phases for this concurrency control protocol:
1. Read phase. A transaction can read values of committed data items from the
2. Validation phase. Checking is performed to ensure that serializability will
3. Write phase. If the validation phase is successful, the transaction updates are
The idea behind optimistic concurrency control is to do all the checks at once;
The optimistic protocol we describe uses transaction timestamps and also requires
6Note that this can be considered as keeping multiple versions of items!
797
Concurrency Control Techniques
each transaction. Recall that the write_set of a transaction is the set of items it writes,
1. Transaction Tj completes its write phase before Ti starts its read phase.
2. Ti starts its write phase after Tj completes its write phase, and the read_set
3. Both the read_set and write_set of Ti have no items in common with the
When validating transaction Ti, the first condition is checked first for each transac-
5 Granularity of Data Items and Multiple
All concurrency control techniques assume that the database is formed of a number
■ A database record
■ A field value of a database record
■ A disk block
■ A whole file
■ The whole database
The granularity can affect the performance of concurrency control and recovery. In
5.1 Granularity Level Considerations for Locking
798
Concurrency Control Techniques
db
r111 r11j r121 r12j r1n1 r1nj r211 r21k r221 r22k r2m1 r2mk. . . . . . . . .
. . .
. . . . . . . . . . . .
. . .
. . .
p11 p12
f1
p1n p21 p22 p2m
f2
Figure 7
First, notice that the larger the data item size is, the lower the degree of concurrency
On the other hand, the smaller the data item size is, the more the number of items
Given the above tradeoffs, an obvious question can be asked: What is the best item
5.2 Multiple Granularity Level Locking
799
Concurrency Control Techniques
Consider the following scenario, with only shared and exclusive lock types, that refers
However, what if transaction T2’s request came before transaction T1’s request? In
To make multiple granularity level locking practical, additional types of locks, called
1. Intention-shared (IS) indicates that one or more shared locks will be
2. Intention-exclusive (IX) indicates that one or more exclusive locks will be
3. Shared-intention-exclusive (SIX) indicates that the current node is locked in
The compatibility table of the three intention locks, and the shared and exclusive
1. The lock compatibility (based on Figure 8) must be adhered to.
2. The root of the tree must be locked first, in any mode.
3. A node N can be locked by a transaction T in S or IS mode only if the parent
4. A node N can be locked by a transaction T in X, IX, or SIX mode only if the
5. A transaction T can lock a node only if it has not unlocked any node (to
800
Concurrency Control Techniques
IS
IX
S
SIX
X
IS
Yes
Yes
Yes
Yes
No
IX
Yes
No
Yes
No
No
S
No
Yes
Yes
No
No
SIX
No
No
Yes
No
No
X
No
No
No
No
No
6. A transaction T can unlock a node, N, only if none of the children of node N
Rule 1 simply states that conflicting locks cannot be granted. Rules 2, 3, and 4 state
1. T1 wants to update record r111 and record r211.
2. T2 wants to update all records on page p12.
3. T3 wants to read record r11j and the entire f2 file.
Figure 9 shows a possible serializable schedule for these three transactions. Only the(- ) is used to
The multiple granularity level protocol is especially suited when processing a mix of
6 Using Locks for Concurrency
Two-phase locking can also be applied to indexes, where the nodes of an index cor-
801
Concurrency Control Techniques
IX(db)
T1
IX(p11)
IX(f2)
unlock(r211)
unlock(r111)
T3
IS(db)
S(r11j)
S(f2)
unlock(r11j)
IX(db)
T2
IX(f1)
unlock(p12)
Figure 9
The tree structure of the index can be taken advantage of when developing a con-
A conservative approach for insertions would be to lock the root node in exclusive
802
Concurrency Control Techniques
not full, then the lock on the root node can be released. This approach can be
Another approach to index locking is to use a variant of the B+-tree, called the B-
Handling the deletion case, where two or more nodes from the index tree merge, is
7 Other Concurrency Control Issues
7.1 Insertion, Deletion, and Phantom Records
803
Concurrency Control Techniques
Next, consider a deletion operation that is applied on an existing data item. For
A situation known as the phantom problem can occur when a new record that is
One solution to the phantom record problem is to use index locking, as discussed
A more general technique, called predicate locking, would lock access to all records
7.2 Interactive Transactions
An approach to dealing with this problem is to postpone output of transactions to
804
Concurrency Control Techniques
7.3 Latches
8 Summary
We presented other concurrency control protocols that are not used often in prac-
Then we turned our attention to the important practical issue of data item granu-
805
Concurrency Control Techniques
Review Questions
ity?
2. What are some variations of the two-phase locking protocol? Why is strict or
3. Discuss the problems of deadlock and starvation, and the different
4. Compare binary locks to exclusive/shared locks. Why is the latter type of
5. Describe the wait-die and wound-wait protocols for deadlock prevention.
6. Describe the cautious waiting, no waiting, and timeout protocols for dead-
7. What is a timestamp? How does the system generate timestamps?
8. Discuss the timestamp ordering protocol for concurrency control. How does
9. Discuss two multiversion techniques for concurrency control.
10. What is a certify lock? What are the advantages and disadvantages of using
11. How do optimistic concurrency control techniques differ from other con-
12. How does the granularity of data items affect the performance of concur-
13. What type of lock is needed for insert and delete operations?
14. What is multiple granularity locking? Under what circumstances is it used?
15. What are intention locks?
16. When are latches used?
17. What is a phantom record? Discuss the problem that a phantom record can
18. How does index locking resolve the phantom problem?
19. What is a predicate lock?
806
Concurrency Control Techniques
Exercises
ability of schedules. (Hint: Show that if a serializability graph for a schedule
21. Modify the data structures for multiple-mode locks and the algorithms for
22. Prove that strict two-phase locking guarantees strict schedules.
23. Prove that the wait-die and wound-wait protocols avoid deadlock and star-
24. Prove that cautious waiting avoids deadlock.
25. Apply the timestamp ordering algorithm to the schedules in Figure A.1(b)
26. Repeat Exercise 25, but use the multiversion timestamp ordering method.
27. Why is two-phase locking not used as a concurrency control method for
28. The compatibility matrix in Figure 8 shows that IS and IX locks are compat-
29. The MGL protocol states that a transaction T can unlock a node N, only if
Selected Bibliography
807
Concurrency Control Techniques
Goodman (1983) discuss multiversion techniques. Multiversion timestamp order-
Other work on concurrency control includes semantic-based concurrency control
808
Transaction T1
write_item(X );
read_item(Y );
write_item(Y );
read_item(X );
read_item(Y );
Transaction T3
read_item(Z );
write_item(Y );
write_item(Z );
read_item(Y );
write_item(Y);
Transaction T2
read_item(Y );
write_item(Y );
read_item(X );
write_item(X );
read_item(Z );
read_item(X );
write_item(X );
(b)
(a)
Schedule E
Time
read_item(X );
read_item(Y );
read_item(Y );
write_item(Y );
read_item(Z );
read_item(Y );
(c)
Schedule F
Time
Transaction T1 Transaction T2 Transaction T3
Transaction T1 Transaction T2 Transaction T3
Figure A.1
809
Database Recovery
In this chapter we discuss some of the techniques thatcan be used for database recovery from failures.
Our emphasis is on conceptually describing several different approaches to recov-
From Chapter 23 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
810
Database Recovery Techniques
concurrency control mechanisms. Certain recovery techniques are best used with
1 Recovery Concepts
1.1 Recovery Outline and Categorization
Recovery from transaction failures usually means that the database is restored to the
1. If there is extensive damage to a wide portion of the database due to cata-
2. When the database on disk is not physically damaged, and a noncatastrophic
Conceptually, we can distinguish two main techniques for recovery from noncata-
811
Database Recovery Techniques
database in any way, so UNDO is not needed. It may be necessary to REDO the
In the immediate update techniques, the database may be updated by some opera-
The UNDO and REDO operations are required to be idempotent—that is, executing
1.2 Caching (Buffering) of Disk Blocks
In general, it is convenient to consider recovery in terms of the database disk pages entries.
1This is somewhat similar to the concept of page tables used by the operating system.
812
Database Recovery Techniques
not, the item must be located on disk, and the appropriate disk pages are copied into
The entries in the DBMS cache directory hold additional information relevant to
Two main strategies can be employed when flushing a modified buffer back to disk.
Hence, a single copy of each database disk block is maintained. The second strategy,
In general, the old value of the data item before updating is called the before image
1.3 Write-Ahead Logging, Steal/No-Steal,
When in-place updating is used, it is necessary to use a log for recovery. In this case,
2In-place updating is used in most systems in practice.
813
Database Recovery Techniques
logging, we need to distinguish between two types of log entry information
As mentioned, the DBMS cache holds the cached database disk blocks in main
Standard DBMS recovery terminology includes the terms steal/no-steal and
1. If a cache buffer page updated by a transaction cannot be written to disk
2. If all pages updated by a transaction are immediately written to disk before
The deferred update (NO-UNDO) recovery scheme discussed in Section 2 follows a
814
Database Recovery Techniques
page of a committed transaction may still be in the buffer when another transaction
To permit recovery when in-place updating is used, the appropriate entries required
1. The before image of an item cannot be overwritten by its after image in the
2. The commit operation of a transaction cannot be completed until all the
To facilitate the recovery process, the DBMS recovery subsystem may need to main-
1.4 Checkpoints in the System Log
Another type of entry in the log is called a checkpoint.3 A [checkpoint, list of active
The recovery manager of a DBMS must decide at what intervals to take a check-
1. Suspend execution of transactions temporarily.
2. Force-write all main memory buffers that have been modified to disk.
3The term checkpoint has been used to describe more restrictive situations in some systems, such as
815
Database Recovery Techniques
3. Write a [checkpoint] record to the log, and force-write the log to disk.
4. Resume executing transactions.
As a consequence of step 2, a checkpoint record in the log may also include addi-
The time needed to force-write all modified memory buffers may delay transaction
1.5 Transaction Rollback and Cascading Rollback
If a transaction T is rolled back, any transaction S that has, in the interim, read the
Figure 1 shows an example where cascading rollback is required. The read and write
816
(a)
(b)
*
**
**
[start_transaction,T3]
[read_item,T3,C]
[write_item,T3,B,15,12]
[read_item,T2,B]
[write_item,T2,B,12,18]
[write_item,T1,D,20,25]
[write_item,T2,D,25,26]
[start_transaction,T1]
[read_item,T1,D]
[read_item,T2,D]
[read_item,T3,A]
* T3 is rolled back because it
** T2 is rolled back because it
read_item(A)
read_item(D)
read_item(B)
write_item(B)
write_item(D)
System crash
T3T2T1
write_item(B)
write_item(A)
A
30
B
15
12
18
C
40
D
20
25
26
(c) READ(C)
BEGIN
READ(A)WRITE(B)
T3
READ(B)
BEGIN
WRITE(D)READ(D)WRITE(B)
READ(A)
BEGIN
System crash
Time
READ(D) WRITE(D)
T2
T1
Figure 1
Database Recovery Techniques
817
We must now check for cascading rollback. From Figure 1(c) we see that transaction
In practice, cascading rollback of transactions is never required because practical
1.6 Transaction Actions That Do Not Affect
In general, a transaction will have actions that do not affect the database, such as
2 NO-UNDO/REDO Recovery Based
The idea behind deferred update is to defer or postpone any actual updates to the
During transaction execution, the updates are recorded only in the log and in the
Database Recovery Techniques
4Hence deferred update can generally be characterized as a no-steal approach.
818
Database Recovery Techniques
and each transaction changes few items. For other types of transactions, there is the
We can state a typical deferred update protocol as follows:
1. A transaction cannot change the database on disk until it reaches its commit
2. A transaction does not reach its commit point until all its REDO-type log
Notice that step 2 of this protocol is a restatement of the write-ahead logging (WAL)
For multiuser systems with concurrency control, the concurrency control and
Procedure RDU_M (NO-UNDO/REDO with checkpoints). Use two lists of
The REDO procedure is defined as follows:
Procedure REDO (WRITE_OP). Redoing a write_item operation WRITE_OP con-
Figure 2 illustrates a timeline for a possible schedule of executing transactions.
819
Database Recovery Techniques
System crash TimeCheckpoint
T2
T3
T5
T4
t1 t2
Figure 2
We can make the NO-UNDO/REDO recovery algorithm more efficient by noting that,
If a transaction is aborted for any reason (say, by the deadlock detection method), it
1. A transaction does not record any changes in the database on disk until after
2. A transaction will never read the value of an item that is written by an
Figure 3 shows an example of recovery for a multiuser system that utilizes the recov-
3 Recovery Techniques Based
In these techniques, when a transaction issues an update command, the database on
820
Database Recovery Techniques
(a) T1
write_item(D)
[checkpoint]
(b)
read_item(B)
read_item(D)
write_item(D)
read_item(A)
read_item(C)
write_item(C)
read_item(B)
read_item(A)
write_item(A)
[start_transaction,T1]
[start_transaction, T2]
[write_item, T1, D, 20]
[commit, T1]
[commit, T4]
[start_transaction, T4]
[start_transaction, T3]
[write_item, T4, B, 15]
[write_item, T2, B, 12]
[write_item, T4, A, 20]
[write_item, T3, A, 30]
[write_item,T2, D, 25]
T2 and T3 are ignored because they did not reach their commit points.
T4 is redone because its commit point is after the last system checkpoint.
System crash
T2 T3 T4
Figure 3
applied immediately to disk; it is just possible that some updates are applied to disk
Provisions must be made for undoing the effect of update operations that have been
821
Database Recovery Techniques
the force strategy for deciding when updated main memory buffers are written
If the transaction is allowed to commit before all its changes are written to the data-
When concurrent execution is permitted, the recovery process again depends on the
Procedure RIU_M (UNDO/REDO with checkpoints).
1. Use two lists of transactions maintained by the system: the committed trans-
2. Undo all the write_item operations of the active (uncommitted) transactions,
3. Redo all the write_item operations of the committed transactions from the log,
The UNDO procedure is defined as follows:
Procedure UNDO (WRITE_OP). Undoing a write_item operation write_op con-
As we discussed for the NO-UNDO/REDO procedure, step 3 is more efficiently done
822
Database Recovery Techniques
Current directory
Database disk
Shadow directory
Page 5 (old)
Page 1
Page 4
Page 2 (old)
Page 3
Page 6
Page 2 (new)
Page 5 (new)
1
2
3
4
5
6
1
2
3
4
5
6
Figure 4
beginning of the log). Whenever an item is undone, it is added to a list of undone
4 Shadow Paging
During transaction execution, the shadow directory is never modified. When a
5The directory is similar to the page table maintained by the operating system for each process.
823
Database Recovery Techniques
To recover from a failure during transaction execution, it is sufficient to free the
In a multiuser environment with concurrent transactions, logs and checkpoints must
5 The ARIES Recovery Algorithm
The ARIES recovery procedure consists of three main steps: analysis, REDO, and
6The actual buffers may be lost during a crash, since they are in main memory. Additional tables stored in
824
Database Recovery Techniques
which REDO operations are applied until the end of the log is reached. Additionally,
In ARIES, every log record has an associated log sequence number (LSN) that is
Common fields in all log records include the previous LSN for that transaction, the
Besides the log, two tables are needed for efficient recovery: the Transaction Table
Checkpointing in ARIES consists of the following: writing a begin_checkpoint record
825
Database Recovery Techniques
occurs during checkpointing, the special file will refer to the previous checkpoint,
After a crash, the ARIES recovery manager takes over. Information from the last
The REDO phase follows next. To reduce the amount of unnecessary work, ARIES
We briefly illustrate horizontal and vertical propagation limits—which are not
3 Mandatory Access Control and Role-Based
The discretionary access control technique of granting and revoking privileges on
Typical security classes are top secret (TS), secret (S), confidential (C), and unclas-
848
Database Security
account, program) and object (relation, tuple, column, view, operation) into one of
1. A subject S is not allowed read access to an object O unless class(S) ≥
2. A subject S is not allowed to write an object O unless class(S) ≤ class(O). This
The first restriction is intuitive and enforces the obvious rule that no subject can
To incorporate multilevel security notions into the relational database model, it is
R(A1, C1, A2, C2, …, An, Cn, TC)
where each Ci represents the classification attribute associated with attribute Ai.
The value of the tuple classification attribute TC in each tuple t—which is the
The apparent key of a multilevel relation is the set of attributes that would have
849
Database Security
This leads to the concept of polyinstantiation,4 where several tuples can have the
We illustrate these concepts with the simple example of a multilevel relation shown
Name Salary JobPerformance TC
EMPLOYEE(d)
Name Salary JobPerformance TC
EMPLOYEE(a)
Name Salary JobPerformance TC
Smith U C40000 CNULL C
EMPLOYEE(b)
Name Salary JobPerformance TC
Smith U UNULL UNULL U
EMPLOYEE(c)
Figure 2
4This is similar to the notion of having multiple versions in the database that represent the same real-
850
Database Security
attributes appearing as null (Figure 2(c)). Thus, filtering introduces null values for
In general, the entity integrity rule for multilevel relations states that all attributes
To illustrate polyinstantiation further, suppose that a user with security clearance C
UPDATE EMPLOYEE
Since the view provided to users with security clearance C (see Figure 2(b)) permits
The basic update operations of the relational model (INSERT, DELETE, UPDATE)
3.1 Comparing Discretionary Access Control
Discretionary access control (DAC) policies are characterized by a high degree of
851
Database Security
any illegal flow of information. Therefore, they are suitable for military and high
3.2 Role-Based Access Control
RBAC can be used with traditional discretionary and mandatory access controls; it
Separation of duties is another important requirement in various commercial
The role hierarchy in RBAC is a natural way to organize roles to reflect the organi-
852
Database Security
antisymmetric. In other words, if a user has one role, the user automatically has
GRANT ROLE full_time TO employee_type1
The above are examples of granting the roles full_time and intern to two types of
Another issue related to security is identity management. Identity refers to a unique
Another important consideration in RBAC systems is the possible temporal con-
RBAC models have several desirable features, such as flexibility, policy neutrality,
3.3 Label-Based Security and Row-Level Access Control
853
Database Security
levels to exposure or corruption, with the goal of maintaining privacy or security.
A policy defined by an administrator is called a Label Security policy. Whenever
The Label Security requirements are applied on top of the DAC requirements for
Security policies are generally created by managers and human resources personnel.
Suppose a user has SELECT privileges on a table. When the user executes a SELECT
3.4 XML Access Control
854
Database Security
signatures and encryption standards for XML. The XML Signature Syntax and
The XML Encryption Syntax and Processing specification defines XML vocabulary
3.5 Access Control Policies for E-Commerce and the Web
A second related requirement is the support for content-based access control.
A third requirement is related to the heterogeneity of subjects, which requires access
855
Database Security
within an organization). For instance, by using credentials, one can simply formu-
It is believed that the XML is expected to play a key role in access control for
4 SQL Injection
■ Unauthorized privilege escalation. This attack is characterized by an indi-
■ Privilege abuse. While the previous attack is done by an unauthorized user,
■ Denial of service. A Denial of Service (DOS) attack is an attempt to make
■ Weak Authentication. If the user authentication scheme is weak, an attacker
5See Thuraisingham et al. (2001).
856
Database Security
4.1 SQL Injection Methods
SQL Manipulation. A manipulation attack, which is the most common type of
SELECT * FROM users WHERE username = ‘jake’ and PASSWORD =
The attacker can try to change (or manipulate) the SQL statement, by changing it as
SELECT * FROM users WHERE username = ‘jake’ and (PASSWORD =
As a result, the attacker who knows that ‘jake’ is a valid login of some user is able to
Code Injection. This type of attack attempts to add additional SQL statements or
Function Call Injection. In this kind of attack, a database function or operating
857
Database Security
For example, the dual table is used in the FROM clause of SQL in Oracle when a user
SELECT SYSDATE FROM dual;
The following example demonstrates that even the simplest SQL statements can be
SELECT TRANSLATE (‘user input’, ‘from_string’, ‘to_string’) FROM dual;
Here, TRANSLATE is used to replace a string of characters with another string of
This type of SQL statement can be subjected to a function injection attack. Consider
SELECT TRANSLATE (“ || UTL_HTTP.REQUEST (‘http://129.107.2.1/’) || ’’,
The user can input the string (“ || UTL_HTTP.REQUEST (‘http://129.107.2.1/’)
4.2 Risks Associated with SQL Injection
■ Database Fingerprinting. The attacker can determine the type of database
■ Denial of Service. The attacker can flood the server with requests, thus
■ Bypassing Authentication. This is one of the most common risks, in which
858
Database Security
■ Identifying Injectable Parameters. In this type of attack, the attacker gath-
■ Executing Remote Commands. This provides attackers with a tool to exe-
■ Performing Privilege Escalation. This type of attack takes advantage of log-
4.3 Protection Techniques against SQL Injection
Bind Variables (Using Parameterized Statements). The use of bind variables
Consider the following example using Java and JDBC:
PreparedStatement stmt = conn.prepareStatement( “SELECT * FROM
stmt.setString(1, employee_id);
stmt.setString(2, password);
Instead of embedding the user input into the statement, the input should be bound
Filtering Input (Input Validation). This technique can be used to remove escape
Function Security. Database functions, both standard and custom, should be
859
Database Security
5 Introduction to Statistical Database Security
A population is a set of tuples of a relation (table) that satisfy some selection condi-
Statistical queries involve applying statistical functions to a population of tuples.
It is the responsibility of a database management system to ensure the confidential-
In some cases it is possible to infer the values of individual tuples from a sequence
Name Ssn Income Address City State Zip Sex Last_degree
PERSON Figure 3
860
Database Security
population consisting of a small number of tuples. As an illustration, consider the
Q1: SELECT COUNT (*) FROM PERSON;
Q2: SELECT AVG (Income) FROM PERSON;
Now suppose that we are interested in finding the Salary of Jane Smith, and we know
(Last_degree=‘Ph.D.’ AND Sex=‘F’ AND City=‘Bellaire’ AND State=‘Texas’)
If we get a result of 1 for this query, we can issue Q2 with the same condition and
The possibility of inferring individual information from statistical queries is
6 Introduction to Flow Control
A flow policy specifies the channels along which information is allowed to move.
861
Database Security
and nonconfidential (N)—and allows all flows except those from class C to class N.
Access control mechanisms are responsible for checking users’ authorizations for
Two types of flow can be distinguished: explicit flows, occurring as a consequence of
Flow control mechanisms must verify that only authorized flows, both explicit and
6.1 Covert Channels
In a simple example of a covert channel, consider a distributed database system in
862
Database Security
for a transaction to commit, both nodes must agree to commit. They mutually can
Some security experts believe that one way to avoid covert channels is to disallow
7 Encryption and Public
The previous methods of access and flow control, despite being strong control
With this background, we adhere to following standard definitions:6
■ Ciphertext: Encrypted (enciphered) data.
6These definitions are from NIST (National Institute of Standards and Technology) from http://csrc.nist
863
Database Security
■ Plaintext (or cleartext): Intelligible data that has meaning and can be read or
■ Encryption: The process of transforming plaintext into ciphertext.
■ Decryption: The process of transforming ciphertext back into plaintext.
Encryption consists of applying an encryption algorithm to data using some pre-
7.1 The Data Encryption and Advanced
The Data Encryption Standard (DES) is a system developed by the U.S. govern-
7.2 Symmetric Key Algorithms
The major liability associated with secret-key algorithms is the need for sharing the
7.3 Public (Asymmetric) Key Encryption
864
Database Security
functions rather than operations on bit patterns. They address one drawback of
A public key encryption scheme, or infrastructure, has six ingredients:
1. Plaintext. This is the data or readable message that is fed into the algorithm
2. Encryption algorithm. This algorithm performs various transformations
3. and 4. Public and private keys. These are a pair of keys that have been
5. Ciphertext. This is the scrambled message produced as output. It depends
6. Decryption algorithm. This algorithm accepts the ciphertext and the
As the name suggests, the public key of the pair is made public for others to use,
1. Each user generates a pair of keys to be used for the encryption and decryp-
2. Each user places one of the two keys in a public register or other accessible
3. If a sender wishes to send a private message to a receiver, the sender encrypts
865
Database Security
4. When the receiver receives the message, he or she decrypts it using the
The RSA Public Key Encryption Algorithm. One of the first public key schemes
Two keys, d and e, are used for decryption and encryption. An important property is
7.4 Digital Signatures
A digital signature consists of a string of symbols. If a person’s digital signature were
7.5 Digital Certificates
866
Database Security
statement. Certificates are issued and signed by a certification authority (CA). The
The digital certificate itself contains various types of information. For example,
1. The certificate owner information, which is represented by a unique identi-
2. The certificate also includes the public key of the owner.
3. The date of issue of the certificate is also included.
4. The validity period is specified by ‘Valid From’ and ‘Valid To’ dates, which are
5. Issuer identifier information is included in the certificate.
6. Finally, the digital signature of the issuing CA for the certificate is included.
8 Privacy Issues and Preservation
If all data were available at a single warehouse, violating only a single repository’s
Privacy is an important area of ongoing research in database management. It is
867
Database Security
requirements while they are being defined and enforced. Providing access control
9 Challenges of Database Security
9.1 Data Quality
9.2 Intellectual Property Rights
9.3 Database Survivability
868
Database Security
in addition to making every effort to prevent an attack and detecting one in the
■ Confinement. Take immediate action to eliminate the attacker’s access to the
■ Damage assessment. Determine the extent of the problem, including failed
■ Reconfiguration. Reconfigure to allow operation to continue in a degraded
■ Repair. Recover corrupted or lost data and repair or reinstall failed system
■ Fault treatment. To the extent possible, identify the weaknesses exploited in
The goal of the information warfare attacker is to damage the organization’s opera-
To date, issues related to database survivability have not been sufficiently investi-
10 Oracle Label-Based Security
10.1 Virtual Private Database (VPD) Technology
VPD provides access control based on policies. These VPD policies enforce object-
869
Database Security
interface (API) that allows security policies to be attached to database tables or
VPD is enabled by associating a security “policy” with a table, view, or synonym. An
10.2 Label Security Architecture
Figure 4 shows the sequence of discretionary access control (DAC) and label secu-
10.3 How Data Labels and User Labels Work Together
Compartments allow a finer classification of sensitivity of the labeled data. All data
870
Database Security
Oracle User
Request for Data in SQL
Check DAC
Control
Check Virtual Private
Process and Execute
Enforce Label-
Oracle Data Server
Table Level
Table
Data Rows
Label Security
Row Level
VPD-Based
User-Defined
Figure 4
Source: Oracle (2007)
Groups are used to identify organizations as owners of the data with corresponding
If a user has a maximum level of SENSITIVE, then the user potentially has access to
As shown in Figure 5, User 1 can access the rows 2, 3, and 4 because his maximum
11 Summary
871
Database Security
User Labels
HS FIN : WR
S FIN : WR_SAL
Legend for Labels
Maximum
All compartments to which
Minimum
Required
All compartments to which
User Label
Data Label
Rows in Table Data Labels
Row 1
Row 2
Row 3
Row 4
S CHEM, FIN : WR
HS FIN : WR_SAL
U FIN
C FIN : WR_SAL
Figure 5
Source: Oracle (2007)
the introduction we covered various issues related to security including data sensi-
Security enforcement deals with controlling access to the database system as a whole
872
Database Security
idea of the various ways SQL injection can be prevented. The issues related to flow
Review Questions
tion, access control, data encryption, privileged (system) account, database
2. Which account is designated as the owner of a relation? What privileges does
3. How is the view mechanism used as an authorization mechanism?
4. Discuss the types of privileges at the account level and those at the relation
5. What is meant by granting a privilege? What is meant by revoking a
6. Discuss the system of propagation of privileges and the restraints imposed
7. List the types of privileges available in SQL.
8. What is the difference between discretionary and mandatory access control?
9. What are the typical security classifications? Discuss the simple security
10. Describe the multilevel relational data model. Define the following terms:
11. What are the relative merits of using DAC or MAC?
12. What is role-based access control? In what ways is it superior to DAC and
13. What are the two types of mutual exclusion in role-based access control?
14. What is meant by row-level access control?
15. What is label security? How does an administrator enforce it?
873
Database Security
16. What are the different types of SQL injection attacks?
17. What risks are associated with SQL injection attacks?
18. What preventive measures are possible against SQL injection attacks?
19. What is a statistical database? Discuss the problem of statistical database
20. How is privacy related to statistical database security? What measures can be
21. What is flow control as a security measure? What types of flow control exist?
22. What are covert channels? Give an example of a covert channel.
23. What is the goal of encryption? What process is involved in encrypting data
24. Give an example of an encryption algorithm and explain how it works.
25. Repeat the previous question for the popular RSA algorithm.
26. What is a symmetric key algorithm for key-based security?
27. What is the public key infrastructure scheme? How does it provide security?
28. What are digital signatures? How do they work?
29. What type of information does a digital certificate include?
Exercises
31. What are some of the current outstanding challenges for database security?
32. Consider the relational database schema in Figure A.1 (at the end of this
a. Account A can retrieve or modify any relation except DEPENDENT and
b. Account B can retrieve all the attributes of EMPLOYEE and DEPARTMENT
c. Account C can retrieve or modify WORKS_ON but can only retrieve the
d. Account D can retrieve any attribute of EMPLOYEE or DEPENDENT and
e. Account E can retrieve any attribute of EMPLOYEE but only for
f. Write SQL statements to grant these privileges. Use views where
874
Database Security
33. Suppose that privilege (a) of Exercise 32 is to be given with GRANT OPTION
34. Consider the relation shown in Figure 2(d). How would it appear to a user
Selected Bibliography
Many papers discuss different techniques for the design and protection of statistical
Halfond et al. (2006) helps understand the concepts of SQL injection attacks and
Multilevel security is discussed in Jajodia and Sandhu (1991), Denning et al. (1987),
875
Database Security
Farahmand et al. (2005). Row-level access control is explained in detail in Oracle
Recent advances as well as future challenges for security and privacy of databases are
DEPARTMENT
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
EMPLOYEE
DEPT_LOCATIONS
Dnumber Dlocation
PROJECT
Pname Pnumber Plocation Dnum
WORKS_ON
Essn Pno Hours
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Dname Dnumber Mgr_ssn Mgr_start_date
Figure A.1
876
Distributed Databases
In this chapter we direct our attention to distributeddatabases (DDBs), distributed database management
DDB technology resulted from a merger of two technologies: database technology,
Several distributed database prototype systems were developed in the 1980s to
From Chapter 25 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
877
Distributed Databases
Organizations continue to be interested in the decentralization of processing (at the
In this chapter we discuss distributed databases, their architectural variations, and
Section 1 introduces distributed database management and related concepts.
For a short introduction to the topic of distributed databases, Sections 1, 2, and 3
1 Distributed Database Concepts1
We can define a distributed database (DDB) as a collection of multiple logically
Distributed databases are different from Internet Web files. Web pages are basically
1The substantial contribution of Narasimhan Srinivasan to this and several other sections in this chapter
878
Distributed Databases
Websites in various forms does not qualify as a DDB by the definition given earlier.
1.1 Differences between DDB and Multiprocessor Systems
■ Connection of database nodes over a computer network. There are multi-
■ Logical interrelation of the connected databases. It is essential that the
■ Absence of homogeneity constraint among connected nodes. It is not nec-
The sites may all be located in physical proximity—say, within the same building or
Networks may have different topologies that define the direct communication
1.2 Transparency
879
Distributed Databases
sites connected by a computer network, so additional types of transparencies are
Consider the company database in Figure A.1 in Appendix: Figures at the end of this
■ Data organization transparency (also known as distribution or network
■ Replication transparency. As we show in Figure 1, copies of the same data
■ Fragmentation transparency. Two types of fragmentation are possible.
EMPLOYEES
PROJECTS
All
All
EMPLOYEES
PROJECTS
WORKS_ON
San Francisco
San Francisco
San Francisco
EMPLOYEES
PROJECTS
WORKS_ON
Los Angeles
Los Angeles and
Los Angeles
EMPLOYEES
PROJECTS
WORKS_ON
New York
All
New York
EMPLOYEES
PROJECTS
WORKS_ON
Atlanta
Atlanta
Atlanta
Chicago
New York
Los Angeles Atlanta
San Francisco
Communications
Figure 1
880
Distributed Databases
that are subsets of the tuples (rows) in the original relation. Vertical frag-
■ Other transparencies include design transparency and execution trans-
1.3 Autonomy
1.4 Reliability and Availability
To construct a system that is reliable, we can adopt several approaches. One com-
881
Distributed Databases
1.5 Advantages of Distributed Databases
1. Improved ease and flexibility of application development. Developing and
2. Increased reliability and availability. This is achieved by the isolation of
3. Improved performance. A distributed DBMS fragments the database by
4. Easier expansion. In a distributed environment, expansion of the system in
The transparencies we discussed in Section 1.2 lead to a compromise between ease
882
Distributed Databases
1.6 Additional Functions of Distributed Databases
■ Keeping track of data distribution. The ability to keep track of the data dis-
■ Distributed query processing. The ability to access remote sites and trans-
■ Distributed transaction management. The ability to devise execution
■ Replicated data management. The ability to decide which copy of a repli-
■ Distributed database recovery. The ability to recover from individual site
■ Security. Distributed transactions must be executed with the proper man-
■ Distributed directory (catalog) management. A directory contains infor-
These functions themselves increase the complexity of a DDBMS over a centralized
2 Types of Distributed Database Systems
883
Distributed Databases
The first factor we consider is the degree of homogeneity of the DDBMS software.
Figure 2 shows classification of DDBMS alternatives along orthogonal axes of dis-
B
Distribution
Heterogeneity
Legend:
A: Traditional centralized database
B: Pure distributed database systems
C: Federated database systems
D: Multidatabase or peer to peer
C D
A
Figure 2
884
Distributed Databases
a very high degree of local autonomy. The term federated database system (FDBS)
We briefly discuss the issues affecting the design of FDBSs next.
2.1 Federated Database Management Systems Issues
■ Differences in data models. Databases in an organization come from a vari-
■ Differences in constraints. Constraint facilities for specification and imple-
3The term multidatabase system is not easily applicable to most enterprise IT environments. The notion of
885
Distributed Databases
■ Differences in query languages. Even with the same data model, the lan-
Semantic Heterogeneity. Semantic heterogeneity occurs when there are differ-
■ The universe of discourse from which the data is drawn. For example, for
■ Representation and naming. The representation and naming of data ele-
■ The understanding, meaning, and subjective interpretation of data. This is
■ Transaction and policy constraints. These deal with serializability criteria,
■ Derivation of summaries. Aggregation, summarization, and other data-
The above problems related to semantic heterogeneity are being faced by all major
Just as providing the ultimate transparency is the goal of any distributed database
886
Distributed Databases
refers to the ability of a component DBS to execute local operations without inter-
3 Distributed Database Architectures
3.1 Parallel versus Distributed Architectures
■ Shared memory (tightly coupled) architecture. Multiple processors share
■ Shared disk (loosely coupled) architecture. Multiple processors share sec-
These architectures enable processors to communicate without the overhead of
4If both primary and secondary memories are shared, the architecture is also known as shared everything
887
Distributed Databases
parallel databases. Figure 3a illustrates a parallel database (shared nothing), whereas
(a)
(b)
Switch
CPU
Computer System 1
Memory
DB CPU
Computer System 2
Memory
DB
CPU
Memory
DB
Computer System n
Central Site
Site
Site
Site
Site
DB1 DB2
Communications
(c)
Site 1
Site 2
Site 4
Site 3
Communications
Figure 3
888
Distributed Databases
User
Stored
Global Conceptual Schema (GCS)
External
User
External
Local Conceptual Schema (LCS) Local Conceptual Schema (LCS)
Local Internal Schema (LIS) Local Internal Schema (LIS)
Stored
Site 1 Site nSites 2 to n–1
Figure 4
3.2 General Architecture of Pure Distributed Databases
889
Distributed Databases
User
Interactive Global Query
Stored
Global Query Compiler
Global Query Optimizer
Global Transaction Manager
Local Transaction Manager
Local Query
and Execution
Local
Stored
Local Transaction Manager
Local Query
and Execution
Local
Figure 5
ate results. The latter is particularly important in queries involving joins. Having
3.3 Federated Database Schema Architecture
890
External
Federated
. . .
. . .
. . .
. . .
. . .
Component
Local
Component
External
External
Federated
Export
Component
Local
Component
Export
Export
Distributed Databases
Figure 6
Source: Adapted from Sheth and
conceptual schema (full database definition) of a component database, and the
All the problems related to query processing, transaction processing, and directory
5For a detailed discussion of the autonomies and the five-level architecture of FDBMSs, see Sheth and
891
Distributed Databases
Client
(Web browser, HTML, JavaScript, Visual Basic, . . .)
HTTP Protocol
Application server
(Application program, JAVA, C/C++, C#, . . .)
Database server
(Database access, SQL, PSM, XML, . . .)
ODBC, JDBC, SQL/CLI, SQLJ
Figure 7
3.4 An Overview of Three-Tier Client-Server Architecture
In the three-tier client-server architecture, the following three layers exist:
1. Presentation layer (client). This provides the user interface and interacts
2. Application layer (business logic). This layer programs the application
892
Distributed Databases
as security checks, identity verification, and other functions. The application
3. Database server. This layer handles query and update requests from the
Exactly how to divide the DBMS functionality between the client, application
In this architecture, the application server may also refer to a data dictionary that
1. The application server formulates a user query based on input from the
2. Each database server processes the local query and sends the results to the
3. The application server combines the results of the subqueries to produce the
The application server is responsible for generating a distributed execution plan for
893
Distributed Databases
If the DDBMS has the capability to hide the details of data distribution from the
4 Data Fragmentation, Replication,
In this section we discuss techniques that are used to break up the database into log-
4.1 Data Fragmentation
Before we decide on how to distribute the data, we must determine the logical units
We may want to store the database information relating to each department at the
6Of course, in an actual situation, there will be many more tuples in the relation than those shown in
894
Distributed Databases
Horizontal Fragmentation. A horizontal fragment of a relation is a subset of
Vertical Fragmentation. Each site may not need all the attributes of a relation,
Notice that each horizontal fragment on a relation R can be specified in the rela-
(R) operation. A set of horizontal fragments whose conditions
A vertical fragment on a relation R can be specified by a πLi
tional algebra. A set of vertical fragments whose projection lists L1, L2, …, Ln include
895
Distributed Databases
complete vertical fragmentation of R. In this case the projection lists satisfy the fol-
■ L1 ∪ L2 ∪ … ∪ Ln = ATTRS(R).
PK(R) is the primary key of R.
To reconstruct the relation R from a complete vertical fragmentation, we apply the
Two horizontal fragments that are neither complete nor disjoint are those defined
Mixed (Hybrid) Fragmentation. We can intermix the two types of fragmenta-
A fragmentation schema of a database is a definition of a set of fragments that
An allocation schema describes the allocation of fragments to sites of the DDBS;
896
Distributed Databases
stored. If a fragment is stored at more than one site, it is said to be replicated. We
4.2 Data Replication and Allocation
The other extreme from full replication involves having no replication—that is,
Between these two extremes, we have a wide spectrum of partial replication of the
Each fragment—or each copy of a fragment—must be assigned to a particular site
7For a proposed scalable approach to synchronize partially replicated databases, see Mahajan et al.
897
Distributed Databases
4.3 Example of Fragmentation, Allocation, and Replication
According to these requirements, the whole database in Figure A.2 can be stored at
We must now fragment the WORKS_ON relation and decide which fragments of
In Figure 9, the union of fragments G1, G2, and G3 gives all WORKS_ON tuples for
898
Distributed Databases
(a)
(b)
Fname
John B Smith 123456789 30000 333445555 5
Franklin T Wong 333445555 40000 888665555 5
K Narayan 666884444 38000 333445555 5
A English 453453453 25000 333445555 5
Ramesh
Joyce
EMPD_5
Minit Lname Ssn Salary Super_ssn Dno
Data at site 2
Data at site 3
Fname
Alicia J Zelaya 999887777 25000 987654321 4
Jennifer S Wallace 987654321 43000 888665555 4
V Jabbar 987987987 25000 987654321 4Ahmad
EMPD_4
Minit Lname Ssn Salary Super_ssn Dno
Dname
Research 5 333445555 1988-05-22
DEP_5
Dnumber Mgr_ssn Mgr_start_date Dnumber
5 Bellaire
5 Sugarland
5 Houston
DEP_5_LOCS
Location
Dname
Administration 4 987654321 1995-01-01
DEP_4
Dnumber Mgr_ssn Mgr_start_date
Essn
123456789 1
123456789 2
666884444
453453453
453453453
333445555
333445555
333445555
333445555
1
2
2
3
10
20
3
32.5
7.5
20.0
20.0
10.0
10.0
10.0
10.0
40.0
WORKS_ON_5
Pno Hours Pname
Product X 1
Product Y 2
Product Z 3
Bellaire
Sugarland
Houston
PROJS_5
Pnumber Plocation
5
5
5
Dnum
Essn
333445555 10
999887777 30
999887777
987987987
987987987
987654321
987654321
10
30
30
20
10
10.0
30.0
35.0
5.0
20.0
15.0
10.0
WORKS_ON_4
Pno Hours Pname
Computerization 10
New_benefits 30
Stafford
Stafford
PROJS_4
Pnumber Plocation
4
4
Dnum
Dnumber
4 Stafford
DEP_4_LOCS
Location
Figure 8
899
Distributed Databases
Essn
123456789 1 32.5
123456789 2 7.5
3 40.0
1 20.0
2 20.0
2 10.0
3 10.0
666884444
453453453
453453453
333445555
333445555
G1
1C = C and (Pno in (SELECT
Employees in Department 5
Pno Hours Essn
333445555 10 10.0
G2
C2 = C and (Pno in (SELECT
Pno Hours Essn
333445555 20 10.0
G3
C3 = C and (Pno in (SELECT
Pno Hours
Essn
G4
(b)
(c)
(a)
C4 = C and (Pno in (SELECT
Employees in Department 4
Pno Hours Essn
999887777 30 30.0
999887777 10 10.0
987987987 10 35.0
987987987 30 5.0
987654321 30 20.0
G5
C5 = C and (Pno in (SELECT
Pno Hours Essn
987654321 20 15.0
G6
C6 = C and (Pno in (SELECT
Pno Hours
Essn
G7
C7 = C and (Pno in (SELECT
Employees in Department 1
Pno Hours Essn
G8
C8 = C and (Pno in (SELECT
Pno Hours Essn
888665555 20 Null
G9
C9 = C and (Pno in (SELECT
Pno Hours
Figure 9
900
Distributed Databases
an EMPLOYEE tuple or a PROJECT tuple at sites 2 and 3. Hence, we place the union
5 Query Processing and Optimization in
Now we give an overview of how a DDBMS processes and optimizes a query. First
5.1 Distributed Query Processing
1. Query Mapping. The input query on distributed data is specified formally
2. Localization. In a distributed database, fragmentation results in relations
3. Global Query Optimization. Optimization consists of selecting a strategy
8For a detailed discussion of optimization algorithms, see Ozsu and Valduriez (1999).
901
Distributed Databases
4. Local Query Optimization. This stage is common to all sites in the DDB.
The first three stages discussed above are performed at a central control site, while
5.2 Data Transfer Costs of Distributed Query Processing
We illustrate this with two simple sample queries. Suppose that the EMPLOYEE and
6 bytes, and the size
Q: πFname,Lname,Dname(EMPLOYEE Dno=Dnumber DEPARTMENT)
The result of this query will include 10,000 records, assuming that every employee is
Fname
EMPLOYEE
Site 1:
10,000 records
Site 2:
Minit Lname Ssn Salary Super_ssn DnoBdate Address Sex
Dname
DEPARTMENT
Dnumber Mgr_ssn Mgr_start_date
Fname field is 15 bytes long
100 records
Dname field is 10 bytes long
Figure 10
902
Distributed Databases
The query is submitted at a distinct site 3, which is called the result site because the
1. Transfer both the EMPLOYEE and the DEPARTMENT relations to the result
2. Transfer the EMPLOYEE relation to site 2, execute the join at site 2, and send
3. Transfer the DEPARTMENT relation to site 1, execute the join at site 1, and
If minimizing the amount of data transfer is our optimization criterion, we should
Q�: πFname,Lname,Dname( DEPARTMENT Mgr_ssn=Ssn EMPLOYEE)
Again, suppose that the query is submitted at site 3. The same three strategies for
1. Transfer both the EMPLOYEE and the DEPARTMENT relations to the result
2. Transfer the EMPLOYEE relation to site 2, execute the join at site 2, and send
3. Transfer the DEPARTMENT relation to site 1, execute the join at site 1, and
Again, we would choose strategy 3—this time by an overwhelming margin over
1. Transfer the EMPLOYEE relation to site 2, execute the query, and present the
2. Transfer the DEPARTMENT relation to site 1, execute the query at site 1, and
903
Distributed Databases
A more complex strategy, which sometimes works better than these simple strate-
5.3 Distributed Query Processing Using Semijoin
To illustrate this, consider the following strategy for executing Q or Q�:
1. Project the join attributes of DEPARTMENT at site 2, and transfer them to site
2. Join the transferred file with the EMPLOYEE relation at site 1, and transfer
3. Execute the query by joining the transferred file R or R� with DEPARTMENT,
Using this strategy, we transfer 340,400 bytes for Q and 4,800 bytes for Q�. We lim-
The semijoin operation was devised to formalize this strategy. A semijoin opera-
R
A=B S). In
B
then joining F with R, thus leading to the strategy discussed here.
Notice that the semijoin operation is not commutative; that is,
R S ≠S R
904
Distributed Databases
5.4 Query and Update Decomposition
On the other hand, a DDBMS that supports full distribution, fragmentation, and
To determine which replicas include the data items referenced in a query, the
In our earlier example, the guard conditions for fragments at site 1 (Figure A.2) are
For query decomposition, the DDBMS can determine which fragments may
905
Distributed Databases
(a) EMPD5
guard condition: Dno=5
attribute list: * (all attributes Dname, Dnumber, Mgr_ssn, Mgr_start_date)
attribute list: * (all attributes Dnumber, Location)
attribute list: * (all attributes Pname, Pnumber, Plocation, Dnum)
attribute list: * (all attributes Essn, Pno,Hours)
(b) EMPD4
guard condition: Dno=4
attribute list: * (all attributes Dname, Dnumber, Mgr_ssn, Mgr_start_date)
attribute list: * (all attributes Dnumber, Location)
attribute list: * (all attributes Pname, Pnumber, Plocation, Dnum)
attribute list: * (all attributes Essn, Pno, Hours)
OR Pno IN (πPnumber (PROJS4))
Figure 11
conditions. For example, consider the query Q: Retrieve the names and hours per
Q: SELECT Fname, Lname, Hours
906
Distributed Databases
Suppose that the query is submitted at site 2, which is where the query result will be
T1 ← πEssn(PROJS5 Pnumber=PnoWORKS_ON5)
This decomposition can be used to execute the query by using a semijoin strategy.
6 Overview of Transaction Management
The global and local transaction management software modules, along with the
As can be seen in Figure 5, an additional component called the global transaction
907
Distributed Databases
The transaction manager passes to the concurrency controller the database opera-
6.1 Two-Phase Commit Protocol
6.2 Three-Phase Commit Protocol
These problems are solved by the three-phase commit (3PC) protocol, which essen-
908
Distributed Databases
The main idea is to limit the wait time for participants who have committed and are
6.3 Operating System Support
The following are the main benefits of operating system (OS)-supported transac-
■ Typically, DBMSs use their own semaphores9 to guarantee mutually exclu-
■ Specialized hardware support for locking can be exploited to reduce associ-
■ Providing a set of common transaction support operations though the ker-
7 Overview of Concurrency Control
For concurrency control and recovery purposes, numerous problems arise in a dis-
■ Dealing with multiple copies of the data items. The concurrency control
9Semaphores are data structures used for synchronized and exclusive access to shared resources for
909
Distributed Databases
■ Failure of individual sites. The DDBMS should continue to operate with its
■ Failure of communication links. The system must be able to deal with the
■ Distributed commit. Problems can arise with committing a transaction that
■ Distributed deadlock. Deadlock may occur among several sites, so tech-
Distributed concurrency control and recovery techniques must deal with these and
7.1 Distributed Concurrency Control Based
To deal with replicated data items in a distributed database, a number of concur-
A number of different methods are based on this idea, but they differ in their
Primary Site Technique. In this method a single primary site is designated to be
910
Distributed Databases
of the centralized locking approach. For example, if all transactions follow the two-
Although all locks are accessed at the primary site, the items themselves can be
Primary Site with Backup Site. This approach addresses the second disadvantage
Primary Copy Technique. This method attempts to distribute the load of lock
Choosing a New Coordinator Site in Case of Failure. Whenever a coordina-
If a backup site X is about to become the new primary site, X can choose the new
911
Distributed Databases
existed, or if both the primary and the backup sites are down, a process called
7.2 Distributed Concurrency Control Based on Voting
The voting method is considered a truly distributed concurrency control method,
7.3 Distributed Recovery
■ The message was not delivered to Y because of communication failure.
■ Site Y is down and could not respond.
■ Site Y is running and sent a response, but the response was not delivered.
Without additional information or the sending of additional messages, it is difficult
Another problem with distributed recovery is distributed commit. When a transac-
912
Distributed Databases
have recorded the local effects of the transactions permanently in the local site log
8 Distributed Catalog Management
Three popular management schemes for distributed catalogs are centralized cata-
Centralized Catalogs. In this scheme, the entire catalog is stored in one single
Fully Replicated Catalogs. In this scheme, identical copies of the complete cata-
Partially Replicated Catalogs. The centralized and fully replicated schemes
913
Distributed Databases
9 Current Trends in Distributed Databases
9.1 Cloud Computing
Performance costs associated with partial failures and global synchronization were
The metadata of a distributed database system should be decoupled from its actual
10“The Claremont Report on Database Research” is available at http://db.cs.berkeley.edu/claremont/
914
Distributed Databases
cloud services available today, including application services (salesforce.com), stor-
9.2 Peer-to-Peer Database Systems
10 Distributed Databases in Oracle13
Oracle provides support for homogeneous, heterogeneous, and client server archi-
12A semantic path describes the higher-level relationship between two domains that are dissimilar but
915
Distributed Databases
distinguish between local and remote objects semantically. Using synonyms, this
In a heterogeneous architecture, at least one of the databases in the network is a
In the client-server architecture, the Oracle database system is divided into two to refer to tables uniquely.
Oracle uses a two-phase commit protocol to deal with concurrent distributed trans-
Oracle’s distributed database architecture is shown in Figure 12. A node in a distrib-
Communication in such a distributed heterogeneous environment is facilitated
916
Distributed Databases
Server
DEPT Table
Application
HQ
Connect to . . .
Oracle
EMP Table
Sales
Transaction
.
.
.
Network
INSERT INTO EMP@SALES . . . ;
DELETE FROM DEPT . . . ;
SELECT . . .
COMMIT;
Server
Oracle
Database Link
Figure 12
Source: From Oracle (2008). Copyright ©
is responsible for establishing and managing connections between a client applica-
Oracle supports database links that define a one-way communication path from
CREATE DATABASE LINK sales.us.americas;
917
Distributed Databases
establishes a connection to the sales database in Figure 12 under the network
Data in an Oracle DDBS can be replicated using snapshots or replicated master
■ Basic replication. Replicas of tables are managed for read-only access. For
■ Advanced (symmetric) replication. This extends beyond basic replication
CREATE SNAPSHOT SALES_ORDERS AS
Oracle groups snapshots into refresh groups. By specifying a refresh interval, the
Heterogeneous Databases in Oracle. In a heterogeneous DDBS, at least one
■ Distributed transactions. Under the two-phase commit mechanism, trans-
■ Transparent SQL access. SQL statements issued by an application are trans-
■ Pass-through SQL and stored procedures. An application can directly
■ Global query optimization. Cardinality information, indexes, and so on at
■ Procedural access. Procedural systems like messaging or queuing systems
918
Distributed Databases
In addition to the above, data dictionary references are translated to make the non-
From a security perspective, Oracle recommends that if a query originates at site A
10.1 Directory Services
Oracle supports LDAP Version 3 and online directories through Oracle Internet
Figure 13 illustrates the architecture of the Oracle Internet Directory. The main
■ Oracle directory server. Handles client requests and updates for informa-
■ Oracle directory replication server. Stores a copy of the LDAP data from
■ Directory administrator: Supports both GUI-based and command line-
11 Summary
919
Distributed Databases
Oracle
Server
Oracle Net
Oracle
Oracle
LDAP Clients
Directory
Figure 13
Source: From Oracle (2005). Copyright ©
distribution transparency and the related concepts of fragmentation transparency
920
Distributed Databases
control and recovery techniques used in DDBMSs, and then reviewed some of the
We described some of the facilities in Oracle to support distributed databases. We
Review Questions
bases?
2. What additional functions does a DDBMS have over a centralized DBMS?
3. Discuss what is meant by the following terms: degree of homogeneity of a
4. Discuss the architecture of a DDBMS. Within the context of a centralized
5. What are the main software modules of a DDBMS? Discuss the main func-
6. Compare the two-tier and three-tier client-server architectures.
7. What is a fragment of a relation? What are the main types of fragments? Why
8. Why is data replication useful in DDBMSs? What typical units of data are
9. What is meant by data allocation in distributed database design? What typi-
10. How is a horizontal partitioning of a relation specified? How can a relation
11. How is a vertical partitioning of a relation specified? How can a relation be
12. Discuss the naming problem in distributed databases.
13. What are the different stages of processing a query in a DDBMS?
14. Discuss the different techniques for executing an equijoin of two files located
921
15. Discuss the semijoin method for executing an equijoin of two files located at
16. Discuss the factors that affect query decomposition. How are guard condi-
17. How is the decomposition of an update request different from the decompo-
18. List the support offered by operating systems to a DDBMS and also their
19. Discuss the factors that do not appear in centralized systems that affect con-
20. Discuss the two-phase commit protocol used for transaction management in
21. Compare the primary site method with the primary copy method for dis-
22. When are voting and elections used in distributed databases?
23. Discuss catalog management in distributed databases.
24. What are the main challenges facing a traditional DDBMS in the context of
25. Discuss briefly the support offered by Oracle for homogeneous, heteroge-
26. Discuss briefly online directories, their management, and their role in dis-
Exercises
ments at sites 2 and 3 are as shown in Figure 9 and the fragments at site 1 are
a. For each employee in department 5, retrieve the employee name and the
b. Print the names of all employees who work in department 5 but who
Distributed Databases
922
28. Consider the following relations:
BOOKS(Book#, Primary_author, Topic, Total_stock, $price)
Total_stock is the total number of books in stock and Inventory_value is the
a. Give an example of two simple predicates that would be meaningful for
b. How would a derived horizontal partitioning of STOCK be defined based
c. Show predicates by which BOOKS may be horizontally partitioned by
d. Show how the STOCK may be further partitioned from the partitions in
29. Consider a distributed database for a bookstore chain called National Books
B1: BOOK1: $price up to $20
Similarly, BOOK_STORES are divided by ZIP Codes into:
S1: EAST: Zip up to 35000
Assume that STOCK is a derived fragment based on BOOKSTORE only.
a. Consider the query:
SELECT Book#, Total_stock
WHERE Ssn = NEW.Supervisor_ssn ) )
Figure 2
933
Enhanced Data Models for Advanced Applications
The CREATE TRIGGER statement specifies a trigger (or active rule) name—
The ON clause specifies the relation on which the rule is specified—EMPLOYEE for
The optional WHEN clause is used to specify any conditions that need to be checked
The four triggers (active rules) R1, R2, R3, and R4 illustrate a number of features of
Thus, rule R1 is triggered after an INSERT operation is applied to the EMPLOYEE
Rule R2 is similar to R1, but it is triggered by an UPDATE operation that updates the
3As we will see, it is also possible to specify BEFORE instead of AFTER, which indicates that the rule is
934
Enhanced Data Models for Advanced Applications
::= CREATE TRIGGER
( AFTER I BEFORE ) ON
[ FOR EACH ROW ] ] ;
::= {OR } ::= INSERT I DELETE I UPDATE [ OF { , } ] ::=
Figure 3
It is important to note the effect of the optional FOR EACH ROW clause, which sig-
UPDATE EMPLOYEE
Because the above statement could update multiple records, a rule using row-level
As a second example, suppose we want to check whenever an employee’s salary is
Figure 3 shows the syntax for specifying some of the main options available in
6Assuming that an appropriate external procedure has been declared. This is a feature that is available in
935
1.2 Design and Implementation Issues
The previous section gave an overview of some of the main concepts for specifying
The second issue concerns whether the triggered action should be executed before,
A related issue is whether the action being executed should be considered as a separate
Let us assume that the triggering event occurs as part of a transaction execution. We
1. Immediate consideration. The condition is evaluated as part of the same
2. Deferred consideration. The condition is evaluated at the end of the trans-
Enhanced Data Models for Advanced Applications
936
Enhanced Data Models for Advanced Applications
3. Detached consideration. The condition is evaluated as a separate transac-
The next set of options concerns the relationship between evaluating the rule condi-
The Oracle system (see Section 1.1) uses the immediate consideration model, but it
Another issue concerning active database rules is the distinction between row-level
One of the difficulties that may have limited the widespread use of active rules, in
R1: CREATE TRIGGER T1
UPDATE TABLE2
R2: CREATE TRIGGER T2
INSERT INTO TABLE1 VALUES ( … );
Figure 4
7STARBURST also allows the user to start rule consideration explicitly via a PROCESS RULES com-
937
Enhanced Data Models for Advanced Applications
problem briefly, consider the rules in Figure 4. Here, rule R1 is triggered by an
If active rules are to reach their potential, it is necessary to develop tools for the
1.3 Examples of Statement-Level Active Rules
We now give some examples to illustrate how rules can be specified in the STAR-
The three active rules R1S, R2S, and R3S in Figure 5 correspond to the first three
In STARBURST, the basic events that can be specified for triggering the rules are the
In statement-level semantics, the rule designer can only refer to the transition tables
8Note that the WHEN keyword specifies events in STARBURST but is used to specify the rule condition
938
Enhanced Data Models for Advanced Applications
R1S: CREATE RULE Total_sal1 ON EMPLOYEE
SET D.Total_sal = D.Total_sal +
WHERE D.Dno IN ( SELECT Dno FROM INSERTED );
R2S: CREATE RULE Total_sal2 ON EMPLOYEE
OR EXISTS ( SELECT * FROM OLD-UPDATED WHERE Dno IS NOT NULL )
SET D.Total_sal = D.Total_sal +
WHERE D.Dno = N.Dno ) –
WHERE D.Dno = O.Dno )
D.Dno IN ( SELECT Dno FROM OLD-UPDATED );
R3S: CREATE RULE Total_sal3 ON EMPLOYEE
SET D.Total_sal = D.Total_sal +
WHERE D.Dno = N.Dno )
( SELECT SUM (O.Salary) FROM OLD-UPDATED AS O
WHERE D.Dno IN ( SELECT Dno FROM OLD-UPDATED );
Figure 5
inserted in a single insert statement, we have to check if at least one of the newly
EXISTS (SELECT * FROM INSERTED WHERE Dno IS NOT NULL )
is checked, and if it evaluates to true, then the action is executed. The action updates
939
Enhanced Data Models for Advanced Applications
department, we use the SUM aggregate function to ensure that all their salaries are
Rule R2S is similar to R1S, but is triggered by an UPDATE operation that updates the
In our example, it is more complex to write the statement-level rules than the row-
The execution model for active rules in STARBURST uses deferred consideration.
1.4 Potential Applications for Active Databases
9As in the Oracle examples, rules R1S and R2S can be written without a condition. However, it may be
940
Enhanced Data Models for Advanced Applications
Active rules can also be used to enforce integrity constraints by specifying the types
Other applications include the automatic maintenance of derived data, such as the
1.5 Triggers in SQL-99
2 Temporal Database Concepts
941
Enhanced Data Models for Advanced Applications
developed since the early days of database usage. However, in creating these applica-
In this section, we will introduce some of the concepts that have been developed to
T1: CREATE TRIGGER Total_sal1
T2: CREATE TRIGGER Total_sal2
EXISTS ( SELECT * FROM O WHERE O.Dno IS NOT NULL )
Figure 6
942
Enhanced Data Models for Advanced Applications
information, and some of the different dimensions of time that may be needed.
2.1 Time Representation, Calendars,
For temporal databases, time is considered to be an ordered sequence of points in
Because there is no known beginning or ending of time, one needs a reference point
In SQL2, the temporal data types include DATE (specifying Year, Month, and Day as
11Unfortunately, the terminology has not been used consistently. For example, the term interval is often
943
Enhanced Data Models for Advanced Applications
Event Information versus Duration (or State) Information. A temporal data-
For example, an employee may have worked in a company from August 15, 2003
A time period is represented by its start and end time points [START-TIME, END-
Valid Time and Transaction Time Dimensions. Given a particular event or fact
However, a different interpretation can be used, where the associated time refers to
Other interpretations can also be intended, but these are considered to be the most
12This is the same as an anchored duration. It has also been frequently called a time interval, but to avoid
944
Enhanced Data Models for Advanced Applications
(a)
Name
EMP_VT
Salary DnoSsn Supervisor_ssn Vst Vet
Name Salary Supervisor_ssnSsn Tst Tet
(b)
(c)
Dname
DEPT_VT
EMP_TT
Dname Total_sal Manager_ssnDno
Dno
Tst Tet
DEPT_TT
Total_salDno Manager_ssn Vst Vet
Name Salary Supervisor_ssnSsn Dno Tst Tet
EMP_BT
Dname Total_sal Manager_ssnDno Tst Tet
DEPT_BT
Vst Vet
Vst Vet
Figure 7
other interpretations are intended for time, the user can define the semantics and
The next section shows how these concepts can be incorporated into relational
2.2 Incorporating Time in Relational Databases
Valid Time Relations. Let us now see how the different types of temporal data-
Consider how the EMP_VT relation differs from the nontemporal EMPLOYEE rela-
15A nontemporal relation is also called a snapshot relation because it shows only the current snapshot
945
Enhanced Data Models for Advanced Applications
Name
Smith 123456789 25000 5 333445555 2002-06-15 2003-05-31
Smith 123456789 30000 5 333445555 2003-06-01 Now
333445555 25000 4 999887777 1999-08-20 2001-01-31
333445555 30000 5 999887777 2001-02-01 2002-03-31
333445555 40000 5 888665555 2002-04-01 Now
222447777 28000 4 999887777 2001-05-01 2002-08-10
666884444 38000 5 333445555 2003-08-01 Now
Wong
Wong
Wong
Brown
Narayan
. . .
. . .
EMP_VT
Ssn Salary Dno Supervisor_ssn Vst Vet
Dname
Research
Research
DEPT_VT
5 888665555 2002-03-312001-09-20
333445555 2002-04-015 Now
Dno Manager_ssn Vst Vet
Figure 8
information that is valid (in the real world) only during the time period [V.Vst, V.Vet],
Figure 8 shows a few tuple versions in the valid-time relations EMP_VT and
946
Enhanced Data Models for Advanced Applications
It is important to note that in a valid time relation, the user must generally provide
The action that corresponds to deleting an employee in a nontemporal database
The operation to insert a new employee would correspond to creating the first tuple
Notice that in a valid time relation, the nontemporal key, such as Ssn in EMPLOYEE,
Valid time relations basically keep track of the history of changes as they become
Transaction Time Relations. In a transaction time database, whenever a change
16A combination of the nontemporal key and the valid end time attribute Vet could also be used.
947
Enhanced Data Models for Advanced Applications
Figure 1 into a transaction time database, then the two relations EMPLOYEE and
In EMP_TT, each tuple V represents a version of an employee’s information that was
Bitemporal Relations. Some applications require both valid time and transac-
Now consider how an update operation would be implemented on a bitemporal
17The uc variable in transaction time relations corresponds to the now variable in valid time relations. The
948
Enhanced Data Models for Advanced Applications
Name
Smith 123456789 25000 5 333445555 2002-06-15
Smith 123456789 25000 5 333445555 2002-06-15
123456789 30000 5 333445555 2003-06-01
333445555 25000 4 999887777 1999-08-20
333445555 25000 4 999887777 1999-08-20
333445555 30000 5 999887777 2001-02-01
333445555 30000 5
5
4
4
5
999887777
888667777
999887777
999887777
333445555
2001-02-01
2002-04-01
2001-05-01
2001-05-01
2003-08-01
2002-06-08, 13:05:58
2003-06-04, 08:56:12
2003-06-04, 08:56:12
1999-08-20, 11:18:23
2001-01-07, 14:33:02
2001-01-07, 14:33:02
2002-03-28, 09:23:57
2002-03-28, 09:23:57
2001-04-27, 16:22:05
2002-08-12, 10:11:07
2003-07-28, 09:25:37
2003-06-04,08:56:12
uc
uc
2001-01-07,14:33:02
uc
2002-03-28,09:23:57
uc
uc
2002-08-12,10:11:07
uc
uc
Now
2003-05-31
Now
Now
2001-01-31
Now
2002-03-31
Now
Now
2002-08-10
Now
Smith
Wong
Wong
Wong
Wong
Wong 333445555
Brown 222447777
Brown 222447777
Narayan
. . .
40000
28000
28000
38000666884444
EMP_BT
Ssn Salary Dno Supervisor_ssn Vst Vet Tst Tet
Dname
Research
Research
DEPT_VT
5 888665555 Now2001-09-20
888665555 2001-09-205 1997-03-31
Dno Manager_ssn Vst Vet
2001-09-15,14:52:12
2002-03-28,09:23:57
Tst
2001-03-28,09:23:57
uc
Research 333445555 2002-04-015 Now 2002-03-28,09:23:57 uc
Tet
Figure 9
time point before VT in the given valid time granularity and that transaction T has a
1. Make a copy V2 of the current version V; set V2.Vet to VT−, V2.Tst to TS(T),
2. Make a copy V3 of the current version V; set V3.Vst to VT, V3.Vet to now,
3. Set V.Tet to TS(T) since the current version is no longer representing correct
As an illustration, consider the first three tuples V1, V2, and V3 in EMP_BT in Figure
949
Enhanced Data Models for Advanced Applications
the updating transaction, ‘2003-06-04,08:56:12’. Note that this is a retroactive
Similarly, when Wong’s salary and department are updated (at the same time) to
Next, let us illustrate how a delete operation would be implemented on a bitempo-
Implementation Considerations. There are various options for storing the
Another option that is available is to vertically partition the attributes of the tempo-
It is important to note that bitemporal databases allow a complete record of
950
Enhanced Data Models for Advanced Applications
base will still be available as a previous database state for querying purposes. A data-
2.3 Incorporating Time in Object-Oriented Databases
The previous section discussed the tuple versioning approach to implementing
In attribute versioning, a single complex object is used to store all the temporal, ordered by valid start time.
Whenever an attribute is changed in this model, the current attribute version is
For bitemporal databases, each attribute version would have a tuple with five com-
The object lifespan would also include both valid and transaction time dimensions.
951
Enhanced Data Models for Advanced Applications
class TEMPORAL_SALARY
attribute Date Valid_end_time;
};
class TEMPORAL_DEPT
attribute Date Valid_end_time;
};
class TEMPORAL_SUPERVISOR
attribute Date Valid_end_time;
};
class TEMPORAL_LIFESPAN
attribute Date Valid end time;
class EMPLOYEE_VT lifespan;
attribute string Name; Sal_history; Dept_history; Supervisor_history;
};
Figure 10
2.4 Temporal Querying Constructs
So far, we have discussed how data models may be extended with temporal con-
In nontemporal relational databases, the typical selection conditions involve attrib-
952
Enhanced Data Models for Advanced Applications
current tuples. Following that, the attributes of interest to the query are specified by
((Salary > 30000) AND (Dno = 5))
The projected attribute would be Name. In a temporal database, the conditions may
Some of the more common operations used in queries are as follows:
[T.Vst, T.Vet] INCLUDES [T1, T2] Equivalent to T1 ≥ T.Vst AND T2 ≤ T.Vet
22
[T.Vst, T.Vet] BEFORE [T1, T2] Equivalent to T1 ≥ T.Vet
23
[T.Vst, T.Vet] MEETS_AFTER [T1, T2] Equivalent to T2 + 1 = T.Vst
Additionally, operations are needed to manipulate time periods, such as computing
■ [T1, T2] intersection [T3, T4] is empty.
■ T3 is not the time point following T2 in the given granularity.
■ T1 is not the time point following T4 in the given granularity.
The latter conditions are necessary to ensure unique representations of temporal
21A complete set of operations, known as Allen’s algebra (Allen, 1983), has been defined for compar-
953
Enhanced Data Models for Advanced Applications
into a single time period [T1, T4]. This is called coalescing of time periods.
To illustrate how pure time conditions can be used, suppose a user wants to select all
[T.Vst, T.Vet] OVERLAPS [2002-01-01, 2002-12-31]
Typically, most temporal selections are applied to the valid time dimension. For a
In addition to pure time conditions, other selections involve attribute and time
[T.Vst, T.Vet]OVERLAPS [2002-01-01, 2002-12-31] AND (T.Dno = 5)
Finally, we give a brief overview of the TSQL2 query language, which extends SQL
■ (valid time relation with valid time
■ (valid time relation with valid time
■ AND TRANSACTION (bitemporal rela-
■ AND TRANSACTION (bitemporal rela-
The keywords STATE and EVENT are used to specify whether a time period or time
954
Enhanced Data Models for Advanced Applications
2.5 Time Series Data
As another example, consider the daily sales dollar amount at each store of a chain
Because of the specialized nature of time series data and the lack of support for it in
More recently, some commercial DBMS packages are offering time series exten-
3 Spatial Database Concepts24
3.1 Introduction to Spatial Databases
24The contribution of Pranesh Parimala Ranganathan to this section is appreciated.
955
Enhanced Data Models for Advanced Applications
Table 1 Common Types of Analysis for Spatial Data
Analysis Type Type of Operations and Measurements
Measurements Distance, perimeter, shape, adjacency, and direction
Spatial analysis/statistics Pattern, autocorrelation, and indexes of similarity and topology using
Flow analysis Connectivity and shortest path
Location analysis Analysis of points and lines within a polygon
Terrain analysis Slope/aspect, catchment area, drainage network
Search Thematic search, search by region
Geographical Information Systems (GIS), and they are used in areas such as envi-
A spatial database is optimized to store and query data related to objects in space, position. A traditional B+-tree index based on
Table 1 shows the common analytical operations involved in processing geographic
25List of GIS analysis operations as proposed in Albrecht (1996).
956
Enhanced Data Models for Advanced Applications
erties of single objects (such as the area, the relative size of an object’s parts, com-
There are also topological relationships among spatial objects. These are often used
3.2 Spatial Data Types and Models
■ Map Data26 includes various geographic or spatial features of objects in a
26These types of geographic data are based on ESRI’s guide to GIS. See
957
Enhanced Data Models for Advanced Applications
vehicles and other moving objects can be represented by a sequence of point
■ Attribute data is the descriptive data that GIS systems associate with map
■ Image data includes data such as satellite images and aerial photographs,
Models of spatial information are sometimes grouped into two broad categories:
3.3 Spatial Operators
■ Topological operators. Topological properties are invariant when topologi-
958
Enhanced Data Models for Advanced Applications
■ Projective operators. Projective operators, such as convex hull, are used to
■ Metric operators. Metric operators provide a more specific description of
Dynamic Spatial Operators. The operations performed by the operators men-
Spatial Queries. Spatial queries are requests for spatial data that require the use
■ Range query. Finds the objects of a particular type that are within a given
■ Nearest neighbor query. Finds an object of a particular type that is closest to
■ Spatial joins or overlays. Typically joins the objects of two types based on
3.4 Spatial Data Indexing
to pages of secondary memory), so that objects in a particular spatial region can be
959
Enhanced Data Models for Advanced Applications
1. Specialized indexing structures that allow efficient search for data objects
2. Instead of creating brand new indexing structures, the two-dimensional
We give an overview of some of the spatial indexing techniques next.
Grid Files. Grid files are used for indexing of data on multiple attributes. They can
R-Trees. The R-tree is a height-balanced tree, which is an extension of the B+-tree
1. The structure of each index entry (or index record) in a leaf node is (I,
2. Every node except the root node must be at least half full. Thus, a leaf node
31A Prolog system typically has a number of different equality predicates that have different interpreta-
970
Enhanced Data Models for Advanced Applications
A query typically involves a predicate symbol with some variable arguments, and its
5.3 Datalog Notation
A number of built-in predicates are included in Datalog, which can also be used to
Figure 15
tions whose contents can be inferred by the inference engine. Figure 15 shows a
A program or a rule is said to be safe if it generates a finite set of facts. The general
BIG_SALARY(Y ) :– Y>60000
Here, we can get an infinite result if Y ranges over all possible integers. But suppose
BIG_SALARY(Y ) :– EMPLOYEE(X), Salary(X, Y ), Y>60000
In the second rule, the result is not infinite, since the values that Y can be bound to
BIG_SALARY(Y ) :– Y>60000, EMPLOYEE(X ), Salary(X, Y )
In this case, the rule is still theoretically safe. However, in Prolog or any other system
HAS_SOMETHING(X, Y ) :– EMPLOYEE(X )
977
Enhanced Data Models for Advanced Applications
REL_ONE(A, B, C ).
SELECT_ONE_A_EQ_C(X, Y, Z ) :– REL_ONE(C, Y, Z ).
e. On 2004-04-28,12:54:02, the database was changed to indicate that Wong
f. On 2004-05-05,13:07:33, the database was changed to indicate the rehir-
40. Show how the updates given in Exercise 39, if applied in sequence, would
41. Add the following facts to the sample database in Figure 11:
SUPERVISE(ahmad, bob), SUPERVISE(franklin, gwen).
First modify the supervisory tree in Figure 11(b) to reflect this change. Then
985
Enhanced Data Models for Advanced Applications
42. Consider the following set of facts for the relation PARENT(X, Y), where Y is
PARENT(a, aa), PARENT(a, ab), PARENT(aa, aaa), PARENT(aa, aab),
Consider the rules
r1: ANCESTOR(X, Y) :– PARENT(X, Y)
which define ancestor Y of X as above.
a. Show how to solve the Datalog query
ANCESTOR(aa, X)?
and show your work at each step.
b. Show the same query by computing only the changes in the ancestor rela-
[This question is derived from Bancilhon and Ramakrishnan (1986).]
43. Consider a deductive database with the following rules:
ANCESTOR(X, Y) :– FATHER(X, Y)
Notice that FATHER(X, Y) means that Y is the father of X; ANCESTOR(X, Y)
Consider the following fact base:
FATHER(Harry, Issac), FATHER(Issac, John), FATHER(John, Kurt).
a. Construct a model-theoretic interpretation of the above rules using the
b. Consider that a database contains the above relations FATHER(X, Y ),
c. Show a complete Datalog program with fact-based and rule-based literals
44. Consider the following rules:
REACHABLE(X, Y) :– FLIGHT(X, Y)
where REACHABLE(X, Y) means that city Y can be reached from city X, and
986
Enhanced Data Models for Advanced Applications
a. Construct fact predicates that describe the following:
i. Los Angeles, New York, Chicago, Atlanta, Frankfurt, Paris, Singapore,
ii. The following flights exist: LA to NY, NY to Atlanta, Atlanta to
b. Is the given data cyclic? If so, in what sense?
c. Construct a model-theoretic interpretation (that is, an interpretation
d. Consider the query
REACHABLE(Atlanta, Sydney)?
How will this query be executed? List the series of steps it will go through.
e. Consider the following rule-defined predicates:
ROUND-TRIP-REACHABLE(X, Y) :–
DURATION(X, Y, Z)
Draw a predicate dependency graph for the above predicates. (Note:
f. Consider the following query: What cities are reachable in 12 hours from
g. Consider the predicate population(X, Y), where Y is the population of
Selected Bibliography
STARBURST rules are described in Widom and Finkelstein (1990). Early work on
987
Enhanced Data Models for Advanced Applications
and Chakravarthy (1990). A glossary for temporal databases is given in Jensen et al.
Temporal normalization is defined in Navathe and Ahmed (1989). Paton (1999)
The book by Shekhar and Chawla (2003) consists of all aspects of spatial databases
Multimedia database modeling has a vast amount of literature—it is difficult to
The early developments of the logic and database approach are surveyed by Gallaire
988
while Levesque (1984) provides a discussion of incomplete knowledge in light of
Aho and Ullman (1979) provide an early algorithm for dealing with recursive
The original paper on magic sets is by Bancilhon et al. (1986). Beeri and
Chang (1981) describes an early system for combining deductive rules with rela-
Zaniolo (1990) reviews the theoretical background and the practical importance of
Enhanced Data Models for Advanced Applications
989
DEPT_LOCATIONS
Dnumber
Houston
Stafford
Bellaire
Sugarland
Dlocation
DEPARTMENT
Dname
Research
Administration
Headquarters 1
5
4
888665555
333445555
987654321
1981-06-19
1988-05-22
1995-01-01
Dnumber Mgr_ssn Mgr_start_date
WORKS_ON
Essn
123456789
123456789
666884444
453453453
453453453
333445555
333445555
333445555
333445555
999887777
999887777
987987987
987987987
987654321
987654321
888665555
3
1
2
2
1
2
30
30
30
10
10
3
10
20
20
20
40.0
32.5
7.5
10.0
10.0
10.0
10.0
20.0
20.0
30.0
5.0
10.0
35.0
20.0
15.0
NULL
Pno Hours
PROJECT
Pname
ProductX
ProductY
ProductZ
Computerization
Reorganization
Newbenefits
3
1
2
30
10
20
5
5
5
4
4
1
Houston
Bellaire
Sugarland
Stafford
Stafford
Houston
Pnumber Plocation Dnum
DEPENDENT
333445555
333445555
333445555
987654321
123456789
123456789
123456789
Joy
Alice F
M
F
M
M
F
F
1986-04-05
1983-10-25
1958-05-03
1942-02-28
1988-01-04
1988-12-30
1967-05-05
Theodore
Alice
Elizabeth
Abner
Michael
Spouse
Daughter
Son
Daughter
Spouse
Spouse
Son
Dependent_name Sex Bdate Relationship
EMPLOYEE
Fname
John
Franklin
Jennifer
Alicia
Ramesh
Joyce
James
Ahmad
Narayan
English
Borg
Jabbar
666884444
453453453
888665555
987987987
F
F
M
M
M
M
M
F
4
4
5
5
4
1
5
5
25000
43000
30000
40000
25000
55000
38000
25000
987654321
888665555
333445555
888665555
987654321
NULL
333445555
333445555
Zelaya
Wallace
Smith
Wong
3321 Castle, Spring, TX
291 Berry, Bellaire, TX
731 Fondren, Houston, TX
638 Voss, Houston, TX
1968-01-19
1941-06-20
1965-01-09
1955-12-08
1969-03-29
1937-11-10
1962-09-15
1972-07-31
980 Dallas, Houston, TX
450 Stone, Houston, TX
975 Fire Oak, Humble, TX
5631 Rice, Houston, TX
999887777
987654321
123456789
333445555
Minit Lname Ssn Bdate Address Sex DnoSalary Super_ssn
B
T
J
S
K
A
V
E
Houston
1
4
5
5
Essn
5
Figure A.1
990
Enhanced Data Models for Advanced Applications
Project
change_project
RESEARCH_
Course
assign_to_course
TEACHING_
Degree_program
change_degree_program
GRADUATE_
Class
change_classification
UNDERGRADUATE_
Position
hire_staff
STAFF
Rank
promote
FACULTY
Percent_time
hire_student
STUDENT_ASSISTANT
Year
DEGREE
. . .
Salary
hire_emp
EMPLOYEE
new_alumnus
. . .
ALUMNUS
Major_dept
change_major
STUDENT
Name
age
PERSON
Figure A.2
991
Introduction to Information
Information retrieval deals mainly with unstructureddata, and the techniques for indexing, searching, and
This chapter is organized as follows. In Section 1 we introduce information retrieval
1This chapter is coauthored with Saurav Sahay of the Georgia Institute of Technology.
From Chapter 27 of Fundamentals of Database Systems, Sixth Edition. Ramez Elmasri and
992
Introduction to Information Retrieval and Web Search
1 Information Retrieval (IR) Concepts
1.1 Introduction to Information Retrieval
HOUSES(Lot#, Address, Square_footage, Listed_price)
This is an example of structured data. We can compare this relation with home-
With the advent of the World Wide Web (or Web, for short), the volume of unstruc-
993
Introduction to Information Retrieval and Web Search
Historically, information retrieval is “the discipline that deals with the structure,
RDBMS (relational database management system) vendors are providing modules
IR systems go beyond database systems in that they do not limit the user to a spe-
An IR system can be characterized at different levels: by types of users, types of data,
Types of Users. The user may be an expert user (for example, a curator or a
2See Salton’s 1968 book entitled Automatic Information Organization and Retrieval.
994
Introduction to Information Retrieval and Web Search
example, students trying to find information about a new topic, researchers try-
Types of Data. Search systems can be tailored to specific types of data. For
Types of Information Need. In the context of Web search, users’ information
Navigational search refers to finding a particular piece of information (such as
Levels of Scale. In the words of Nobel Laureate Herbert Simon,
What information consumes is rather obvious: it consumes the attention of its
This overabundance of information sources in effect creates a high noise-to-signal
3See Broder (2002) for details.
995
Introduction to Information Retrieval and Web Search
BitTorrent, which allows sharing of music in the form of audio files, as well as spe-
1.2 Databases and IR Systems: A Comparison
Whereas databases have fixed schemas defined in some data model such as the rela-
The result of a database query is an exact answer; if no matching records (tuples) are
Table 1 A Comparison of Databases and IR Systems
Databases
■ Structured data
network) model is predominant
correct)
IR Systems
(e.g., vector space model)
documents
and measures of effectiveness (may be
996
Introduction to Information Retrieval and Web Search
information most relevant to that query. Whereas database systems maintain a large
1.3 A Brief History of IR
The earlier experiments with document retrieval systems such as SMART7 in the
A search engine is a practical application of information retrieval to large-scale
5See Luhn (1957) “A statistical approach to mechanized encoding and searching of literary information.”
997
in search engine technology, where search engines are trying to discover different
While continuous progress is being made to tailor search results to the needs of an
1.4 Modes of Interaction in IR Systems
There are two main modes of interaction with IR systems—retrieval and brows-
Introduction to Information Retrieval and Web Search
9See www.ncbi.nlm.nih.gov/pubmed/.
998
Introduction to Information Retrieval and Web Search
Web search combines both aspects—browsing and retrieval—and is one of the
1.5 Generic IR Pipeline
In a statistical approach, documents are analyzed and broken down into chunks of
Semantic approaches to IR use knowledge-based techniques of retrieval that
Figure 1 shows the various stages involved in an IR processing system. The steps
Figure 2 shows a simplified IR processing pipeline. In order to perform retrieval on
999
Introduction to Information Retrieval and Web Search
Document 3
Document 1
Preprocessing
Modeling
Indexing
Stopword removal
Retrieval models
Inverted index construction
SEARCH INTENT
Need/Search
Query Formation
Query Processing
Searching
Relevance
Legend
next iteration
Choice of search strategy
Keywords, Boolean, phrase,
Conversion from humanly
Storing user’s
Metadata
Ranking results
External data
Document
Figure 1
taining these words—and the document properties, such as date of creation, author,
2 Retrieval Models
1000
Introduction to Information Retrieval and Web Search
Documents
EXTRACT
FEEDBACK
FETCH
PROCESS
Inverted Index
COMPARE
Documents
RANK
Two tickets tickled slightly angst-riden
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Document #4
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Document #3
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Document #2
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Document #1
Result #3
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Result #2
orifices. Two Jabberwockies sacrificed subways,
and two mosly bourgeois orifices towed Kermit.
Five very progressive fountains annoyingly
tickled the partly speedy dog, even though
two putrid sheep laughed almost noisily.
Result #1
Index
D1 1 1 0 1 1 0…
W1 1 1 1 0 0 1…
SEARCH INTENT
Figure 2
2.1 Boolean Model
Boolean retrieval models lack sophisticated ranking algorithms and are among the
1001
Introduction to Information Retrieval and Web Search
documents as well as other properties of documents, such as date of creation,
2.2 Vector Space Model
The similarity assessment function that compares two vectors is not inherent to the
In the vector model, the document term weight wij (for term i in document j) is repre-
In the formula given above, we use the following symbols:
■ dj is the document vector.
■ q is the query vector.
■ wij is the weight of term i in document j.
■ wiq is the weight of term i in query vector q.
■ |V| is the number of dimensions in the vector that is the total number of
TF-IDF uses the product of normalized frequency of a term i (TFij) in document Dj
cosine( , )
| |
d q
d q
w w
j
j
ij iqi
V
=
×
×
∑ ∑= =×w wiji
iqi
V2
2
| | | |
1002
Introduction to Information Retrieval and Web Search
document. The idea is that terms that capture the essence of a document occur fre-
IDF values can be easily computed for a fixed collection of documents. In case of
In these formulas, the meaning of the symbols is:
■ TFij is the normalized term frequency of term i in document Dj.
Note that if a term i occurs in all documents, then ni = N and hence IDFi = log (1)
Sometimes, the relevance of the document with respect to a query (rel(Dj,Q)) is
The normalization factor (similar to the denominator of the cosine formula) is
The Rocchio10 algorithm is a well-known relevance feedback algorithm based on
q q
d
de
r
ir
r r ir ir
= + −
| | | |
rel( , )D Q TF IDFj i Q ij i= ×∑ ∈
TF f f
IDF N n
ij ij ij
i i
=
= ( )
log /
10See Rocchio (1971).
1003
Introduction to Information Retrieval and Web Search
Here, Dr and Dir are relevant and nonrelevant document sets and α, β, and γ are
2.3 Probabilistic Model
In the probabilistic framework, the IR system has to decide whether the documents
Given the document representation D of a document, estimating the relevance R
P(R|D) = P(D|R) × P(R)/P(D)
A document D is classified as relevant if P(R|D) > P(NR|D). Discarding the constant
P(D|R) × P(R) > P(D|NR) × P(NR)
The likelihood ratio P(D|R)/P(D|NR) is used as a score to determine the likelihood
The term independence or Naïve Bayes assumption is used to estimate P(D|R) using
11For a description of the Cheshire II system, see Robertson (1997).
1004
Introduction to Information Retrieval and Web Search
With some reasonable assumptions and estimates about the probabilistic model
The Okapi weight for Document dj and query q is computed by the formula below.
■ ti is a term.
■ fij is the raw frequency count of term ti in document dj.
■ fiq is the raw frequency count of term ti in query q.
■ N is the total number of documents in the collection.
■ dfi is the number of documents that contain the term ti.
■ dlj is the document length (in bytes) of dj.
■ avdl is the average document length of the collection.
The Okapi relevance score of a document dj for a query q is given by the equation
2.4 Semantic Model
Semantic approaches include different levels of analysis, such as morphological,
okapi( , ) ln
.
)
N df
df
k f
k
j
i
ij=
+
( +10 5
1
1
1 −− + +
×
+
dl
avdl
k f
k fj
iq
iqt q
⎛
⎝
⎞
⎠
∈
( )2
2
1
,,d
∑
14City University of London Okapi System by Robertson, Walker, and Hancock-Beaulieu (1995).
1005
Introduction to Information Retrieval and Web Search
The development of a sophisticated semantic system requires complex knowledge
3 Types of Queries in IR Systems
3.1 Keyword Queries
3.2 Boolean Queries
15See Lenat (1995).
1006
Introduction to Information Retrieval and Web Search
of the search term. ‘–’ is equivalent to AND NOT and means to exclude the term; the
3.3 Phrase Queries
3.4 Proximity Queries
3.5 Wildcard Queries
1007
Introduction to Information Retrieval and Web Search
example, ‘data*’ would retrieve data, database, datapoint, dataset, and so on).
3.6 Natural Language Queries
4 Text Preprocessing
4.1 Stopword Removal
Removal of stopwords from a document must be performed before indexing.
17For details, see Croft et al. (2009), pages 75–90.
1008
Introduction to Information Retrieval and Web Search
percent or more. However, doing so could impact the recall if the stopword is an
4.2 Stemming
A stemming algorithm can be applied to reduce any word to its stem. In English, the
4.3 Utilizing a Thesaurus
UMLS20 is a large biomedical thesaurus of millions of concepts (called the
18See Porter (1980).
1009
Introduction to Information Retrieval and Web Search
Organ or
Function
Physiologic
Biologic
Pathologic
Organism
Cell
Molecular
Cell or
Dysfunction
Disease
Syndrome
Experimental
Mental or
Dysfunction
Neoplastic
Mental
Genetic
Figure 3
Source: UMLS Reference Manual, National Library of Medicine.
WordNet21 is a manually constructed thesaurus that groups words into strict syn-
WordNet is based on the idea of using a controlled vocabulary for indexing, thereby
4.4 Other Preprocessing Steps: Digits, Hyphens, Punctuation
Digits, dates, phone numbers, e-mail addresses, URLs, and other standard types of
21See Fellbaum (1998) for a detailed description of WordNet.
1010
Introduction to Information Retrieval and Web Search
metadata to improve precision and recall (see Section 6 for detailed definitions of
Hyphens and punctuation marks may be handled in different ways. Either the entire
Most information retrieval systems perform case-insensitive search, converting all
4.5 Information Extraction
5 Inverted Indexing
1011
Introduction to Information Retrieval and Web Search
Weights are assigned to document terms to represent an estimate of the usefulness
An inverted index of a document collection is a data structure that attaches distinct
One of the most popular weighting schemes is the TF-IDF (term frequency-inverse
Figure 4 shows an illustration of term-document-position vectors for the four illus-
The different steps involved in inverted index construction can be summarized as
1. Break the documents into vocabulary terms by tokenizing, cleansing,
2. Collect document statistics and store the statistics in a document lookup
3. Invert the document-term stream into a term-document stream along with
Searching for relevant documents from the inverted index, given a set of query
1. Vocabulary search. If the query comprises multiple terms, they are sepa-
1012
This example
Inverted index
Stock market
Stock market
ID
1.
2.
3.
4.
Term
example
inverted
index
market
Document: position
1:2, 1:5
1:8, 2:1
1:9, 2:2, 3:3
3:2, 3:13
Document 1
Document 2
Document 2
Introduction to Information Retrieval and Web Search
used to optimize the search process. Query terms may also be ordered in lex-
2. Document information retrieval. The document information for each term
3. Manipulation of retrieved information. The document information vector
6 Evaluation Measures
Without proper evaluation techniques, one cannot compare and measure the rele-
Figure 4
1013
Introduction to Information Retrieval and Web Search
Evaluation techniques of IR systems measure the topical relevance and user
In Web information retrieval, no binary classification decision is made on whether a
6.1 Recall and Precision
Relevant?
Yes No
Hits
TP
False
FP
Misses
FN
Correct
TN
Retrieved?
Yes
No
☺
☺
�
�
Figure 5
1014
Introduction to Information Retrieval and Web Search
Table 2 Precision and Recall for Ranked Retrieval
Doc. No. Rank Position i Relevant Precision(i) Recall(i)
10 1 Yes 1/1 = 100% 1/10 = 10%
17 5 No 3/5 = 60% 3/10 = 30%
215 7 Yes 4/7 = 57.1% 4/10 = 40%
The notation for Figure 5 is as follows:
■ TP: true positive
■ FP: false positive
■ FN: false negative
■ TN: true negative
The terms true positive, false positive, false negative, and true negative are generally
Recall = |Hits|/|Relevant|
Precision = |Hits|/|Retrieved|
Recall and precision can also be defined in a ranked retrieval setting. The Recall at
q (denoted by r(i)) (di
position i for query q) is the fraction of relevant documents from d1
q in the
q to di
be Si with cardinality | Si |. Let (|Dq| be the size of relevant documents for the query.
Recall r(i) = |Si |/|Dq|
The Precision at rank position i or document di
documents from d1
q in the result set that are relevant:
Precision p(i) = |Si |/i
Table 2 illustrates the p(i), r(i), and average precision (discussed in the next
1015
Introduction to Information Retrieval and Web Search
example, the number of relevant documents for some query = 10. The rank posi-
6.2 Average Precision
Consider the sample precision values of relevant documents in Table 2. The average
6.3 Recall/Precision Curve
6.4 F-Score
One of the properties of harmonic mean is that the harmonic mean of two numbers
F
p r
=
2
F
p r
+
P p i D
i
q
q
1016
Introduction to Information Retrieval and Web Search
7 Web Search and Analysis22
The emergence of the Web has brought millions of users to search for information,
There are other types of search engines besides the ones that regularly crawl the Web
Another source of searchable Web documents is digital libraries. Digital libraries
7.1 Web Analysis and Its Relationship
In addition to browsing and searching the Web, another important activity closely
22The contributions of Pranesh P. Ranganathan and Hari P. Kumar to this section is appreciated.
1017
Introduction to Information Retrieval and Web Search
new information of interest. Application of data analysis techniques for discovery
The goals of Web analysis are to improve and personalize search results relevance
■ Finding relevant information. People usually search for specific informa-
■ Personalization of the information. Different people have different content
■ Finding information of commercial value. This problem deals with finding
Based on the above goals, we can classify Web analysis into three categories: Web
1018
Introduction to Information Retrieval and Web Search
7.2 Searching the Web
Web pages, unlike standard text collections, contain connections to other Web pages
A hub is a Web page or a Website that links to a collection of prominent sites
7.3 Analyzing the Link Structure of Web Pages
1019
Introduction to Information Retrieval and Web Search
helps with navigation and makes it possible to compare/integrate Web page
The PageRank Ranking Algorithm. As discussed earlier, ranking algorithms are
The computation of page ranking follows an iterative approach. PageRank of a Web
Appendix: Overview of the QBE Language
EMPLOYEE(a)
(b)
(c)
(d)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
John B Smith _123456789 P._9/1/60 P._100 Main, Houston, TX _M _25000 _123456789 _3
EMPLOYEE
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
John B Smith P._9/1/60 P._100 Main, Houston, TX
EMPLOYEE
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
John B Smith P._X P._Y
EMPLOYEE
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
John B Smith P. P.
Figure 2
1Negation with the ¬ symbol is not allowed in a condition box.
1060
Appendix: Overview of the QBE Language
For example, the query Q0B: List the social security numbers of employees who work
Some complex conditions can be specified without a condition box. The rule is that
Now consider query Q0C: List the social security numbers of employees who work on
In general, once a query is specified, the resulting values are displayed in the template
A join operation is specified in QBE by using the same variable2 in the columns to
2A variable is called an example element in QBE manuals.
WORKS_ON
(a) Essn Pno Hours
P. > 20
WORKS_ON
(b) Essn Pno Hours
P. _PX _HX
_HX > 20 and (PX = 1 or PX = 2)
CONDITIONS
WORKS_ON
(c) Essn Pno Hours
P. > 201
Figure 3
1061
Appendix: Overview of the QBE Language
work for the ‘Research’ department can be specified as shown in Figure 5(a). Any
To join a table with itself, we specify different variables to represent the different ref-
2 Grouping, Aggregation, and Database
Next, consider the types of queries that require grouping or aggregate functions. A
WORKS_ON
P._ES 1
WORKS_ON
P._EX 1
_EX = _EY
CONDITIONS
Figure 4
3ALL in QBE is unrelated to the universal quantifier.
1062
Appendix: Overview of the QBE Language
Figure 6(a) shows query Q23, which counts the number of distinct salary values in
QBE has a negation symbol, ¬, which is used in a manner similar to the NOT EXISTS
Although the QBE language as originally proposed was shown to support
EMPLOYEE(a)
(b)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
Research
P. _FN _LN _Addr
_DX
_LN _Addr _DX
DEPARTMENT
Dname Dnumber Mgrssn Mgr_start_date
RESULT
P. _E1 _E2 _S1
RESULT
EMPLOYEE
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
_S1 _S2 _Xssn
Figure 5
1063
Appendix: Overview of the QBE Language
There are three QBE operators for modifying the database: I. for insert, D. for delete,
EMPLOYEE(a)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
P.CNT.
EMPLOYEE(b)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
P.CNT.ALL
EMPLOYEE(c)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
P.AVG.ALL P.G.P.CNT.ALL
PROJECT(d)
Pname Pnumber Plocation
_PXP.
Dnum
WORKS_ON
Essn Pno Hours
P.CNT.EX G._PX
CNT._EX > 2
CONDITIONS
Figure 6
EMPLOYEE
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
P. P. _SX
DEPENDENT
Essn Dependent_name Sex Bdate Relationship
Figure 7
1064
Appendix: Overview of the QBE Language
to increase the salary of ‘John Smith’ by 10 percent and also to reassign him to
QBE also has data definition capabilities. The tables of a database can be specified
QBE does not use the linear style of SQL; rather, it is a two-dimensional language
More recently, numerous other user-friendly interfaces have been developed for
EMPLOYEE(a)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
RichardI. MariniK M 37000 987654321 4653298653 30-Dec-52 98 Oak Forest, Katy, TX
EMPLOYEE(b)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
D. 653298653
EMPLOYEE(c)
Fname Minit Lname Ssn Bdate Address Sex Salary Super_ssn Dno
John Smith U._S*1.1 U.4
Figure 8
1065
This page intentionally left blank
Index
Index
/, 53, 56, 101, 280, 375-376, 423, 425, 431-434,
//, 437-438, 492, 501-502
}, 5, 140, 151, 161, 164, 178-179, 181-188, 209,
<>, 93, 117-118
!=, 93, 495
<=, 92-93, 101, 118, 495, 691, 718-719, 960, 970-971
!, 93, 495
&, 342, 425
<, 61, 63, 72-73, 77, 88-90, 92-93, 101, 109, 117-118,
140, 151, 153, 161, 179, 186, 270, 296-297,
337-338, 341, 346, 423-426, 429-433,
439-440, 444-445, 461, 491-496, 578-580,
608-609, 620-621, 640, 651, 653-655,
657-658, 660-662, 666-668, 696-697, 701,
718-719, 744, 826, 905, 923, 970-971, 978,
985
||, 87, 101, 858, 1002
==, 462-463, 495
>, 52, 61, 63, 77, 89-90, 93, 101, 109, 117-118,
::, 140, 935
+, 8, 17, 36, 40, 83, 88, 93, 101, 158, 352, 358-359,
/=, 971
>=, 93, 101, 118, 495, 691, 718-719, 970-971, 977
1
988-989
3
559-560, 565-569, 573-574, 582, 584-586,
4
A
834, 907-909
277, 365, 374, 411, 414, 929, 958
abstract classes, 266
34, 37-38, 40-42, 44-46, 48, 55-56, 105-106,
methods, 48, 329, 334, 348, 454, 476, 606, 673,
Access control, 836, 838, 840-843, 848, 851-856, 862,
database, 836, 838, 840-843, 848, 851-856, 862,
policies of, 841
338-340, 492-494, 498, 745, 757, 812,
action attribute, 493
algorithm, 728
adding, 15, 31, 33, 122, 137-139, 322, 324, 328,
security, 122, 844, 857, 951
85, 88, 92, 101, 105-106, 152, 177, 180,
953-955, 1047
110-111, 123-124, 139, 143-145, 151, 154,
Address space, 612-616
608, 611, 613-615, 617, 629, 637, 654,
base, 1024
617, 619, 654, 666-667
Addressing, 613-614, 853
adjusting, 714
856, 870, 873, 919, 950, 1047
285, 356, 520, 543, 955, 1037, 1042, 1062
Algebra, 55, 71, 75, 81, 82-83, 92-93, 97, 99, 148-200,
algebraic, 71, 686, 708, 713-714, 723-724, 728, 987
399, 415, 510, 538, 551, 555-558, 560-569,
merge, 610, 688-689, 694-695, 699, 702, 704,
representation of, 705
521, 550-586, 613-614, 619, 633-635, 646,
1067
831, 835, 864-865, 873, 901-902, 905,
algorithms:, 613
1016, 1019
577-579, 582-586, 654, 660, 665, 668,
statements, 39
36, 42-43, 46-47, 51-52, 57, 59, 63-67, 70,
Amazon, 3, 915
data, 189
complete, 986
Anchor text, 1019
115-147, 148-200, 201-215, 217-224,
AND function, 129, 406, 1008
48-49, 59-60, 62, 64-65, 69, 72, 75-76,
691-694, 708, 711-713, 720, 734-735,
API, 45, 413, 428, 456, 471, 476, 870
38-46, 48, 52, 64, 70-71, 75, 78, 82, 84, 109,
application layer, 892-893
476
45-48, 51-52, 70, 75, 82-83, 124, 133, 149,
applications of, 1, 342, 679, 745, 930-931, 940,
search trees, 679
497-499, 505, 588, 635, 870-871, 886-893,
client/server, 27-28, 42-46, 51-52, 458
Arguments, 10, 328, 338, 360, 365, 369, 379, 406,
array, 497-499, 503
Arithmetic, 62, 100-101, 134, 169, 438, 581, 583, 613,
expression, 438
Arithmetic operations, 134, 169, 970
list of, 101
414, 479, 492-493, 495-501, 503-505, 591,
accessing, 499-500, 505, 667, 676
503-504
code, 612, 623
623-624, 630, 635, 965
arrays, and, 505
ASCII, 612
350, 355, 381, 416, 486, 523, 842, 853, 965,
Assertion, 62, 70, 83, 92, 115, 131-132, 139
declaration, 940
Assignments, 178, 497-499, 984
389, 867, 887, 944, 961-962, 983, 988, 1025
sequence, 496
types of, 857, 874
103-104, 126, 146, 178-179, 185, 206, 208,
Attributes, 29, 57-61, 63-67, 69-70, 72-76, 79-80, 85,
of entities, 202, 205, 209-210, 214, 234, 261, 265,
Audio, 1, 591, 600, 623, 930, 963-964, 967, 982, 984,
compression, 964
digital signatures, 866
authorization, 17, 38, 83-84, 106, 135, 416, 460, 838,
Autocommit, 467
276, 284, 312, 333, 369, 406-407, 410, 485,
average access time, 610
B
noise, 966
388-389, 401, 403, 405, 407-408, 410-411,
Balanced tree, 657, 960
1068
266, 337, 361, 514-518, 520, 525, 549, 585,
identifying, 232
333, 378, 706, 729, 966, 989, 1004, 1020
231-233, 236, 241, 364, 390, 399, 401-402,
Binary search, 608, 610-611, 629, 632-633, 637-638,
Binary search algorithm, 645
full, 654
666, 669-671, 813-814, 864-865, 1041-1042
635, 636, 667-671, 864, 964, 1042
607-611, 614-615, 617, 619, 623-626,
Block transfer, 596, 599, 629, 631-632, 634,
Blocks, 117, 120, 332, 381, 588, 594-598, 600-611,
record blocking, 602
, 423-424
call, 482, 980
319, 354-355, 416, 424, 427, 455, 480,
Boolean, 87-88, 93, 116-117, 120, 150-152, 178-179,
false, 88, 116-117, 120, 152, 178-179, 386, 408,
true, 88, 93, 116-117, 120, 151-152, 178-179, 386,
Boolean condition, 93, 179, 408
do, 154, 184, 446, 483, 714, 894
brightness, 965
primary, 13
682-683, 695, 702, 959-960
644, 679, 688-690, 698, 715, 723, 726, 729,
Buffering, 14, 17-18, 45, 588, 598, 603, 610, 629-630,
cache, 812
Bug, 857
624-625, 631-632, 634, 640, 659, 670-671,
C
99-100, 107-109, 112, 116-117, 141-142,
C++, 8, 17, 36, 40, 83, 93, 265, 352, 358-359, 363,
C#, 262, 454, 892
495, 600
535, 543, 548, 552, 558, 573
167, 189, 212, 214, 702, 708-709, 711,
cartridges, 598
713
73-74, 87, 91, 95-96, 109, 128, 130, 134,
error, 319, 461, 624, 626, 628, 774
84-85, 138, 196, 237, 276-277, 340-343,
Cell, 80, 590, 667, 963-964, 1010, 1038
networks, 589
Certificate, 866-867, 874, 919
80, 82, 89, 109, 137, 139, 141, 194,
223-224, 243, 267, 313, 318-320, 339-340,
Channels, 628, 836, 839, 861-863, 873-874, 965
chapters, 19, 319, 329, 355-356, 987, 1050, 1052
data and, 88
493-495, 600-603, 612, 630, 634, 858-859,
formatting, 425, 601
Charts, 318-319, 995
131-132, 276-277, 402, 420, 465, 480-481,
Checkpoint, 815-816, 819-822, 825-827, 830, 832
723, 802-803, 960
282, 301-302, 315-317, 331-332, 338, 350,
ciphertext, 863-865
108-109, 112-113, 147, 196, 202, 210, 226,
block, 336, 632
414, 442, 444, 1010
269, 275, 320, 335-336, 342, 345, 347-348
269-271, 274, 281-282, 299-300, 336, 342,
arguments, 360, 365
342, 346, 349, 389, 401
422, 466, 476, 484, 1032
classes and, 269-270, 336, 394, 411, 415, 484, 862,
ClassNotFoundException, 477-478
458, 480-481, 491, 499-500, 505, 628, 887,
Client computer, 40, 491, 505
1069
Client-server architecture, 48, 480, 877, 892, 916,
Client/server interaction, 45
678, 718-719, 736-737, 961-962, 1046
207-209, 237-238, 243, 277-278, 319, 333,
described, 40, 191, 209, 237, 243, 482, 545, 549,
options for, 378, 464, 601
Code generator, 684-685
load factor, 635
color, 151, 208, 210, 281, 424, 496-497, 964-965
columns, 9, 26, 87-88, 117, 152, 169, 189, 280, 348,
indexing, 348, 668-669, 671, 674-676
138-139, 424, 455, 459-467, 470-471, 474,
command line, 476
139-141, 333, 424, 434, 455-456, 458-460,
atomic, 774
872
139-141, 455-456, 458-460, 462-464,
TYPE, 49-50, 140-141, 424, 434, 456, 459-460,
comment, 283, 561, 733, 964
785, 795-796, 811-812, 817-822, 824-825,
Commit point, 756, 758, 776, 811-812, 817-821, 830
124, 130, 141, 150-151, 161-163, 179, 186,
comparison of, 62, 996, 1029
151, 161, 179, 186, 604, 970, 1060
207-209, 212-213, 241, 283, 290-294, 311,
components:, 133, 359, 390, 405-406, 410, 716, 951,
graphical, 350
audio, 964
Computer, 1-2, 5-6, 10, 18, 25-26, 27-30, 38, 40-44,
Computer networks, 44, 877, 924
Computer software, 282
590, 593, 716, 995
Computing systems, 877
740, 747-749, 751-752, 755-756, 758-760,
deadlock, 755, 781, 787-791, 793-794, 805-807,
mutual exclusion, 781
Concurrency control, 11, 20, 24, 39-40, 45, 106, 359,
Condition, 49-50, 65, 73-74, 93, 95-97, 101-102, 104,
conditional, 93, 178, 440, 456, 458, 480, 482, 493,
relational, 93, 178
128-132, 136, 143, 149-150, 152, 162-164,
confidentiality, 837, 855, 860, 865, 871
477-479, 628, 879, 918-919
917, 920
381, 596, 687, 968, 970-971, 973-974, 978,
Constants, 93, 187, 194, 549, 585, 969-970, 974,
named, 194
392, 401-402, 405, 415
401-402, 412-413
overloaded, 373
967, 982-983, 988, 993-994, 997-998,
media, 967, 982, 997
514, 592, 851, 965, 970, 1035-1036, 1048
222, 273, 310-312, 338-340, 344, 411, 467,
execution, 39-40, 332, 359, 467, 740, 748-749,
Label, 836, 853-854, 862, 869-873, 876
Control system, 43
446, 538, 709, 713-714, 728, 1011, 1025
Copyright, 1, 548, 920
705, 715, 717, 719, 722, 724, 726, 728,
overhead, 24-25, 705
CPU, 43, 589-590, 598, 716, 727, 765, 769, 882,
secondary storage, 589-590, 716
create unique index, 672
251, 268, 300, 311, 348, 372, 376, 388-389,
forms, 268, 329, 500, 502, 583, 737
CROSS JOIN, 124, 158, 160, 163
204-205, 208, 210, 212, 236, 238, 242, 247,
Current position, 50
239-240, 243-244, 286, 303, 318-319, 323,
customers, 37, 196, 243, 311, 318, 745, 956-957
771, 790-791, 807, 980
D
105-108, 112, 133, 147, 148-149, 163, 173,
1070
705-707, 716-718, 725-727, 736-737,
Double, 87, 210, 220-221, 223-224, 226-227, 382,
hiding, 365, 879
362-364, 411, 496, 600, 602, 617, 653,
integrity, 18-19, 24, 56, 64, 66, 69-77, 79, 85, 89,
security threats, 836
65-66, 80, 87-88, 105, 133, 163, 189,
validation, 313-314, 484, 772, 780, 794, 797-798,
Data abstraction, 8-11, 21, 28, 268-269, 275
trends, 42, 878
50-52, 879
988, 1018, 1025, 1028, 1034-1036,
Data model, 7, 10, 19, 21, 28-30, 32-33, 47-48, 50-51,
Data processing, 330, 623, 649
189, 364-365, 472, 487, 522, 588, 612, 614,
data structures and, 17, 688, 988
user, 903-904
460, 479, 600, 612, 638, 641, 945, 979,
Character, 5, 57, 87-88, 429, 460, 612
Data types, 3, 10, 22-23, 28, 32, 38, 48, 83-84, 87-88,
Data warehouse, 1026, 1034-1037, 1041-1049
1041-1042, 1044-1049
102-114, 115-117, 124-126, 131-134,
441-453, 454-469, 471-473, 476-478,
Database:, 72, 92, 102, 258, 447, 1045, 1064
Database administrator, 13, 310, 839, 841-842, 853,
database administrators, 13, 733-734, 1048
105, 201-204, 222-223, 232, 234, 236, 264,
database management system (DBMS), 3
84-85, 91, 107-111, 114, 134, 137-139,
Database server, 40, 45-46, 311, 458, 471, 473,
Database systems, 1-2, 8, 10, 17, 19, 21-22, 26, 27,
connecting to, 500
database tables, 115, 140, 503, 846, 870
63, 66, 75-76, 82, 105-106, 131, 198, 216,
MySQL, 48, 106, 311, 491, 500, 746
476, 491, 500, 746, 836, 869, 871-873,
PostgreSQL, 48, 106, 311
105-106, 115, 131, 149, 312, 353, 400,
queries in, 66, 115, 404, 415, 901
359, 365, 380-381, 402, 413, 901, 954,
query results, 46, 50, 403-404, 471, 484, 892-893
493, 943, 951-952, 955-956, 982, 988,
security of, 838, 883
1008, 1045-1046
989
86-89, 92-93, 95, 100-101, 104, 110-111,
205-206, 211-213, 220-223, 227-228, 230,
Date:, 228, 346, 356, 364, 366
date data type, 88
1041, 1043
73-75, 81, 85, 91, 103, 106-107, 109,
Deadlock, 755, 781, 787-791, 793-794, 805-807, 820,
concurrency and, 925
Deadlocks, 755, 792, 796, 807, 910
459-460, 481
536-538, 541-543, 547-548, 552-553,
Decryption, 47, 864-866
134, 137-139, 146, 229, 264, 268, 372, 378,
tool, 859
134, 136-137, 139, 145, 235, 249, 251, 255,
delay, 596, 598, 625, 629, 631-632, 679, 796, 816,
queuing, 625
674, 932
Deletion, 35, 73, 83, 104, 107, 255, 338, 514, 520,
Denial of service, 856, 858, 1027
565-568, 573-574, 582-584, 586
secure, 853
41-43, 52, 56, 62, 64, 105, 109, 201-205,
of databases, 22, 246, 310, 334, 345, 348, 359,
1071
Design process, 202, 226, 234, 236, 259, 264,
desktop, 48, 282, 995, 1034-1035
312, 318-319, 329-331, 334, 343, 347-348,
services and, 882
739-740, 868
385-386, 388-389, 411, 414, 675, 893, 919,
Dictionary encoding, 675
866-868, 873-874, 957, 993-994, 1017, 1032
digital certificate, 866-867, 874
878, 904, 914, 956-957, 959, 965, 987, 1052
Directories, 631, 823-824, 919, 921-922, 1017, 1022
682-683, 812-813, 823-824, 829-830, 856,
Dirty data, 752
547, 588-635, 636-638, 640-641, 644-645,
Disk access, 40, 622-623, 717
635
755, 1054-1056
881-884, 886-888, 892, 894, 901, 905, 907,
Distributed processing, 877
distributed systems, 311, 878, 922, 925
Division (/), 101
1003
495, 499-500, 855-856, 964, 996-1008,
Document retrieval, 997, 999-1000
420-421, 424-425, 427-428, 435-442,
external, 24, 1017
DOM, 58-60, 64, 156, 228, 356, 578
149-151, 156, 161, 177, 185-190, 194,
Domain constraints, 64, 72, 76, 581
138, 150-151, 186-187, 189, 235, 324, 578,
dot notation, 63, 366-367, 377, 380, 386, 404
254, 261, 339, 382, 437, 467, 469-470,
Double buffering, 588, 598, 603, 610, 629-630,
Double precision, 87, 382
624, 628, 635
553, 844, 936
504, 892, 1018
E
1048
894, 907, 913
electronics, 3
377, 384-385, 387-389, 407-409, 425-429,
elements, 4, 8, 20, 39, 49, 60, 64, 84-85, 117,
form, 39, 363, 409, 439, 492, 497, 1059
else, 462, 472, 475, 482-483, 492, 608, 613, 621,
ELSEIF, 482-483, 696-697
412, 414, 628
871, 873-875
End tag, 423-425, 428, 434, 491
257, 259-261, 266, 299, 314, 332, 334-335,
Enter key, 456
214-221, 229-230, 232, 234, 249-255,
Entity, 7, 19, 29, 56-58, 62, 64, 69-70, 72, 74, 76-77,
622, 1050-1052
247, 280
237, 313, 331, 334, 337, 351, 472-473, 801,
environments, 18, 34, 41-42, 473, 831, 852, 868, 885,
work, 42, 852, 1032
694, 921-922
684, 774-775, 859, 881, 1004, 1043
483-484, 502, 624, 754, 774, 867, 901
934-936, 938, 940-941, 944, 954-955, 981
868, 930-932, 934-938, 941, 943-944, 962,
Excel, 2, 311, 1045
460-461, 467, 474, 509
480, 774
183, 332, 341, 359, 371, 455, 467, 473, 502,
execution:, 39, 754
out-of-order, 881
EXISTS, 13, 33-34, 61, 72, 74, 103, 116, 120-123,
Expert system, 351, 989
191, 198, 380, 403-409, 413-414, 437-439,
built-in, 413-414
pages, 1011, 1018
F
760-761, 776-777, 829-832, 834, 881, 883,
FAT, 516
100, 102, 105-106, 115, 131, 139, 251,
Federated databases, 925
999-1004, 1018, 1031-1032
1072
592, 600-604, 608, 610, 612, 621-622, 632,
File, 4-5, 7-12, 15-22, 24-26, 27, 31, 34, 38, 41-44,
sequential, 592, 597, 607-611, 634, 636, 649-650,
file access, 10-11, 44, 633
40-41, 43, 48, 56, 75-76, 204, 314, 332-334,
access method, 606, 630, 633, 636, 638, 650, 674
610-611, 614-615, 619, 621-622,
field, 592, 599-602, 604, 608, 610-611, 614-615,
HTML, 24, 437, 491, 500
588, 592, 597, 599-602, 604, 608,
streams, 1, 600, 424
109, 145-146, 160, 190, 289-295, 297, 300,
Foreign key constraints, 107, 435
65, 75, 92-93, 120, 122, 130, 134, 150-151,
556-558, 577-578, 581-584, 599, 628,
design a, 277, 545
542, 549, 552, 582, 737
1044
Forms, 3, 18, 24, 27, 36-37, 71, 155, 249, 268, 318,
Forwarding, 46
1008, 1012, 1025
124-126, 128-129, 140, 169-171, 176,
computation of, 772
Function calls, 456, 458, 466, 471-477, 484-485
525, 530, 532, 535-538, 541, 543-544, 551,
Functions:, 40, 470, 479, 497
102, 105, 119-120, 124-126, 134,
point of view, 130
G
OR, 570, 673, 676
853, 943-946, 949, 953-955
705-708, 767-769, 771, 789-791, 807, 915,
Graphics, 14, 357, 496, 498, 892, 1021, 1065
143, 169-170, 177, 189, 195, 208, 409-410,
guidelines, 226, 264-265, 330, 332, 353, 428, 510,
520, 543-544, 733, 744-746, 854
H
, 424
, 424 633-634, 667, 740, 748, 824, 829, 851,
Handles, 27, 40, 46, 50, 458, 862, 892-893, 908, 916,
handling, 14-15, 41, 45, 403, 461, 467, 616, 674, 799,
Hard disks, 593, 596
331, 334, 336, 593, 595-596, 627-629, 749,
Harmonic mean, 1016
694, 718, 722
675-676, 679, 682-683, 695, 699-700, 702,
hash index, 668, 675, 736
, 423-424974, 976, 978, 980-981, 1054-1055
336-337, 354, 607, 631, 637, 744, 870, 909,
Heuristic, 350, 686, 705-706, 708-709, 713-715, 724,
Hexadecimal notation, 88
298, 370, 372-373, 379, 389, 414, 444,
hierarchy of, 36, 207, 389, 589, 853, 1028
462, 471, 484-485
497-500, 502-503, 858, 892-893, 957, 993,
HTML (Hypertext Markup Language), 420
I
351-352, 635, 649, 716, 745, 885, 988
868, 963, 965, 967, 1047
961, 969
1073
958, 963-967, 982, 984, 988, 1029
images and, 598, 958, 964, 966
130, 135, 143, 201, 203-204, 298, 313-319,
implements, 7, 81, 103, 456, 877, 960
82-85, 87-109, 113, 128-143, 148-158,
incremental backup, 41
617, 636-680, 682-683, 691, 717-719, 733,
Indices, 561
582-583, 586, 838-839, 871, 968, 973, 975,
Inference rule, 554, 557
274, 299, 350, 359-361, 368-374, 376,
class hierarchies, 359-360
inheritance using, 395
465, 481, 492-494, 498, 503-504, 551, 555,
109, 132, 140, 365, 384-385, 387-388,
inserting, 74, 132-133, 255, 365, 501-502, 505, 610,
files, 610, 646, 651, 653-654, 660, 662, 735
543-544, 604-605, 610, 614, 616, 619, 621,
installation, 48, 85
219-220, 230-234, 236, 238, 243, 264, 269,
Instance method, 375-376, 378
227, 229-230, 232-234, 238, 243, 268-270,
instruction set, 671
unsigned, 382
Integration, 321-323, 325-327, 350, 352-353, 355-356,
Integrity constraints, 18-19, 26, 56, 64, 66, 69-72, 74,
domain constraints, 64, 72, 76, 581
intellectual property, 868, 873, 876
493, 892-893, 993, 998, 1028-1030
46-47, 51-52, 106, 149, 314, 336, 350, 369,
Comparable, 1022
482
Interference, 11, 758-760, 797-798
782-783
853, 855, 868, 878, 914, 919-922, 1026
Internet and, 868
189-190, 260, 388, 666, 687, 692, 696-697,
interviewing, 233
369, 440, 454-489, 505-506, 747-779,
history, 759-760, 776-777, 982, 993, 997,
Inverted index, 999-1001, 1006-1007, 1011-1013,
I/O (input/output), 595, 716
882
546, 581, 600, 734, 744, 750-754, 757, 760,
Iterate, 387, 474
457, 462, 468-470, 479, 484, 504
cursor, 468, 485
J
457-459, 466-471, 476-480, 484-486, 500,
Class Library, 484
Java code, 467
JDBC, 471
strings in, 505
500, 892-893
Job, 12-13, 138, 205, 233, 249, 253, 265, 271-272,
Join:, 162, 164
292, 294, 440, 514, 518, 532, 572, 698-701,
Join ordering, 723-724
K
649-650, 654-657, 659-660, 666, 676-678,
keyboard, 282, 628
131, 205, 209, 216, 234-235, 241, 289-295,
candidate, 65-66, 76-77, 524, 526, 530, 532-533,
Sense, 241
L
66, 70, 83, 105, 119, 150, 177, 188, 198,
Laptops, 897, 995
shape, 957
368, 412, 446, 496, 509, 617, 760-763, 767,
Leaf, 166-167, 257-258, 266, 422, 427, 429, 436-437,
legacy systems, 622, 685
929, 993-994, 997, 1017, 1029-1030, 1032
open source, 51
280, 303, 311, 328, 350-351, 428, 465, 546,
Line:, 461, 476
722
1019-1020, 1023, 1027
Linux, 491, 626
1074
101-102, 107, 129, 134, 152-153, 158,
Lists, 36, 84, 102, 129, 163, 166, 276, 351, 388, 409,
Literal, 87-88, 93, 100, 381-382, 386-387, 411, 495,
Literal value, 362-363, 386
1043-1044
143-146, 162-163, 199-200, 205-206,
Lock table, 782-783, 785, 788, 799
794-801, 803-808, 822, 863, 909-911, 925
911
803-807, 819, 908-912
503-504, 555, 562, 567, 694-695, 698-700,
loops, 316, 329, 456, 458, 462, 480, 483, 495, 694,
prompt, 462
M
994, 1025
603-605, 611, 644, 688-690, 695, 698-699,
maintainability, 1045
401-402, 413, 415, 457, 523, 543, 667, 672,
mapping, 23, 32-35, 38, 56, 60, 62, 92, 203-204,
value, 60, 62, 211, 291, 294-295, 297-298, 300,
Maps, 1, 23, 25, 277, 613-614, 852, 955-956, 963, 998
Mass storage, 41, 590
1037-1038
Maximum, 70, 87-88, 125, 169, 171, 218-219, 229,
Maximum value, 125, 792
554-555, 624, 713, 767, 832, 944, 962, 970,
measurement, 57, 350, 956, 1029
guided, 1028
260, 262, 264, 269, 283, 298, 397, 408-409,
Memory, 17-18, 40, 43, 60, 282, 329, 411, 589-592,
allocation, 603, 607
821-822
695, 698, 716-717, 719-722, 727, 881,
memory cards, 590
930, 963-964, 982, 993-994
Metadata, 728, 877, 883, 885, 891, 913-914, 919, 925,
3D objects, 988
360-361, 365-367, 369-373, 375-378, 402,
Add, 10, 265, 367, 629, 674, 728, 804
721-722, 791, 848
334, 342, 345, 358, 360, 372-374, 378-379,
class name, 401
Metrics, 992, 1014-1015, 1029
350, 434, 509, 626, 662, 700, 735, 797, 854,
Mod, 607, 611-614, 619, 621, 625, 633, 701, 866
869, 887
249, 259, 261, 268, 272-275, 283-285, 303,
theory, 56, 273, 988
106, 216, 236, 246-247, 258, 268-269, 271,
activity, 11, 28, 335, 348, 961, 1029
1005
Modem, 283
783, 897, 905
458, 480-482, 893, 907, 920-921, 994
803-804, 940-941, 1058
640, 661-662, 713-714, 767, 842, 908
981-983, 987-988, 993-994, 996
Multiple, 4, 8, 11, 14-16, 18, 25, 31, 42-43, 45-47, 88,
declarations, 8
394, 399, 414
822-824, 830-831, 838, 1037
575, 578, 584
N
493
371, 378, 380, 394, 403-404, 406-408, 412,
names, 2-3, 7-8, 10, 20, 30, 38, 56-59, 62-63, 65, 67,
Namespace, 434
1075
National Library of Medicine, 1009-1010
189, 194, 294, 514-515, 518-519, 542, 546,
Natural language processing, 1018
1027-1028
129-130, 139, 141, 164, 195, 208, 235, 237,
Nested relation, 164, 528-529, 532
726
334, 399, 416, 598, 605, 622, 627-628,
Network management, 416
data communications and, 878
868, 879, 881-882, 902, 924, 930, 958, 988,
New York Stock Exchange, 955
437-439, 619, 627, 651-652, 654-660, 662,
children, 438, 801
803, 862-863
Noise, 861, 867-868, 962, 966, 995, 1023
NOR, 30, 74, 98, 213, 259, 386, 401, 497, 519,
Normal, 61, 152, 170, 350, 509-510, 520, 523-528,
Normalization, 16, 56, 64, 350, 508-549, 550, 552,
normalizing, 529, 531, 541, 548, 584, 1041
178, 182, 187, 194, 201-202, 205-206,
null character, 460
O
oriented programming, 17, 22, 249, 265, 358-359,
use an, 33, 45, 106, 204, 405
106, 249, 259, 265, 320, 334-335, 339-340,
requirements analysis, 334
266-271, 277-278, 335-339, 344, 357-365,
distance of, 959, 966
984
ODBC, 45, 56, 105, 471, 485, 500-501, 892-893, 916
462-464, 471, 490-491, 505, 604-605,
Open addressing, 613-614
491, 594, 628, 740, 750, 769, 812, 857, 882,
Operating system (OS), 38, 909
835, 863, 922, 998
Operations, 10-12, 19, 22-23, 26, 28, 30-32, 35, 37,
operator overloading, 360, 371, 373, 414
348, 589, 684-730, 732, 897, 901-903, 918,
search engine, 997, 1027
715-721, 723-728, 744-745, 889-890, 907,
OR function, 481-482, 551
723-724, 729, 760, 764, 766, 852, 863, 918,
Orthogonality, 413, 415
716, 749, 865, 892, 1042, 1059
640, 649, 661-663, 680, 754
P
741, 799-801, 803, 805, 812-815, 823-828,
Page numbers, 422, 637
425, 490-491, 493-494, 504, 594, 637,
1004-1005, 1008, 1011, 1017-1027, 1031
812-814, 825-826, 830, 999, 1021
Paging, 739, 810, 823-824, 829-831, 835
panels, 1023
989, 1032
754, 858-859, 1054, 1056
332-333, 379, 406, 456, 465, 468, 472-475,
Parent, 36, 50, 221, 245, 256, 346, 428, 435, 437-438,
Parent class, 346
741-742, 861, 895, 910, 914, 921, 923,
Pascal, 17, 612
501-502, 857, 859, 864, 919
437-439, 481, 493, 499, 610, 691-693, 727,
Path expression, 404, 410, 438-439
332-333, 339, 607, 636, 690, 692, 694, 706,
Pattern, 100, 437-438, 537, 625, 956, 962, 964, 1000,
Pattern recognition, 962
1018, 1024-1027, 1033, 1047
312-317, 331-334, 350, 352, 588-590, 592,
Peripherals, 627
1018
348, 351, 353, 400, 688-689, 695, 700-701,
Phone numbers, 57-58, 239, 277, 377, 742, 1010
332-333, 348, 350-351, 354-355, 733, 736,
Physical tables, 741
strategy, 705, 715
1076
318, 350-351, 355, 369-371, 380-382, 388,
pointer, 49, 362, 462, 472-473, 600, 602-605,
pointers, 22, 332, 603, 614, 616, 619, 622, 634,
point-to-point connections, 627
area, 372, 956-957
310, 384, 387-388, 409, 416, 418, 440, 462,
power, 42-43, 92, 102, 115, 164, 177, 189, 212, 216,
Precedence, 767-769, 771, 778
1014-1016, 1018, 1029-1030
262, 265-266, 274, 672, 744, 804, 806-807,
Predicates, 185, 869, 923, 956-957, 959, 961,
Prediction, 957, 961
Documents, 1018, 1023-1024
Prefixes, 456, 595, 1009
677-678, 691, 718, 721-722, 736-737, 741
105, 137, 145-146, 160, 211, 289-297, 300,
Primary keys, 66, 69, 80, 289-293, 435, 523-524, 530,
Prime number, 614
867-868, 872-874, 876
privacy issues, 867, 873
873
873, 875
870-875, 883
Probing, 695, 700-702, 723
481-482, 509-510, 528, 532, 543, 550-551,
Procedures, 19, 23, 28, 46, 48, 62, 119, 287, 319, 344,
665, 812, 854-855, 859, 870, 893, 916, 918,
Process, 3, 5, 11, 17, 30, 35-36, 45, 50, 64, 93, 130,
states, 205, 340-341, 581, 761, 781, 863-864
274, 313, 328-329, 336, 598, 654, 739-740,
suspended, 740, 749
37-38, 40, 42-45, 48, 75, 78, 82-84, 106,
processors, 20, 428, 430, 749, 882
714
40-42, 203, 236, 256-257, 267, 299, 340,
Program code, 19, 333, 402, 456, 461, 465, 481, 484,
Program modules, 455, 480-481
455, 458-460, 462-464, 466-467, 472,
Programming, 7, 14, 17-18, 21-22, 35-36, 38, 45-46,
bugs, 754
358-360, 367, 371, 380, 411, 416, 459,
Programming errors, 754
51, 83, 93, 105, 334, 358-360, 365-367, 373,
Programs, 3-5, 7-11, 14, 19-21, 24, 27, 31, 33-34,
context of, 590, 892
714-715, 979
568-569, 675, 696, 708, 711-712, 714-715,
Prolog, 62, 549, 585, 968-970, 973, 975, 977, 979,
Properties, 2, 10, 12, 48, 65, 204-205, 229, 246, 252,
of algorithms, 510
520-522, 525, 533-534, 536, 538, 541-542,
873, 876
541-542, 551, 556, 558-562, 564-569,
Property rights, 868, 873, 876
793, 797-801, 803-807, 811, 813-816, 822,
LAN, 628
protocols, 56, 440-441, 756-757, 759, 767, 771-772,
prototyping, 15, 333
Q
46, 66, 83, 96-99, 101-102, 105-109,
Query, 4, 7, 13-14, 17-18, 21-23, 34, 36-40, 45-46, 48,
Query:, 118, 129, 152-153, 194, 528, 665, 703, 706,
Query compiler, 39, 889-890
708, 714, 716-717, 728-730, 902, 905
148-149, 166, 177, 190, 198, 245, 359, 365,
Query optimization, 39, 45, 130, 183, 348, 684-686,
Query processing, 17-18, 22, 55, 61, 589, 684-730,
Queue, 782-783, 788, 791
Queuing, 624-625, 918
R
211, 348, 350, 352, 403, 405, 424, 439, 611,
Range query, 667, 959
464, 485, 590-591, 593, 595-598, 603, 605,
1077
607-608, 610, 624-625, 629-630, 634,
Read operation, 772, 802, 913
716, 726, 749, 754, 783, 794, 796, 940,
read/write heads, 596
56, 75, 80, 109, 132-133, 220, 244, 248,
recording, 594, 756, 812, 830
360, 624, 740, 747-748, 750-751, 754-763,
Recovery manager, 750, 755, 815, 826, 828-829, 881,
recursion, 171, 968, 980, 989
138-140, 216, 226, 276-277, 291-294, 355,
References, 69-70, 76, 85-86, 90, 97, 105, 109,
Reflection, 3
95-97, 102-106, 118-120, 126, 128, 137-139,
Relation schema, 58, 60, 62-65, 69, 75-76, 297, 508,
Relational algebra, 55, 71, 75, 81, 82-83, 92-93, 97,
Relational calculus, 55, 72, 83, 148-200, 301, 506,
Relational database, 19, 22, 35, 37, 44, 55-81, 82-85,
Relational database model, 204, 849
76-78, 84-85, 107-111, 114, 143-144,
Relational database system, 83
Relational databases, 11, 21-22, 29, 48, 56, 66, 82,
relational expressions, 109, 191
75-76, 81, 84, 92, 101, 148-149, 153, 177,
Relational operators, 71, 188, 190, 195, 714, 728,
Relations, 9, 28, 56-57, 59, 61-64, 66-67, 69-76,
Relationship, 7, 19, 24, 29, 46, 49, 56, 62, 67-68,
Relationship set, 214-216, 218, 223, 230, 234, 236,
Relationships, 5, 18, 21, 23, 26, 28-30, 32, 50, 62, 69,
release, 244, 787-788, 869-870, 908-909
190, 738, 1065
896-898, 901, 905, 913, 918-921, 924,
reporting, 7, 173, 313, 387, 1037
124, 141, 169, 177, 190, 234, 283, 291, 293,
requirements engineering, 355
747
652, 813, 913, 977
505, 606, 614, 622, 684, 693, 736, 749, 913,
Return type, 468, 481
214-215, 217, 222, 227-229, 234, 236,
922, 960, 1031
852-853, 872
Rollback, 756, 762-763, 774-775, 793, 795, 814,
Roll-up, 1039-1040, 1045
500, 651-652, 655-657, 659-662, 665, 703,
Root node, 166, 437, 651-652, 655-657, 660-662, 665,
Rotation, 398, 593, 596, 943, 958, 966, 1045, 1054
1054-1056
469, 503-504, 562, 668-672, 674, 725-726,
RSA, 866, 874
257, 328, 372, 427, 495, 553-554, 557-558,
Rules, 19-20, 46, 63-64, 69, 71, 75, 83, 119, 179-180,
Run-length encoding, 675
S
785-789, 791-792, 794, 801-802, 807, 809,
Scheduling, 332, 625, 740
Schema, 27-28, 30-35, 37-38, 49-53, 56, 58-60,
Science, 2, 5-6, 26, 30, 54, 57, 108, 112, 147, 246,
Screens, 331
1007-1010, 1017, 1019, 1026-1027,
Search keys, 668, 676
1078
Search tree, 652-654
619, 637, 648-649, 651, 668, 673, 675-676,
Searching the Web, 1017, 1019
677-678, 719-722, 727, 741
83-84, 106, 122, 135, 165, 205, 211,
authenticity, 841-842
Security threats, 836
1054-1056
128-131, 133-136, 140, 142, 149-155, 160,
Selection, 93, 95, 97, 140, 150-152, 158, 160, 164,
Selections, 520, 736, 954
356
510-511, 513, 521-523, 528, 543, 550, 559,
Semaphores, 909
177, 187-188, 203, 316, 328, 335-339,
Sequence:, 663-664
776-779, 780-781, 785, 787, 791-792,
Serializable schedule, 766-767, 769, 776-778, 785,
server, 27-28, 40-48, 51-52, 55, 106, 311-312, 329,
servers, 3, 22, 27, 40, 42-47, 312, 458, 460, 593,
compatibility, 45
services, 40, 44, 46, 48, 311, 331, 627, 856, 862, 866,
882, 914-917, 919, 963, 999, 1017-1018,
classification of, 48
sessions, 852
696-697, 702-703, 705, 728-729, 981
866-867
262, 279, 461-462, 468, 472-473, 477, 511,
Sockets, 855
34, 38, 40-45, 47-48, 51-52, 56, 228, 246,
malicious, 4
software and, 8, 13-14, 40, 43, 334, 627
314, 332, 334-335, 349
789-791, 804, 855, 879, 904, 962
687-689, 694, 699, 702, 704, 716, 722, 730,
sound, 282, 554, 575, 579, 967
Source, 2, 40-41, 48, 51, 55, 189, 336, 356, 425, 456,
source code, 336, 465, 471, 483-484
filtering, 1027
329, 333, 342, 344, 374, 386, 389, 391, 394,
Speed, 17, 34, 248, 251-253, 297, 306, 588-590, 596,
Spindle, 593
440, 455, 774, 854-855, 863-864, 893, 935,
Star schema, 1040-1042, 1048
75-76, 78, 80, 92, 94, 109, 111, 114, 174,
389-390, 392, 396, 433, 512, 516-517, 519,
Statecharts, 339
131-132, 135, 139-140, 189, 367, 459-461,
Statement-level trigger, 935, 941
219, 339-341, 366, 511, 515, 521-522, 542,
exit, 341, 777
Statistics, 40-41, 312, 333, 724, 730, 735, 739-740,
Stemming, 1000, 1009, 1012, 1029, 1031-1032
402, 415, 465, 473, 477, 528, 532, 623,
Storage devices, 20, 588-589, 591-592, 597, 627-629,
backup, 591, 597, 829
storage management, 627, 824
435-436, 447, 514, 525, 588, 593, 600, 613,
Storing data, 8, 600
366, 382, 385, 391, 393, 396-397, 403-406,
default value, 89
411, 484, 493-496, 503, 505, 600, 612, 859,
concatenation of, 666
Stripe, 626
401-402, 405-406, 408-410, 600
75, 92, 140, 148, 166, 210, 273, 312,
decision, 319, 401, 1035-1036
296-299, 307, 401-402, 551, 578, 582, 1010
1079
517, 570, 596, 624, 668, 687, 726, 753, 804,
Sun Microsystems, 471, 476
299-301, 322, 402
558, 574-575
70, 103, 189, 263-264, 271, 275, 311, 329,
SUPREME, 866
hubs and, 627
657, 866, 969, 971, 1002-1003, 1050-1052,
Synchronization, 329, 347, 862, 914
188-189, 367, 377-378, 394, 402-404,
details, 36, 38
821, 829-831, 840
T
, 424base table, 139
115, 117, 119, 123-124, 129-130, 133-141,
attributes of, 119, 123-124, 134, 160, 173, 193,
Master, 742, 918, 941
Tag, 423-428, 434-435, 437-438, 491, 493, 497, 500
493, 500, 967
755, 845, 853, 880, 915, 963, 994-995, 997,
TCP/IP, 629
, 424, 496348, 351-352, 413, 441, 589-590, 593, 595,
Temperature, 940, 958
564, 586, 767-768, 770-771, 773, 776, 863,
automated, 314
346, 420, 423-427, 447, 465, 471, 483,
Text:, 494
alternative, 430
1031-1032
average, 333
63-65, 82-83, 87-89, 101, 104, 114, 125,
Time:, 1056
772, 780, 791-795, 797, 803-808, 866,
Timestamps, 772, 780, 789-792, 795, 797, 799, 803,
Timing, 734-735, 862-863
424, 453, 485, 489, 507, 547, 738, 985
309-310, 314, 316, 319, 321, 325, 328-329,
Arrow, 1052
Top-down design, 322, 509
, 423-424, 497236-239, 256, 262, 275-279, 344, 472, 476,
Trademark, 471, 476
75-76, 83-84, 203-204, 283, 310-312, 315,
Transaction file, 610
1054-1055
transparency, 879-882, 886, 889, 894, 905, 913, 916,
Traversal, 800
657
656-657, 660, 665, 674-675, 679, 706-708,
implementations of, 657, 982
941-942
733-746
91-93, 95-99, 102-103, 105, 116-124, 126,
Tuple variable, 93, 119, 178-181, 184, 190, 194, 687,
Two-dimensional array, 479
801, 805-808, 822, 925
U
296, 298, 302, 331-333, 339, 349, 374-376,
Underflow, 662, 664-665
120, 122, 131, 145-146, 237, 241, 243-244,
United States, 57, 205, 207, 521-522, 837, 864, 886,
Universal access, 1017
70-77, 90-91, 102, 104-109, 132-138, 142,
Update command:, 465
316, 455, 465, 493, 734-735, 737, 752, 775,
upgrades, 628, 788
988, 1009, 1018, 1024-1027, 1029, 1031,
1080
USB (Universal Serial Bus), 590
374, 378, 382, 386, 389-390, 392, 413-414,
User-defined functions, 378
123, 130-131, 143, 201-202, 204, 212-213,
UTF, 431
V
805-806, 859, 1028
87-89, 95, 98, 100-101, 103, 105, 107,
initial, 31, 87, 212, 216, 351, 377, 492, 619,
truth, 116, 179, 184, 974
92-93, 95, 100-105, 116-119, 122, 124-126,
undefined, 61, 116
186-187, 190, 194, 360, 363, 404-405, 408,
variable declarations, 459
194, 360-361, 365, 369, 403-404, 410-411,
data type of, 479
Vector, 892, 965, 996, 999-1000, 1002-1004, 1013,
vector graphics, 892
966-967, 982, 998, 1017, 1029
View, 11-13, 25, 32-33, 35, 51-52, 85, 104, 130,
View:, 847
1034, 1037, 1043-1044
W
273, 285, 312, 329-331, 420-425, 427, 433,
Web analytics, 1026
1018-1021, 1027, 1033
493-494, 504, 878, 892, 993-994, 996,
Web server, 46, 491, 500, 858
name of, 1027
353, 414-415, 448, 456, 485, 505, 544-546,
Where-clause, 120, 687, 727, 743, 775, 857
35-52, 55, 59-66, 70-77, 79-81, 82-84,
With check option, 137
533, 569, 843, 853, 964, 995, 999-1000,
frequency of, 1002, 1008
1017-1019, 1033
World Wide Web (WWW), 878
Wound-wait, 789-791, 806-807
121-122, 124-126, 129-131, 133, 139, 154,
Write operation, 755, 757, 763, 766, 772, 793-794,
writing, 20, 85, 96, 105, 131, 178, 187, 194, 253, 318,
X
450-453, 836, 854-856, 876, 892-893, 926,
XML (Extensible Markup Language), 420
442-448
Y
740, 968
Z
404, 429, 439, 495-496, 621, 754, 813, 870,
ZIP codes, 670, 923, 956
1081
Cover
Post navigation