A major learning task confronting workers today is mastering the rapidly changing information technology (IT) on their desktops. A MITRE research project, Organization Wide Learning (OWL), is assisting MITRE staff in learning new IT skills in an effective and time-efficient manner. The OWL project is based on the development of a software tool that logs the IT applications of individual users, pools and analyzes the aggregate data, and recommends the use of specific features to individuals based on the user patterns of their peers.

Figure 1. The OWL Toolbar Button

Figure 2 displays a sample OWL log. The first five rows record general information: the logger version, the date and time stamp, and the author; followed by the platform, processor, and version of Word. At this point detailed logging begins. Each time the user enters a Word command, the logger adds a row to the log file. Each row contains a time stamp, the command name, and possibly one or more arguments. For example, the row beginning 17:11:34 records these facts: at 5:11:34 p.m. the author used the FileOpen command to open the file "Notes for UM'99." The author then performed some minor editing (copy, paste, etc.), then printed the file. The log does not record text a user enters; this omits some potentially useful information but preserves users' privacy and makes logging more acceptable. Logging captures a detailed record of a user's activities but the record may be sketchy however, because since the logged data is neither a complete census of the user's actions (for example, the user might work on other machines), nor a random sample, but rather an arbitrary selection.

Figure 2. A Sample OWL Log

MITRE Senior Engineer Peter Schaefer, who wrote the current version of the logger in Visual Basic for Applications (VBA), included an auto-update feature, which automatically distributes new versions of the logger by a transparent process. As he explains, "The logger can log any application that has a VBA interface."

Analyzing the Logged Data

A portion of summarized user data for one calendar year is displayed below in Table 1. While the pooled data, when graphed, form a smooth curve, individual user models vary not only in the number of distinct commands used, but also in the relative proportions of the commands used. For example, the second most frequently used Edit command, Edit Clear (the Delete Forward key), was used by only 10 of the 16 users logged in 1997: four users did not use the command at all and two others used it only once or twice, probably accidentally.

Sample OWL Log

An individual who is not using a command that is used by others, might use the command if told about it. Similarly, underuse of a command may indicate a willingness to learn more ways to apply the command. Overuse may indicate reliance on a weak general-purpose command, such as Delete, when a more powerful specific command, such as DeleteWord, might be more appropriate.

A given volume of logged data will provide more reliable estimates of the user's knowledge of more frequently used commands than less frequently used ones. Table 1 illustrates the frequency of many common commands. For the less frequently used, a different sort of analysis should be done. There is a high correlation between volume of observed data and number of distinct commands used. Thus, we must be careful not to equate our nonobservation of a command with a lack of knowledge of that command on the user's part. It may simply be that enough data have not been acquired to observe it.

Table 1. Command Sequences and Percentages

Learning opportunities (nonuse, underuse, overuse, and edge of use) can be prioritized and presented to the user in terms of learning recommendations. Table 2 shows a portion of one user's information. Learning recommendations determined by pooling the knowledge of a set of peers and by individualizing the instruction (by showing a user what commands others have utilized), may result in recommendations that the individual finds particularly useful. These recommendations can reduce the effort of finding instruction while simultaneously increasing the benefit of learning.

The first column of Table 2 lists the 10 Edit commands that are most frequently used (commands nominally under Word's Edit menu), sequenced by their overall frequency of use.

The second column of Table 2 lists the expected value for each of these 10 commands. The expected value is the amount of use the command would have had if the individual had used it in a manner consistent with the use of related commands and consistent with others' use of the command.

The third column of Table 2 lists the actual use of these commands during the time this individual was logged.

The expected values are a new kind of expert model, one that is unique to each individual and each moment in time. The reason for differences between observed and expected values (between one's actual and expert model) might have several explanations, such as the individual's tasks, preferences, experiences, or hardware, but we are most interested in instances when the difference indicates a lack of knowledge or skill.

The fourth column of Table 2 contains various symbols. These are indicators of learning opportunities useful to an automated tutoring process. For example, these indicators are data that, combined with domain and curriculum knowledge, would result in recommendations for learning. The five symbols are: "OK," " " (blank), "New," "More," and "Alt."

Table 2. Individualized recommendations for User 274.

Notes:

1. OK: The person is using this command as expected.
2. New: The person is not using this command; OWL recommends the individual learn how to use it.
3. More: The person rarely uses this command. OWL recommends the individual learn more ways to use it.

A command whose expected value is zero need not be learned, and can be ignored; its indicator is a blank (not shown). A command that has an expected value, but is one the individual has never used, is a command the individual probably would find useful if learned. Its indicator is "New." A command whose amount of use is reasonably close to the expected value can also be ignored. The current value of "reasonably close" was set arbitrarily. Eventually the value can be determined empirically. The indicator for a command that is reasonably close to the expected value is "OK." A command that is used less than expected may be a component of text-editing tasks that are unknown to the user but potentially valuable; its indicator is "More." A command that is used more than expected may indicate ignorance of more powerful ways to accomplish some text editing tasks; its indicator is "Alt" (recommending an alternative command).

Users access the individualized recommendations that OWL has generated for them by clicking the "OWL Tip" button (Figure 1) to bring up the Tip window displayed in Figure 3. The Tip window displays prioritized lists of command names, comments, and priority rankings. Users can review the list of tips to select the ones they consider most promising. Clicking on the command name brings the user to the portion of the help system that describes usage of the command.

Figure 3. The OWL Tip Window.

Internet Applications Engineer Deborah Joy of MITRE who generated these customized sets of tips by hand for each user. Deborah is confident that the new automated tip generation process will improve the system for the growing number of volunteer users across the corporation.

To summarize, not only has IT become the medium in which much work is performed, IT skills have become a significant portion of workers' knowledge. In contrast to other tasks, IT tasks are observable, and can be logged and analyzed for several purposes. The OWL system utilizes the logged data on IT tasks and builds expert models based on a comparison of the pooled knowledge of individual users. The result is the development of individualized instruction based on comparing each individual's knowledge with the pooled knowledge of multiple users.  

Page last updated: October 7, 1999   |   Top of page