SPY

1 January 2009 Tool
Archived Latest release: 1 February 2013

SPecification RecoverY: inferring behavior models of stateful software components, and using them at run time to detect when a component's behavior changes

Overview

SPY — SPecification RecoverY — infers behavior models of stateful software components from their executions, rather than asking developers to write those models by hand. Given a component and a way to exercise it, SPY recovers a model describing how the component actually behaves: which sequences of operations are legal, and what each of them does to the component’s state.

Two flavors of model are produced for a component. A behavioral equivalence model describes its behavior precisely, within a small scope. A protocol behavior model generalizes that into the component’s protocol of interaction — the order in which its operations may legitimately be called.

SPY was developed in the deepse group↗ at Politecnico di Milano. The section below covers SPY-monitoring, the branch of the work that puts the recovered models to use at run time.

SPY-monitoring

Applications routinely depend on components they do not control, and those components change: an update fixes one behavior and quietly alters another, or tightens a protocol that callers were relying on. SPY-monitoring watches for exactly that. It observes an external component while the system runs, keeps its behavior model up to date with what it sees, and reports a misbehavior the moment the component stops matching the model.

Monitoring proceeds in two phases. Before deployment, a trial phase builds the initial models, drawing on whatever knowledge of the component is already available — in the packaged example, the parameters and call sequences found in the component’s own JUnit test suite. Those initial models are necessarily partial: a behavior the test suite never exercises cannot appear in them. So at run time the models are refined with the behaviors actually observed, which both completes the picture and separates genuinely new behavior from a real change. Execution traces collected from the running system are then checked against the models, and a trace the model cannot explain is flagged as a functional change in the component.

The approach was evaluated on two components. StorageService is a purpose-built example that lets clients store data in compressed form over RMI, chosen because it combines a non-trivial interaction protocol with container-like behavior; faults injected into the server — making its methods always succeed, or always throw — were detected as behavioral changes. The Java client of RabbitMQ served as the real-world case: the server was modified so that a queue may be declared only once, turning a previously harmless redeclaration into an error, and SPY-monitoring correctly reported the protocol change.

The tool package ships the monitor together with the StorageService server, client, and test suite, plus the RabbitMQ client and both the original and modified servers. Its workflow runs as a short sequence of scripts: collect instances and generate a component wrapper from the test suite, generate the trial-phase models from that wrapper, then run the application with monitoring enabled against a configuration file.

Technologies

Publications

13 September 2013 Paper
16748 words · 84 minutes

Detecting component changes at run time with behavior models

Modern software systems are composed of several services which may be developed and maintained by third parties and thus they can change independently and without notice during the system’s runtime execution. In such systems, changes may possibly be a threat to system functional correctness, and thus to its reliability. Hence, it is important to detect them as soon as they happen to enable proper reaction. Change detection can be done by monitoring system execution and comparing the observed execution traces against models of the services composing the application. Unfortunately, formal specifications for services are not usually provided and developers have to infer them. In this paper we propose a methodology which exactly addresses these issues by using software behavior models to monitor component execution and detect changes. In particular, we describe a technique to infer behavior model specifications with a dynamic black box approach, keep them up-to-date with run time observations and detect behavior changes. Finally, we present a case study to validate the effectiveness of the approach in component change detection for a component that implements a complex, real communication protocol.

Detecting component changes at run time with behavior models
2 June 2012 Paper
3293 words · 17 minutes

Runtime Monitoring of Component Changes with Spy@Runtime

We present Spy@Runtime, a tool to infer and work with behavior models. Spy@Runtime generates models through a dynamic black box approach and is able to keep them updated with observations coming from actual system execution. We also show how to use models describing the protocol of interaction of a software component to detect and report functional changes as soon as they are discovered. Monitoring functional properties is particularly useful in an open environment in which there is a distributed ownership of a software system. Parts of the system may be changed independently and therefore it becomes necessary to monitor the component's behavior at run time.

Runtime Monitoring of Component Changes with Spy@Runtime
17 October 2011 Paper
6036 words · 31 minutes

Runtime Monitoring of Functional Component Changes with Behavior Models

We consider the problem of run-time discovery and continuous monitoring of new components that live in an open environment. We focus on extracting a formal model—which may not be available upfront—by observing the behavior of the running component. We show how the model built at run time can be enriched through new observations (dynamic model update). We also use the inferred model to perform run-time verification. That is, we try to identify if any changes are made to the component that modify its original behavior, contradict the previous observations, and invalidate the inferred model.

1 May 2010 Paper
6180 words · 31 minutes

Behavior model based component search: an initial assessment

We focus on the problem of searching components based on semantic queries on their provided interface. Although semantics-based search has long been advocated as a key enabler in the context of component-based software development and, more recently, service-oriented computing, no practical and scalable approach has been proposed yet. This paper presents a promising model-based search technique for interface behaviors based on operational specifications, called behavioral equivalence models (Bems). Semantic queries are expressed equationally, following an algebraic specification style. The search engine tries to match specifications against queries. This can be done quite efficiently by encoding Bems into relational models and queries into relational logic formulae, whose satisfiability is checked with the SAT-based constraint solver KodKod. We can report on an initial very promising assessment of the proposed technique, which has been applied to searching components in Java libraries providing container functionalities.

Workshop No tags assigned
20 March 2010 Paper
8018 words · 41 minutes

Automatic Cross Validation of Multiple Specifications: A Case Study

The problem of formal software specification has been addressed and discussed since the infancy of software engineering. However, among all the proposed solutions, none is universally accepted yet. Many different formal descriptions can in fact be given for the same software component; thus, the problem of determining the consistency relation among those descriptions becomes relevant and potentially critical. In this work, we propose a method for comparing two specific kinds of formal specifications of containers. In particular, we check the consistency of intensional behavior models with algebraic specifications. The consistency check is performed by generating a behavioral equivalence model from the intensional model, converting the algebraic axioms into temporal logic formulae, and then checking them against the model by using the NuSMV model checker. An automated software tool which encodes the problem as model checking has been implemented to check the consistency of recovered specifications of relevant Java classes.

Automatic Cross Validation of Multiple Specifications: A Case Study
16 May 2009 Paper
8988 words · 45 minutes

Synthesizing intensional behavior models by graph transformation

This paper describes an approach (SPY) to recover the specification of a software component from the observation of its run-time behavior. It focuses on components that behave as data abstractions. Components are assumed to be black boxes that do not allow any implementation inspection. The inferred description may help understand what the component does when no formal specification is available. SPY works in two main stages. First, it builds a deterministic finite-state machine that models the partial behavior of instances of the data abstraction. This is then generalized via graph transformation rules. The rules can generate a possibly infinite number of behavior models, which generalize the description of the data abstraction under an assumption of “regularity” with respect to the observed behavior. The rules can be viewed as a likely specification of the data abstraction. We illustrate how SPY works on relevant examples and we compare it with competing methods.

Synthesizing intensional behavior models by graph transformation