Guy Madison, Psychology, Umeå University

2008-2014

The goal of this project was to explore the musical pulse as a means for co-ordinating human action. To this end, sensorimotor synchronisation (SMS) was studied in response to three different kinds of stimulus sequences: continuous interval change (increasing and decreasing), pseudo-random and unpredictable perturbations from isochrony, and the natural variability introduced by another human operator. The project occupied myself (Dr. Guy Madison) on 100 percent of full time, Dr. Björn Merker on 50 percent and a research assistant part time during the years 2003 and 2004. The project was exploratory, and achieved its goals by adapting well-tried experimental methods to adress new task domains: a variety of tasks involving perturbed or changing tempo as well as that of establishing and maintaining cooperative synchrony between two human performers. Human performance in these respects was documented, new methods for analyzing and displaying such data were developed, novel phenomena in the tapping domain were discovered which call into question common assumptions based upon metronome-guided tapping, and a number of issues to be pursued in future studies were delineated.

Despite the fact that cooperative synchrony is a central feature of common human activities like music, dance and drill, it remains largely unexplored in the laboratory. This lent a pioneering aspect to our studies, and made it necessary to explore a number of ancillary issues in order to make headway in the analysis of cooperative synchrony. For example, contrary to what is the case in the usual laboratory task of metronome-guided tapping, there is no absolute temporal standard or time giver in cooperative synchrony. Since both performers produce response sequences with a variety of deviations from isochrony, the measure of ?synchronization error? is no longer as readily defined as in the standard laboratory tapping task. We therefore felt a need to study participants? performance to sequences that were irregular (like those of another human performer), but were so in a controlled manner for ease of analysis and comparison with the results of cooperative sequences in which both performers vary idiosyncratically. To this end we adopted sequences into which we introduced pseudo-random perturbations on a continuous mathematically controlled basis. To our initial surprise, participants showed motor reactions to these pertrubations even when they were decidedly below the threshold of conscious awareness. This serendipitous discovery, which two other labs along with our own have recently reported, turned out to be of such basic interest that we subjected it to thorough study. The results were reported in Neuroscience Letters in 2004 and in two forthcoming articles, and are described in more detail in the section 2, below.
Because both participants in cooperative tapping produce sequences with temporal varianbility, the maintenance of synchrony occasionally requires a participant to deliberately accelerate or decelerate his tapping in order to avoid a collapse of the cooperative sequence. Since such strategies may look similar to episodes of spontaneous drift, we felt a need to characterize the nature of spontaneous and stimulus-driven tempo change, another poorly documented area of human timing behavior. These studies are described in greater detail in section 1, below. The total extent of empirical data collection in the project as a whole is summarised Table 1, in terms of the number of participants, data points, session time, and so forth. Each row represents one experimental design, that may be part of one study, or used in one or several studies.

Table 1. Overview of data gathering. Expt. refers to a specific experimental design.
Expt. Participants Sessions Trials Data/trial Data Duration (hrs)
coop 2 8 40,000 12
dprod 14 1 9 ~300 38,170 7
sync1 42 1 35 181,440 35
sioi1 20 1 44 50,800 17
sioi2 20 1 42 50,000 20
sioi3 20 2 41 108,000 27
sioi4 8 2 22 24,640 16
syncd7 22 2 17 79,200 15
syncd8 21 1 28 86,310 16
breakp 10 10 18 113 407,000 100
Sum 165 1,065,560 265

On the basis of the results of these studies, we have so far submitted nine articles for publication, four of which have already appeared in print, while the remaining five are under review. Another three articles are in preparation, that is, all data are collected and analysed, and most of the text has been written. With those included, all of the project?s principal findings are being reported in peer-reviewed journals. That does not mean that further publications are excluded. For example, our pseudo-random perturbation tracking data warrant a more detailed analysis, and may lead to yet another paper. The time may also be ripe for summarizing the perspective on timing mechanisms generated by our recent studies in the form of one or two review articles.

1. Continuous tempo change in stimulus sequences and synchronisation
In the first study (Madison & Merker, 2005), musicians and nonmusicians synchronised drum-beating movements to sound sequences composed of 90 successive inter stimulus intervals (ISI) that increased or decreased by a constant (linear change). This constant was 0.5, 1, 2, 3, or 4 ms. Participants were instructed to continue to produce beats without interruption after the end of a sequence (and were stopped when 30 had been registered), but without a specification of how to continue: In particular for the clearly detectable levels of change, participants had in principle several options, for example to produce isochronous intervals or try to extrapolate the preceding change. Synchronisation appeared both smooth and accurate for all levels of changing tempo. However, actions systematically preceded sounds with increasing intervals and lagged behind sounds with decreasing intervals by about 10 ms on average, indicating that the stimulus change was not fully predicted. There was a tendency for the continuation intervals to reverse direction compared to the preceding synchronisation intervals, suggesting that the system retains information about the tempo change. We had considered the possibility that participants would spontaneously continue the stimulus change during their production sequence after the stimuli, but this did not happen. Rather, some of the largest amounts of change were continued by some participants, but not at all very precisely. Another unpredicted result was a tendency for continuation intervals to show a brief ?rebound?, that is, decrease somewhat following increasing sequences and vice versa, in particular for small and presumably subliminal rates of change. This might reflect an averaging of the last few inter stimulus intervals, expressed in the responses once the stimulus cease and do no longer drive sensorimotor tracking. It might also reflect a contrast effect between the ceasing stimuli - and their continually changing intervals - and an internal representation of change, in which case change is actually represented by the underlying mechanism.
This possibility was addressed by the second study (Madison & Merker, 2004c) in which participants were instructed either to produce isochronous intervals or to extrapolate the change after the stimuli ceased. In the first case, intervals again ?rebounded? by a small amount, while participants were on average quite successful in extrapolating the preceding change. This was, however, strongly asymmetrical: on average approximately about 50 percent for increasing intervals and 150 percent for decreasing intervals across all levels of change.

Figure 1. Graphical representation of the type of stimulus sequences used in Madison and Merker (2005b). Examples show increasing and decreasing intervals and linear and geometrical change, all with 1.33 percent change per interval). ISI is on the ordinate and position in the sequence on the abscissa.
These results confirmed the human timing system?s ability to represent change of intervals, but continuation (production) is a special case of behaviour affected by a number of largely unknown parameters. The third study (Madison & Merker, 2005b) therefore recast the change-no change nature of the stimulus into a pure synchronisation task. An elaborate pattern of interval change, depicted in Figure 1, provided the additional advantage of obtaining steady-state synchronisation performance to isochronous intervals before sequences of changing intervals. Again, effects of change of preceding intervals were found, this time in terms of the direction of the asynchronies, which during subsequent isochronous intervals switched compared with the preceding changing intervals.
In conclusion, these studies reveal a number of novel characteristics of human synchronisation behaviour. These characteristics are inconsistent with, and challenge extant models of SMS (Hary & Moore, 1987; Large & Palmer, 2002; Mates, 1994a, 1994b, Pressing & Jolley-Rogers, 1997; Schulze, 1992; Semjen, Schulze, & Vorberg, 2000; Vorberg & Wing, 1996; Vos & Helsper, 1992).

2. Pseudo-random sequences
In the first study (Madison & Merker, 2004a), we found that pseudo-random perturbations are copied in the response interval following the stimulus interval, even for as small perturbations as 3 ms. Sufficient statistical power to obtain this result, given the high level of noise in human tapping data, was acheived in part by adopting a deterministic deviation pattern according to the procedure of Kolakoski (1966). Since 3 ms is much smaller than the detection threshold for an interval change, this result offers a powerful window on the level of the timing mechanism with highest temporal resolution, i.e. its most detailed level. Varying the magnitude of stimulus perturbations across a wide range from 1.5 to 96 ms (0.25-16% of the stimulus interval), we found that the responses reflected approximately 70 percent of the stimulus perturbations when they were smaller than about 10 percent of the interval, and that this proportion decreased for larger perturbations. Since 10 percent approximately coincides with the detection threshold, it seems likely that participants perform worse when they can consciously detect the perturbations and try to predict future ones. However, to determine this, we let people rate the detectability of perturbations for each stimulus sequence in subsequent experiments.

In Madison (2004b) a wider range of perturbations was applied across a series of experiments in order to find a possible upper limit. Since performance typically exhibits a non-continuous function of the interval, a wide range of intervals was also applied, again in order to explore possible limits. Preliminary results show that the reactions disappear for perturbation exceeding 20-25 percent of the interval, but that they persist for the entire range of intervals used from 154-2236 ms. Further analysis will be directed to studying the detailed behaviour in terms of response interval, asynchronies, and variability, for all different unique positions in the Kolakoski pattern.

3. Deliberate tempo change in produced sequences
Another aspect of natural variability is deliberate change imposed by the human performer. Participants were instructed to start beating isochronously and then to ?change the rate as slowly as possible?, and this was done both from high rates to low rates and vice versa. Preliminary results show that the resulting sequences are remarkably smooth, and exhibit a surprising continuity in the changing rate of tapping (Madison & Merker, in preparation b). Considerable modelling efforts have so far failed to reveal any clear-cut mathematical function that might describe this change across intervals and participants, and the study has accordingly been assigned a low priority pending more results from our sensorimotor synchronisation studies with continuous interval change.

4. Co-operative isochronous sequence production
Natural variability in human interval production is a mixture of interval-to-interval variation, often negatively correlated, irregular trends in the mean interval, and purely random deviations (Madison, 2000; 2001; 2004a, in press). These independent sources of noise make the humanly produced "beat" far less predictable than that of a metronome, yet this is the stimulus with which humans typically synchronise their behaviour in making music, in dancing, and in other co-operative rhythmic activities. We exploited the ecological validity of this variability in an experimental design contrasting sequences produced individually with sequences produced in co-operation, that is, when two participants can hear each other and maintain a beat together. As already described, the complexity of these data necessitated a number of ancillary studies to aid in the interpretation of the cooperative tapping results. They also required us to develop appropriate methods of analysis and data display, a time-consuming effort.

One of the methods that has proved useful is to compute crosscorrelations for a sliding window of the two produced interval sequences, after filtering out any errors and extreme intervals. The crosscorrelations were significantly positive for lags 1 and ?1 most of the time, but were close to zero for all other lags. Since human reactions are found in the interval following the deviation, this indicates that both participants continually reacted to deviations imposed by the other, although to an extent that varies over time. Another observation is that the mean asynchrony varied in an apparently unsystematic way: Sometimes one person lagged behind the other most of the time, on the order of 5-20 ms, and sometimes they ?took turns? in an irregular and non- periodic fashion. Interestingly, these asynchronies appear to be unrelated to the crosscorrelations, which suggests that the reactions are based on intervals, not on asynchronies. Finally, the fractional correlation measured within individual sequences was smaller when two people were involved than when produced by a single person. In other words, two persons acting co-operatively produce more stable sequences than they can do on their own (Madison & Merker, 2005c). This result is by no means an obvious or given one, considering the variability encumbering human sequence production. We had, however, predicted as much on the basis of intuitions developed in the course of our own experience as musicians, and were gratified to find them confirmed by our empirical results.

5. Sequence production and duration-specificity
As mentioned above, several studies have been directed to describing duration-specificity in produced sequences (Madison, 2001, 2004a, 2005) and in the detection of temporal deviations (Madison, 2004b). However, it has not been possible to determine precise breakpoints because the sampling of levels of inter response interval (IRI) has been too sparse. Because this issue turned out to be critical for our interpretation of the results from pseudo-random perturbations, deliberate tempo change, and co-operative sequence production, it was addressed by a separate study (Madison, in press).

Previous studies have indicated that serial dependency, presumably best captured in terms of the scaling factor of fractional Gaussian noise (fGn), is the most sensitive measure of the differences in process that occur as the IRI changes (Madison, 2004a). One consequence of long range correlation is that the IRIs can eventually drift away from the initial ones entrained by the ISIs, but instead of constituting a nuisance variable, this random variability in mean IRI was exploited to obtain a close to continuous sampling of mean IRI. Based on the SD and IRI from this previous study (ibid.), a series of ISI (500-1624 ms) with geometric progression was chosen such that an almost continuous sampling of mean IRI was be obtained. Thus, using mean IRI rather than ISI as the independent variable, the scaling factor H of fGn increased with IRI in a non-linear fashion, and exhibited breakpoints close to 800 and 1000 ms. These results will be applied in future more detailed analyses of pseudo-random perturbations, deliberate tempo change, and co-operative sequence production.

To summarise, the two years covered by this project have allowed us to address a set of central empirical issues involving temporal factors in human timing and co-ordination. We have examined the role of isochrony, the musical pulse derived from it, and various deviations from isochrony in human synchronising performance to both artificial stimuli and the ?live? sequences generated by another human performer. The results of these studies have already started appearing in the technical journals of the field, and further publications are under review. It is with satisfaction, therefore, that we look back upon this productive period as we continue to explore through experiments and modelling the nature of human timing performance.

Articles published or in press
Madison, G. (in press). Duration-specificity in the long range correlation of human serial interval production. Physica D.
Madison, G., & Merker, B. (2005a). Timing of action during and after synchronization with linearly changing intervals. Music Perception, 22, 441-459.
Madison, G., & Merker, B. (2004a). Human sensorimotor tracking of continuous subliminal deviations from isochrony. Neuroscience Letters, 370, 69-73. doi:10.1016/j.neulet.2004.07.094.
Madison, G. (2003a). Perception of jazz and other groove-based music as a function of tempo. In R. Kopiez, A. Lehmann, I. Wolther, and C. Wolf (Eds.), Proceedings of the 5th triennial ESCOM conference, 365-367.
Articles in review
Madison, G., & Merker, B. (2005b). Synchronization with alternating sequences of isochronous and monotonically changing intervals.
Madison, G. (2004b). Impact of duration and inter onset interval on the reproduction of pseudoregular auditory patterns.
Madison, G., & Merker, B. (2004c). Timing of action during and after synchronization with geometrically changing intervals.
Madison, G. (2005). Duration-specificity in isochronous serial interval production: differences between local variability, drift, and fractal correlation.
Madison, G., & Merker, B. (2005c). The cooperative beat - sensorimotor synchronization between individuals. Precision, mutual dependency, and serial dependency.
Articles in preparation
Madison, G. (a). Sensorimotor tracking of pseudo-random perturbations across a wide range of temporal intervals and deviations.
Madison, G., & Merker, B. (b). Dynamics of deliberate tempo change. Properties of self-induced continuous increase and decrease of successive time intervals.
Madison, G. (c). Variability in serial interval production is negatively correlated with variability in sensorimotor synchronization: Implications for mechanism.
Conference contributions
Madison, G. Perception of jazz and other groove-based music as a function of tempo. Oral presentation at the 5th triennial ESCOM conference, Hanover, Germany, September 8-13, 2003.
Madison, G., & Merker, B. Consistency in listeners? ratings as a function of listening time. Oral presentation at the Stockholm music acoustics conference, Stockholm, Sweden, August 6-9, 2003. 
Madison, G., & Merker, B. Entrainment to temporal drift? Oral presentation at Rhythm perception and production workshop, Ile de Tatihou, France,  June 21-25, 2003.
References
Hary, D. & Moore, G. P. (1987). Synchronizing human movement with an external clock source. Biological Cybernetics, 56, 305-311.
Kolakoski, W. (1966). Problem 5304. American Mathematics Monthly, 73, 681-682.
Large, E. W. & Palmer, C. (2002). Perceiving temporal regularity in music. Cognitive Science, 26, 1-37.
Madison, G. (2000). On the nature of variability in isochronous serial interval production. In P. Desain & L. Windsor (Eds.), Rhythm perception and production (pp. 95-113). Lisse: Swets & Zeitlinger.
Madison, G. (2001). Variability in isochronous tapping: higher-order dependencies as a function of inter tap interval.  Journal of Experimental Psychology: Human Perception and Performance, 27, 411-422.
Madison, G., & Merker, B. (2002). On the limits of anisochrony in pulse attribution. Psychological Research, 66, 201-207. DOI: 10.1007/s00426-001-0085-y
Madison, G. (2004a). Fractal modelling of isochronous serial interval production. Biological Cybernetics, 90, 105-112. DOI: 10.1007/s00422-003-0453-3
Madison, G. (2004b). Detection of linear temporal drift in sound sequences: principles and empirical evaluation. Acta Psychologica, 117, 95-118. DOI:10.1016/j.actpsy.2004.05.004
Mates, J. (1994a). A model of synchronization of motor acts to a stimulus sequence: I. Timing and error corrections. Biological Cybernetics, 70, 463-473.
Mates, J. (1994b). A model of synchronization of motor acts to a stimulus sequence: II. Stability analysis, error estimation and simulations. Biological Cybernetics, 70, 475-484.
Pressing, J. & Jolley-Rogers, G. (1997). Spectral properties of human cognition and skill. Biological Cybernetics, 76, 339-347.
Schulze, H. H. (1992). The error correction model for the tracking of a random metronome: Statistical properties and an empirical test. In F.Macar, V. Pouthas, & W. J. Friedman (Eds.), Time, action and cognition (pp. 275-286). Dordrecht, the Netherlands: Kluwer.
Semjen, A., Schulze, H. H., & Vorberg, D. (2000). Timing precision in continuation and synchronization tapping. Psychological Research, 63, 137-147.
Vorberg, D. & Wing, A. M. (1996). Modeling variability and dependence in timing. In H.Heuer & S. W. Keele (Eds.), Handbook of perception and action. Vol. 3. Motor skills (pp. 181-262). London: Academic Press.
Vos, P. G. & Helsper, E. L. (1992). Tracking simple rhythms: On-beat versus off-beat performance. In F.Macar, V. Pouthas, & W. J. Friedman (Eds.), Time, action and cognition (pp. 287-299). Dordrecht, the Netherlands: Kluwer.