Part I. A Tutorial Introduction

Sections

• Introduction.
• The Execution Model. Introduces i-pass, k-pass, and a-pass. Shows how SAOL and SASL work together.
• A Simple Example. Fixed-frequency sine wave oscillator. Introduces instr structure.
• Running the Example. Shows how to use sfront to generate audio and MP4-SA files. Computes compression ratio for the first example.
• Playing a Melody. Second example. Plays a melody with the sine-wave oscillator. Shows how to combine i-pass, k-pass, and a-pass computations.
• In Reverberant Stereo. Third example. Shows how to use send and route commands to make effects pipelines. Shows how to change SAOL variables from a SASL score line.

Introduction

MPEG-4 Structured Audio (MP4-SA) is an ISO/IEC standard (edited by Eric Scheirer) that specifies sound not as audio data, but as a computer program that generates audio when run.

This tutorial introduction shows how to write programs using the MP4-SA programming language SAOL (pronounced sail) and the MP4-SA score language SASL (pronounced sassil) to create audio content. We start with a simple programming example, and add more features to introduce different aspects of SAOL and SASL.

At the end of the tutorial, the reader should have a broad overview of the MP4-SA system. The rest of the book will revisit the issues introduced in this tutorial, and show in detail how MP4-SA works.

As you might have noticed, this tutorial is a bit lengthy; if you just want a quick look at MPEG-SA in action, you might want to look at this simple example instead.

The Execution Model

When we hear a sound, we are sensing fluctuations of air pressure with time. The temporal structure of the pressure waveform has a significant effect on whether or not we find a sound pleasing or interesting.

This structure happens on several different time scales. Notes in a musical composition change on a time scale of hundreds of milliseconds. The timbre of sounds changes on a time scale of tens of milliseconds. The actual sound waveform changes on a time scale of tens of microseconds.

When programmers write software to create audio content in a conventional language like C or Java, much of the effort focuses on managing these three time scales of sound. Since the code is structured around timing issues, it is difficult to group all the code for modeling a specific musical instrument in one place. Instead, the code becomes separated into different sections for different time scales, and the methods of communicating between the time scales become hard to track and modify.

In MP4-SA, time management is a part of the language itself. A SAOL program executes by moving a simulated clock forward in time, performing calculations along the way in a synchronous fashion. For interactive applications (for example, musicians playing MIDI controllers that send data to a SAOL program) this simulated clock is in sync with real time. For off-line applications, like converting a MP4-SA bitstream into an audio file for later playback, the simulated clock is unrelated to real time.

The clock is set to 0 at the start of program execution, and moves forward at the audio sample rate (called the a-rate). In addition to the a-rate, there is a slower time scale for timbre changes and note sequencing, called the control rate or k-rate.

The a-rate and k-rate values may not be varied during execution, and the a-rate must be evenly divisible by k-rate. By default, the a-rate is 32 kHz and the k-rate is 100 Hz. We use the terms a-cycle and k-cycle to describe the moments in time that repeat with a-rate and k-rate frequency, respectively.

Sound creation in MP4-SA can be compared to a musician playing notes on an instrument. A SAOL subprogram (called an instr -- SAOL and SASL keywords are in bold face in this text) serves as the instrument. SASL commands (called score lines) act to play notes on SAOL instrs. Just as a musician playing a polyphonic instrument can play several notes concurrently, many instances of a SAOL instr can be active at one time, making sounds corresponding to notes launched by different score lines in a SASL file.

A SAOL instr contains all the instructions for playing a note: code that runs at note launch, code that models timbre evolution at the k-rate, and the code to generate audio samples at the a-rate. Because time management is a part of the SAOL language, computation at these different rates can be written compactly in one place in the file, and communication across these time scales can be expressed easily.

The right panel shows the execution trace of a note played on a SAOL instrument, which is part of a SAOL program that has an audio rate of 40kHz and a control rate of 100 Hz. This note is scheduled to start at a simulated clock time of 0.995 second. However, since notes are launched in SAOL at k-rate granularity, the note launch is delayed to 1.0 seconds.

At 1.0 seconds, instance execution begins with the i-pass, where instr code for note launch executes. Following the i-pass is the first k-pass, where instr code for timbral evolution executes. Following the first k-pass is the first a-pass, where instr code to generate an audio sample runs.

After the first a-pass, simulated time advances by the audio sample period (the inverse of the audio sample rate) and another a-pass is executed. Execution continues in this fashion until  a-rate/k-rate  a-passes have run.

At this point, the first execution cycle of the instance is complete. The subsequent execution cycles consist of a k-pass followed by a set of a-passes. The i-pass code only runs during the first execution cycle.

Note that the analogy of a musician (SAOL) playing notes (SASL) only loosely applies to MP4-SA. In practice, not all SAOL instrs are used for instrument modeling, and instrument models don't always map one-to-one onto instrs in a SAOL program. In addition, there are other ways to trigger notes in MP4-SA besides SASL score lines, including MIDI files and commands run in other SAOL instrs.

Instance Execution Trace

Audio rate   : 40,000 Hz
Control rate :    100 Hz
Start time   :  0.995 s 


Time (s)   Cycle     Pass     X-# 


0.990000   k-cycle
           a-cycle
0.990025   a-cycle
0.990050   a-cycle
0.990075   a-cycle
  ...

0.999925   a-cycle
0.999950   a-cycle
0.999975   a-cycle
                            
1.000000   k-cycle   i-pass  -- 
                     k-pass    |  
           a-cycle   a-pass    |  
1.000025   a-cycle   a-pass    | 1  
1.000050   a-cycle   a-pass    |    
1.000075   a-cycle   a-pass    |  
   ...       ...       ...     |   
1.009975   a-cycle   a-pass  -- 
                            

                            
1.010000   k-cycle    k-pass --    
           a-cycle    a-pass   |  
1.010025   a-cycle    a-pass   |
1.010050   a-cycle    a-pass   | 2
1.010075   a-cycle    a-pass   |
   ...       ...       ...     |   
1.019975   a-cycle    a-pass --
                         


1.020000   k-cycle    k-pass --    
           a-cycle    a-pass   |  
1.020025   a-cycle    a-pass   |
1.020050   a-cycle    a-pass   | 3
1.020075   a-cycle    a-pass   |
   ...       ...       ...     |   
1.029975   a-cycle    a-pass --
                         


1.030000   k-cycle    k-pass --    
           a-cycle    a-pass   |  
1.030025   a-cycle    a-pass   |
1.030050   a-cycle    a-pass   | 4
1.030075   a-cycle    a-pass   |
   ...       ...       ...     |   
1.039975   a-cycle    a-pass --


X-# column indicates the passes
that make up each execution cycle
of the instance.

A Simple Example

Our first SAOL instr is a simple sine wave oscillator. Our first SASL score triggers this oscillator.

The panel on the right shows this SASL score. It is a list of commands that trigger actions relative to the clock time. These times are expressed as 32-bit floating point numbers, and may use the full resolution of the representation. However, SASL commands are processed at k-rate granularity. At each k-cycle all unexecuted commands whose times are less than or equal to the current clock time are executed.

SASL commands are one line long; the newline marks the end of the command. The first number on the line indicates the trigger time. However, this time is not in units of seconds of time; instead, it is measured in beats of score time. A global tempo value is used to convert beats to seconds.

By default, the global tempo is 60 beats per minute, or 1 beat per second. In the first example in this tutorial, we use this default global tempo, so that all SASL times can be thought of as beats or seconds interchangeably.

We now look at the SASL score on the right panel in detail. The first line of the file is an instr command that triggers the SAOL sine-wave oscillator instrument. The first field (0.25) indicates the start time for the instrument. The second field (tone) indicates the name of the SAOL instr to trigger. The final field (4.0) is the duration that tone should play, in units of beats (and in this example, seconds).

The final line in the SASL file is an end command. It signals the simulated clock time to end program execution via its trigger time (4.50 seconds).

The right panel shows the SAOL file for the instr tone, a fixed-frequency sine-wave generator. The file begins with several comment lines. SAOL comments begin with the characters // and end with a newline; characters inside comments are ignored by MP4-SA tools.

The keyword instr starts the definition, followed by the name of the instrument (tone), and an empty pair of parenthesis. As in C function syntax, curly brackets enclose the body of the definition, which consists of a variable declaration block followed by a code block.

Three variables (x, y and init) are declared as type asig, one of the three signal variable types (asig, ksig, and ivar). Signal variables, the only type of variables that can be used in SAOL expressions, hold 32-bit floating point numbers, and are initialized to 0 at the time a SASL command instantiates an instrument. The asig type indicates that computation involving these variables runs during the a-cycle.

Next, we reach the code block. The syntax of this code seems quite familiar to C programmers: a simple assignment statement, followed by an if statement, followed by two more assignment statements, and finally what appears to be a call to an undefined function output().

What may be surprising, in light of the earlier discussion on SAOL's execution model, is that the statements in the code block aren't marked to indicate the rate of computation (a-rate, k-rate, or i-rate). In SAOL, the time scale of statements is inferred from the time scale of variables used in the statement. As we shall see as we analyze each statement, each statement in this code block runs at a-rate.

If we accept that each statement executes at a-rate, we can visualize the combined behavior of the SAOL and SASL code (see the SASL code reproduced in the panel to the right). Starting at the first k-cycle after 0.25s, the code block is executed on every a-pass. In this example, the a-pass happens 32,000 times per second (the default audio rate value). The state of variables x, y and init is maintained between a-passes. Execution stops when the first k-cycle after 4.25s occurs (4.25s is the sum of the start time (0.25s) and the duration 4.0s), at which time this instance of the instr tone is destroyed.

We now return to the SAOL file, and analyze each statement in the code block in the instr tone (reproduced in the panel on the right). Recall that this instrument generates a fixed-frequency sine wave.

The first statement is a simple assignment statement, setting a constant value used in the algorithm. Assignment statements execute at the rate of the variable on the left hand side of the statement; in this case, the variable a is an a-rate variable, and so the statement runs at a-rate. Note that operator = is the only assignment operator in SAOL; C assignment operators such as += and ++ are not a part of the language.

The assignment is followed by an if statement. Like its C equivalent, the code block (enclosed by curly brackets) executes if the value of the guard expression (enclosed by parenthesis) has a non-zero value. In this case, the code block is executed exactly once; the variable init is initialized to zero during instrument instantiation, and is set to a non-zero value in the code block of the if statement. The purpose of this if statement is to initialize x.

The statements in the code block of the if statement are a-rate, and as a result, the if statement also runs at a-rate.

The next two assignment statements do the real work of the instrument; these statements implement a classic iterative algorithm for sine and cosine generation, that is guaranteed to be stable. The value of the variable a sets the frequency of the waveforms (in this example, the value of a maps to a frequency of 1000 Hz). The initial value of x sets the amplitude of the waveform. Since x and y are both a-rate, both statements execute at the a-rate.

The final statement is an output statement, part of a class of SAOL statements that use the syntax of C functions. The output statement is defined as a-rate, and serves as the mechanism to communicate audio samples outside of an instr. Audio in SAOL is read and written to special 32-bit floating-point variables called buses, which are initialized to zero at the start of each a-pass.

SAOL has a special bus, output_bus, that represents the final audio output of an MP4-SA system. In this example, the output statement adds data onto the output_bus (the default behavior in SAOL). In this case, it adds a sample of the sine wave, which will vary in value from -0.5 to +0.5 (since x is set to 0.5 in the if statement during the first a-pass).

After an a-pass is complete, the output_bus is clipped to the range -1.0 to +1.0, and sent to the audio output device (usually a file or a sound card). If no instrs are running during an a-pass, the initial zero value of the output_bus is sent to the audio output device, resulting in silence. In this example, a silent sections happen before the instr tone is triggered and after the instr tone is destroyed.

Running the Example

In this section we show how to use the SAOL to C translation program sfront to run the example above. We assume a UNIX command-line shell environment.

The process begins by running sfront, using the syntax shown on the right panel. The -orc option is used to specify the SAOL file, and the -sco option is used to specify the SASL file.

By default, sfront creates a file sa.c, a C program that, if compiled and executed, produces the audio output specified by the SAOL and SASL code. We direct the output to the audio file output.wav by using the -aout option.

Finally, since MP4-SA is a compression format, we would also like to create a compact version of the ASCII SAOL and SASL files, for efficient transmission and storage of the example. We create the MP4-SA binary file sine.mp4 using the -bitout option.

We show the files created by sfront on the right panel. The sine.mp4 doesn't have a link, since as a binary file it may crash certain Web browsers if displayed as an ASCII file. Note that sine.mp4 is 143 bytes in length, but encodes both sine.saol (286 bytes) and sine.sasl (26 bytes) within it.

Next, we compile the generated sa.c file, and run the executable sa. This produces the audio file output.wav. If you are connected to the Internet, and if your Web browser is set up to handle audio files, you can click on the link to listen to it.

Finally, by dividing the size of the output.wav file (288044 bytes) by the size of the sine.mp4 file (143 bytes), we compute effectiveness of MP4-SA as a lossless compression format. This calculation yields a compression ratio of 2014 -- about 100 times better than a perceptual encoder like MPEG 2 Level 3 (MP3) would do on this example!

Running sfront:

sfront -orc sine.saol -sco sine.sasl\
-bitout sine.mp4 -aout output.wav

Creates the files:

sa.c [30636 bytes]

sine.mp4 [143 bytes]




Compiling and executing sa.c:

gcc -O2 sa.c -lm -o sa ; ./sa



Creates the file:

output.wav [288044 bytes, a Web link]

Playing a Melody

In this section, we rewrite the SAOL and SASL programs to play a melody with the sine-wave oscillator. We also upgrade the audio output to DAT quality, and eliminate the annoying click at the start of each note. To implement these improvements, we take full advantage of SAOL's ability to express computation that runs at different time scales.

The right panel shows the new SAOL program. The file begins with a global block, an optional part of a SAOL program that controls the execution environment of all SAOL instruments.

In this global block, we initialize two SAOL system parameters, srate (the audio sampling rate, or a-rate) and krate (the control rate, or k-rate). Note that parameter initializations do not include an = operator.

The remainder of the file holds the instr vtone, an enhanced version of the sine-wave oscillator instr tone from the previous example.

The first change is evident in the instr preamble: vtone has the parameter num. An instr parameter is an i-rate variable. When an instr instance is created by a SASL instr command, its parameters are set to initial values provided on the SASL score line.

In this example, the parameter num sets the pitch of the note, using MIDI note numbering (integers from 0 to 127 that map to notes on the piano keyboard, with 57 mapping to A below middle C).

The main body of vtone begins with declarations for 6 ivar variables (for i-pass computations), 1 ksig variable (for k-pass computations), and 3 asig variables (for a-pass computations).

In this example, comment banners partition the code block for vtone into separate sections for i-pass, k-pass, and a-pass code. This convention greatly improves the maintainability of SAOL programs. We now look at each section in turn, to introduce the interaction between time scales in SAOL programs.

The i-pass section of an instr executes as soon as a new instrument instance is created. During vtone's i-pass, assignment statements set the values of the 6 ivar variables. In the rest of the code block (k-pass and a-pass sections) these ivars will appear on the right-hand side of k-rate and a-rate assignment statements. In this way, an i-pass serves to pre-compute values once that may be used millions of times.

We now look at the i-rate assignment statements in detail. The first statement computes the value of the oscillator frequency control variable a. This statement introduces SAOL opcodes which are similar to C functions. The simple opcodes used here (the trigonometric function sin and the conversion routine cpsmidi) are part of the SAOL core opcode library. The opcode cpsmidi converts a MIDI note number into a frequency (in Hz).

This statement also introduces SAOL standard names. Standard names are read-only variables that convey status information. The s_rate standard name used in this example holds the value of the a-rate (defined in the global block as 48 kHz).

The remaining assignment statements in the i-pass block specify characteristics of the amplitude envelope that shapes the sine wave. The first two statements set default attack and release times for the envelope. The remaining code handles the case where the duration of the note is too short for these defaults; it uses the dur standard name, that holds the duration of the note (in seconds, not beats).

The k-pass section of this example generates the envelope waveform. Since the envelope changes slowly in time relative to the audio sampling rate, we compute an envelope value during the k-pass, and apply it to samples generated in the following set of a-passes. To avoid the "zipper noise" that may happen if amplitude changes abruptly, we use a fast k-rate of 417 microseconds (set in the global block).

To compute the envelope samples, we use core opcode kline, which is specialized for piece-wise linear waveshapes. The parameters of the kline opcode alternate between the end-point values of the linear envelope (0 and 1 in this example) and the durations of the linear segments (computed during the i-pass).

Finally, the a-pass section computes the audio samples of the sine wave. Apart from scaling the final output by the envelope, this code is unchanged from the tone instr in the first example.

The right panel shows the SASL file that drives the SAOL file above. This example introduces the SASL tempo command, which changes the mapping between score beats and simulated clock time. A tempo score line starts with trigger time, followed by the keyword tempo, and the new tempo value (in units of beats per second).

Next is a set of SASL instr commands that play a simple melody. On each score line, the start time, SAOL instr name, and duration fields are followed by the initial value for the vtone parameter num.

Note that the trigger time of the first instr command precedes the trigger time of the last tempo command. SASL score lines do not need to be ordered with respect to trigger times.

If you are connected to the Internet, you can listen to the audio output produced by this example by playing the file output.wav.

0 tempo 110
2 tempo 112.1
4 tempo 114
6 tempo 116
8 tempo 118

1 vtone   1.5 52 
3 vtone   1.5 64 
5 vtone   1   63 
6 vtone   0.5 59 
6.5 vtone 0.5 61 
7 vtone   1   63 
8 vtone   1   64 

10 end

[505,524 bytes, on the Web]

In Reverberant Stereo

In this section we enhance the previous example by adding reverberation and dynamic stereo panning to the melody line. We also add legato phrasing to the melody. In implementing these improvements, we show how to use two instrs together in a SAOL program. We introduce SAOL arrays, global variables, and user-defined buses, as well as SASL control statements.

The Global Block

The right panel shows the SAOL program. In the global block, we declare the global signal variable bal, that is used for dynamic stereo panning. Global variables in SAOL can be read and written by instr instances, and updated by SASL control commands. Global variables may be declared ivar or ksig but not asig.

This example uses two instrs. A new version of the sine-wave generator instrument (vctone) produces notes, and an effects instrument (mixer) places them in the stereo field and adds room reverberation.

In SAOL, effects instrs are not launched by SASL commands. Instead, instances of effects instrs are created at the start of program execution, along with the buses that route audio into and out of the instances.

Two statements in the global block instantiate the effects instr mixer and set up its audio buses. A route statement specifies that audio from all vctone instances sum onto the audio bus tonebus. A send statement creates an instance of the instr mixer, and sets mixer's two parameters to 2 and 0.25. The send statement also creates the bus tonebus, that serves as the audio input to the instr mixer.

Since no route statement exists using mixer, the audio output for mixer adds to the default output_bus. This output_bus is a stereo bus, because we initialize the SAOL system parameter outchannels to 2 in the final statement of the globals block.

The route and send statements introduce several concepts. The first concept is that buses may be arrays (in fact, in SAOL signal variables may also be declared as arrays). However, note that the send statement that defines the bus tonebus does not include an array width specifier. It is possible to declare the width of a bus in a send statement, but it is not required. If the bus width is unspecified, SAOL infers it from the code in instrs that write to the bus, as we see later in this example.

A second concept introduced by the route and send statements is that instruments instantiated by a send statement have audio input as well as audio output. The standard name input, a read-only array variable, is made available in effects instruments, and holds the current value of audio sent to the instrument.

A final concept concerns the execution order of the instrs vctone and mixer. Note that if mixer instances execute before the vctone instance during the a-pass, the bus tonebus is read before it is written.

To prevent this problem, the SAOL language has a set of rules for execution ordering. One ordering rule states that instruments adding signals to a bus execute before instruments reading the bus, ensuring the correct execution order for simple pipeline structures. For more complex structures, the global statement sequence may be used to override this rule to achieve the desired results.

Note that since all buses are set to zero at the beginning of each a-pass, buses cannot hold audio state between a-passes.

Instrument vctone

The sound-generating instr in this example, vctone, is similar to the sine-wave oscillator instr vtone in the last example. The major enhancement is the support of legato voicing, through the variable bend.

The variable bend is declared as an imports variable. A imports variable may be assigned a new value during a k-cycle in the following ways:

1. If a global variable is declared with the same name as the imports variable, the imports variable is assigned the value of the global variable at the start of each k-pass for the instrument instance.
2. If no global variable has the same name as the imports variable, a SASL labelled control statement may modify the variable at the start of each k-cycle.

In this example, the second rule applies, as we use bend to signal legato voicing changes from the SASL score.

The code that implements legato voicing is found in the k-pass section of the code block. The if statement in the k-pass section checks to see bend has been set to a non-zero value by a SASL line, and if so recalculates the oscillator frequency control variable a to reflect the modified pitch value. This if statement also initializes a to the proper value during the first k-pass.

Finally, note that the output statement in the a-rate section has a single scalar parameter (the expression env*y). As a result, we infer that the width of the bus tonebus that this statement writes has a scalar width. If the statement had been

output(env*y, env*y);

the bus tonebus would be a stereo bus.

Instrument mixer

The instr mixer processes the monaural tonebus audio bus, panning the signal in the stereo field and adding reverberation. It accesses tonebus through the standard name input. It places tonebus in the stereo field under direction of the global variable bal, whose value is changed in the SASL file using control score lines.

The instr mixer accesses the global variable bal through the instr signal variable bal, declared with type imports ksig. At the start of the k-pass for the mixer instance, the current value of the global variable bal is copied into the instr bal, and code in mixer accesses this local copy.

The variable declaration block also defines the signal variable arrays pos and out, introducing SAOL array declaration syntax.

The k-pass section of the code block shows how individual elements of SAOL arrays are addressed using C square-bracket notation. This k-pass code updates the left- and right- channel values for the panning weight array pos.

The a-pass section shows how unindexed arrays may be used in SAOL arithmetic expressions. In this example, the expression on the right-hand side of the assignment statement combines an array of width 2 (pos), an array of width 1 (the standard name input), a scalar (wet), and a core opcode with a scalar return value (reverb), to compute a stereo audio signal.

In this computation, all monaural signals are promoted to stereo arrays before the calculation begins. The stereo result is stored in the array out, and added to the stereo output_bus using an output() statement.

The right panel shows the SASL file that drives the SAOL file above. The file begins with a set of SASL instr lines that play a melody on the instr vctone. Two lines are labelled for later reference. This example introduce the label syntax for SASL: a label name, followed by ":", at the start of the instr command.

Next are two labelled control commands. Labelled control statements start with a trigger time, followed by a label defined earlier in the file and the keyword control. The label is used to direct the control change to a particular instrument instance.

The final two fields of the labelled control commands are the name of variable to change, and the new value of the variable. This variable must be defined, with type modifier imports, in the instr launched by the SASL score line associated with the label.

In this example, the labelled control commands modify the bend variable of vctone instances. K-rate code in vctone senses the change and shifts the frequency of the sine-wave oscillator, implementing a legato transition to a new note number.

This example also introduces SASL unlabelled control statements. A set of 5 control statements alter the k-rate global variable bal, changing the spatial position of the melody in the stereo field. In this encoding 0 is hard left, 0.5 is center, and 1 is hard right. We use unlabelled control statements because the instr mixdown that performs the panning is instantiated with a SAOL send statement, not a SASL instr command, and so there is no instantiation score line to label.

If you are connected to the Internet, you can listen to the output of this SAOL and SASL program pair by playing the file output.wav.

       1   vctone  1.5 52 
       3   vctone  1.5 64 
       5   vctone  1   63   
ver  : 6   vctone  1   59 
rain : 7   vctone  2   63 

6.5 ver control bend 2
8   rain control bend 1

0 tempo 120

1   control bal 0
3   control bal 0.2
5   control bal 0.5
6   control bal 0.8
7   control bal 1


10 end

[960,124 bytes, on the Web]

Conclusions

These three examples have introduced, in simple form, most of the basic techniques of SAOL and SASL programs. Notably absent are commands relating to SAOL tables, which are variables specialized for audio samples, and information on user-defined opcodes. Also missing are the syntax and semantic rules for SAOL, especially in regard to variable rates and widths and bus widths. The rest of the book revisits SAOL and SASL in systematic detail, showing all the options and usage rules.

Next: Part II: The SAOL Language