There is no general theory of nonlinear systems. A nonlinear system
with memory can be quite surprising. In particular, it can emit any
output signal in response to any input signal. For example, it could
replace all music by Beethoven with something by Mozart, etc. That
said, many subclasses of nonlinear filters can be successfully
analyzed:

A nonlinear, memoryless, time-invariant ``black box'' can be ``mapped
out'' by measuring its response to an impulse at each amplitude.

A memoryless nonlinearity followed by an LTI filter can similarly be
characterized by a stack of impulse-responses indexed by amplitude (sometimes
called dynamic convolution).

One often-used tool for nonlinear systems analysis is Volterra series
[4].
A Volterra series expansion represents a nonlinear system as a sum of
iterated convolutions:

Here is the input signal, is the output signal, and the
impulse-response replacements are called Volterra
kernels. The special notation
indicates that the second-order kernel is fundamentally
two-dimensional, meaning that the third term above (the first
nonlinear term) is written out explicitly as

Similarly, the third-order kernel is three-dimensional, in
general. In principle, every nonlinear system can be represented by
its (typically infinite) Volterra series expansion. The method is
most successful when the kernels rapidly approach zero as order
increases.

In the special case for which the Volterra expansion reduces to