Compositing is provided by Compiz, which is a compositing window manager. What it means is explained below:
A compositing window manager is a
component of a computer's graphical
user interface that draws windows and
their borders. It also controls how
they display and interact with each
other, and with the rest of the
desktop environment. The main
difference between a compositing
window manager and other window
managers is that instead of outputting
to a common screen, programs each
output first to a separate and
independent buffer, or temporary
location inside the computer, where
they can be manipulated before they
are shown.[1][2]
The window manager then processes and
combines, or composites, output from
these separate buffers onto a common
desktop. The result is that the
programs now behave as independent 2D
or 3D objects.[1] Compositing allows
for advanced visual effects, such as
transparency, fading, scaling,
duplicating, bending and contorting,
shuffling, and redirecting
applications. The addition of a
virtual third dimension allows for
features such as realistic shadows
beneath windows, the appearance of
distance and depth, live thumbnail
versions of windows, and complex
animations, to name just a few.[3][4]
Because the programs draw to the
off-screen buffer, all graphics are
naturally double buffered and thus do
not flicker as they are updated.
The most commonly-used compositing
window managers include the Desktop
Window Manager in Microsoft Windows,
the Quartz Compositor in Mac OS X, and
Compiz, Metacity and KWin for Linux,
FreeBSD and OpenSolaris systems.
http://en.wikipedia.org/wiki/Compositing_window_manager
If you see a black border around docky, then compositing is not properly enabled. The warning you are getting is most likely a result of Docky starting before compositing kicks in.
See https://bugs.launchpad.net/docky/+bug/552273
