My problem was the same, i had IPV6 queries made before IPV4 and they where wasting resources, so I did changed my /etc/gai.conf file as freddyb posted, but my scenario is a little bit different, my server is Natted, so IPv4 will not have precedence over IPv6 with only that line. The solution is also commenting out others lines at gai.conf so it will look like this
#For sites which prefer IPv4 connections change the last line to
precedence ::ffff:0:0/96 100
...
# For sites which use site-local IPv4 addresses behind NAT there is
# the problem that even if IPv4 addresses are preferred they do not
# have the same scope and are therefore not sorted first. To change
# this use only these rules:
#
scopev4 ::ffff:169.254.0.0/112 2
scopev4 ::ffff:127.0.0.0/104 2
scopev4 ::ffff:0.0.0.0/96 14
So your server will try first ipv4 even if you are Natted!