There is a setting in the BIOS for the GPU reserved RAM.The manual should be telling that.
For the moment i can t take a look in my Gigabyte FM2 BIOS to tell you exactly, but it is there ,a little hidden though
You should be able to change between 512megs ,1G and 2 Gigs.
I would choose the last 2.Do some benchmarking with 3D Mark 11 and see if anything changes between the choices.
So now you have 12 Gigs of RAM !?
This means you are using 3 sticks ,which is not good for the memory controller and performance.
Use 2 or 4 sticks and pair them such way that each set /pair is in the reserved slots.This allows the usage of dual channel.One stick or 3 of them will not allow the controller to run dual channel.
Look in the manual for the recommendation on where to install each KIT/pair.
The kits should be alternatively installed.One stick from a KIT in channel A ,the other one in channel B ,do the same with the other KIT.
Set voltage of DDR at 1,6V or 1,65V in BIOS and try again if they don t seem to work at 1600Mhz.
Oh ,and change manually the Command Rate DDR timing in BIOS to 2T ,for some stupid reason most BIOS-es go for the 1T setting by default ,that doesn t run properly with AMD ,at least on higher than 1333Mhz speeds.
So set voltage and that Command Rate DRAM Timing manually before switching to 1600Mhz.