imo point blank its like a bottle neck, if you had a 3'' exhaust you would see 3'' exhaust flows, but since you have the 2.5in to 3'' exhaust you are as of right now only seeing the benifits of a 2.5'' exhaust flow.
if you get a larger downpipe mated with the larger exhaust you will then open up the turbos ability to flow greater volumes of exhaust gases in return letting you see the benifits of a larger exhaust.
there are benifits to a smaller exhaust system if you are running say stock or near stock boost levels. however if i had ur turbo i definitly would be near stock numbers, and i would definitly want ot take advantage of the 3'' exhaust i had by upgrading to a 3'' downpipe too.
some would say smaller wiill yeild you better torque numbers, and better spool times, some people go from 3'' dp to 2.5'' exhuast, others do like you did, really its all up to the person at hand as to what they think works best for them, like you have seen you have getten different responses as to what is better and thats tipical as to when you ask for opinions.
if you are pushing greater volumes of air, the air is becoming restricted by the space it has to release the volume in. if you open up that space, then you will equal the ammount of volume and space and let the exhaust volume flow become greater. you may see downfalls to a 3'' dp when you go if you go to a 3'' dp however in the grand scheme of things its all about power and a larger diameter will allow for larger power numbers.