I have the same vague impression, but I'm not sure. I never really went looking for it. But as you said, if there's some polarization, it stays consistent as the ND is dialed up or down.
Definitely, it's one of the dangers of vND. Also with wide angle lenses during the day, the sky can end up looking funky and not just different.