That's correct, it's called ordinary magisterium. But you forget about the extraordinary magisterium, which states that a doctrine held by all/practically all bishops all over the world is infallible.
Now many bishops had published anti-contraception statements, and those who had not usually did not speak up otherwise. The Papal commission preparing HV used exaclty that argument to show that Paul VI. would contradict infallible doctrine if he took a different position.
Let's make this point a bit clearer:
That little paper wrecked the Catholic church.
No, it cannot be put more tender words. Of course, without it membership, social and political influence, number of priests, and church attendence would still be declining, but on a higher level. It might seem overdone today, but the derision of Papal authority after HV opened the floodgates.