Jacob Levy proposes a reconciliation of sorts:

When I first started researching freedom of association, I was annoyed that so much of the literature was dominated by the case of unions, since that wasn’t a case I was interested in.  I now think that was the wrong reaction, and that I should take the centrality of unions to fights for freedom of association as indicative.  There is widespread revealed-preference demand for associations of something like this form, and as a libertarian I should pay attention to that, not sweep it under the carpet.