Jessica Valenti Flanigan develops a libertarian theory:
[S]exual harassment must be wrong either because a) proposing to pay for sex is wrong or b) radically changing a person’s job description from (e.g. from secretary to prostitute/maid/babysitter) is wrong. That is, the employer’s threat must be to do something that is independently wrong. My intuition is that something like (b) makes sexual harassment wrong. In these extreme cases employees do have the authority to decline certain tasks that employers demand. In this case the employee may say ‘you led me to believe that the job did not require prostitution, so I have been deceived.’ Because it is wrong to deceive people, it is wrong to radically change a person’s job description, and so threatening to fire someone for refusing to comply with an impermissible demand is also impermissible.
My belief is that sexual harassment is wrong because it is an abuse of power. But libertarians have a hard time accepting the potential for abuse of power in the private realm:
Lots of things which we are expected to do in our jobs aren’t explicitly in the contract. Indeed, no employee-employer relationship could function on the basis of strict adherence to contractual terms, which is why "work to rule" is often an effective tactic. And there are lots of jobs where women know, as they agree to work, that there’s a high probability of being asked to have sex by an abusive supervisor or employer… . [Libertarians] can’t and don’t take seriously the realities of private power, of domination, and of the need for someone (the state or the unions or both) to step in and protect people who have nothing against those who believe they are entitled to do what they want on their private domain. In an unequal world, where access to employment is in the hands of the few, then it is certain that at least some of the few will take advantage of their position to abuse and humiliate their subordinates in various ways, including sexually.