alas! this is entirely the opposite of what i was trying to communicate. i apologize for being unclear; communication is not my strong suit. i will try to rephrase and explain more clearly.
given that you do not describe your toxic coworker as being particularly sexist or targeting anyone more than anyone else, your framing of your female coworkers as “victims” and yourself as not-that seems to be entirely due to sexism. this is unfair to them and it is also unfair to you, because you are in this situation with them, not apart from them. if they are victims, so are you. if you feel a sense of responsibility to step up and do something on their behalf, it logically follows that you should also feel a sense of responsibility to step up and do something on your own behalf. you and your colleagues are in the same situation and experiencing the same behaviors! behavior which is unacceptable towards women in the workplace is also unacceptable towards men in the workplace. thinking of men as uniquely resilient and immune to harm because of their gender and women as uniquely weak and susceptible to harm because of their gender is sexist. in this case, it seems to me that sexism is preventing you from considering your colleagues as allies with whom you are sharing a struggle!
just because you are currently dealing with the situation individually does not mean you have to. you and your colleagues could collaborate on potential solutions to the problem that you are all facing together, and in fact i think that you should! abuse of all kinds, including workplace abuse (which this is), thrives on isolation, and worker exploitation thrives where people think of themselves as individuals rather then members of a class with common cause.
i think it is important to distinguish between things which you have to do and things which you feel you have to do; in this case, you absolutely do not have to deal with this alone, because you are not alone. that is what i was trying to communicate (poorly, evidently!) ! i do not imagine it is easy or pleasant to navigate dealing with workplace harassment or initiate collective organizing in a non-unionized workplace with coworkers you don’t especially like, nor do i think there is a guaranteed solution to the problem, mind; just because a hostile work environment exists doesn’t mean it’s legally actionable - even if it’s legally actionable doesn’t mean your bosses care to do anything - etc. but regardless of the particulars, it is nevertheless possible to communicate and collaborate with your coworkers to improve your collective work experience. you could, for example, tell your coworkers that you think the way they interact with your toxic colleague is making things worse; you already know they aren’t being nice to him because they like him or his behavior, because they told you so.
i hope that is more clear!