Modesty isn’t (on one hand) wearing body-hiding, beauty-suppressing clothing to prevent even the possibility of someone having any kind of attraction. Nor, of course, is modesty (on the other hand) wearing body-flaunting, beauty-distorting clothing to ensure every possibility of every kind of attraction.
Modesty and being attractive are not at all at odds with each other. Proper attraction between two people happens through attractiveness of various kinds – i.e. an attractive personality and attractive attire. Being attractive in the truest sense is to act, speak, behave and dress in a way that fosters healthy mutuality and relationship with an other.
Prudism is unattractive in the passive/negative sense, because it witholds the whole person from an other. Exhibitionism (used here to refer to the opposite of prudism) is also unattractive in the active/positive sense, because it forces too much of the person onto an other.
Therefore, the problem with immodesty (whether the immodesty of prudism, or the immodesty of exhibitionism) is not that it is too attractive, but that it is not attractive enough.