The brother, former Democratic congressman Joe Kennedy, describes Robert F Kennedy as "a threat to every American's health and well-being" in a statement.
"None of us will escape the pain he inflicts."
The Minister of Health's sister Kerry Kennedy says in turn that medical decisions "belong in the hands of trained and licensed professionals, not incompetent and misguided leaders", writes AP.
The call for resignation comes after Kennedy defended his attempts to limit the possibility of covid vaccination and the decision to fire the leadership of the infectious disease control agency CDC in a hearing in the senate this week.
President Donald Trump has since defended Kennedy by saying that he "means well", but at the same time questioned the Minister of Health's line on covid vaccine.
I think you have to be very careful when you say that certain people do not need to be vaccinated, said Trump on Friday and added that vaccines "work" and are not "controversial".
Robert F Kennedy has long been a prominent vaccine skeptic and as minister he has continued to question research and experts.
It is not the first time that family members and relatives criticize him. Voices have been raised against both his presidential campaign last year and his nomination as Minister of Health.
