The widespread adoption of AI medical advice creates a hidden patient safety crisis that healthcare systems haven't adequately addressed. While millions seek medical guidance from chatbots daily, the actual reliability of these responses has remained largely untested until now. A comprehensive physician-led evaluation of four major AI chatbots—Claude, Gemini, GPT-4o, and Llama—reveals alarming disparities in safety when answering 222 real patient medical questions across internal medicine, women's health, and pediatrics. The study found problematic response rates ranging from 21.6% with Claude to 43.2% with Llama, while unsafe responses varied from 5% to 13% depending on the platform. These aren't minor inaccuracies but responses with potential to cause serious patient harm. The evaluation framework, called HealthAdvice, represents the first systematic attempt to quantify AI medical advice safety across multiple platforms using real patient scenarios. This research exposes a critical gap in the digital health ecosystem where patients increasingly turn to AI for medical guidance without understanding the substantial variation in safety between platforms. The findings suggest that current AI chatbots lack the clinical guardrails necessary for safe medical advice, particularly concerning given their widespread accessibility and the trust patients place in their responses. While some platforms perform better than others, none achieved safety levels appropriate for unsupervised medical consultation. This creates an urgent need for enhanced clinical validation, improved safety protocols, and clearer patient education about AI limitations before these tools can be safely integrated into healthcare decision-making.