>Given my experience with LLMs I am skeptical that enough prompt engineering and tuning can adequately summarize patient records.

What sort of experience do you have though? I'd say anyone familiar with the use of AI specifically as it related to medical records would probably disagree with you.