I believe that comments encourage users to think of this site like a forum or Reddit or some other form of social media. But posts that would be appropriate in those places aren't appropriate here. I assert that de-emphasizing comments helps focus users on the idea that we should all be here to do work, the work of building a knowledge repository in whatever domain, and not to ‘get help’ or socialize. (One very good reason to do the work is that we learn and grow by asking and answering, but every bad user interaction I've had here or Elsewhere has involved the other user, in my subjective judgment, not having the mindset that they're here to do work.)
IMO, comments would be even better if they were as hidden as Talk pages are on Wikipedia. I don't particularly care if they're deleted or preserved, but I do think that, like Wikipedia, we should be drawing a bright line between the project and any necessary communication that happens around the project. I acknowledge that I am a hypocrite who often gets baited into comment conversations, even though I don't really think that's what comments as they're currently presented should be for—the problem is that we don't have any better tool for coaching users into being better community members, like a private message or chat or talk page system.
Edit: OP raised the use of comments as ‘footnotes’, containing content that should be visible. I disagree with this use; footnotes, like this one, should be made part of the answer, like I am doing now. They should be refined from whatever conversation inspired them and not left tangled up in dialogue for future readers to piece together like historians going over primary sources. This improves the answer, which is what we should be trying to do with all posts.