The language of physics is mathematics, and it cannot be done honestly without mathematics. That makes it inaccessible. The language of literature is English or Chinese or whatever, and that makes it accessible. And literature is about the human condition. Physics is about the nonhuman condition. It’s not a taste that all human beings have.
People are also social creatures, and literature fits in with that. Physics is perceived as a lonesome, nerdy kind of enterprise that has very little to do with human feelings and the things that excite people day-to-day about each other. Yet physicists in their own working environment are very social creatures.