My old friend Dean used to complain about any setting that required a lot of knowledge of setting. "I don't want to read a novel to have to play." Oddly enough, he loved using Forgotten Realms as a setting, but that may be because it was such a generic D&D setting. While you could read a ton about the setting, you didn't necessarily need to do anything other than roll up a character and start playing.
This is always a challenge for home-brew Referees, especially those who use more unusual settings. While the weird and distinctive parts of any setting are what makes it come alive, these things are also the barriers that keep players from feeling like they can just dive in. Consider the difference between Forgotten Realms and say Tekúmel or Glorantha. FR is popular and generally seen as accessible, largely because it seems intuitive to D&D players and anyone who has read a lot of pseudo-Tolkienian heroic fantasy. It's a mish-mash of popular fantasy fiction tropes and contemporary social conventions, and thus is easy to just take in stride. The others have developed societies that seem to have their own cultural assumptions that make it hard to know how to interact. In fact, in Tekúmel, improper social action may have dire consequences — which is certainly where the original idea in the first edition of EPT had characters start out as barbarians.
Now, the original RQ and EPT had far less fluff and developed background than just about any edition of FR. So the truth isn't so much about "reading a novel" as Dean put it, but in how much effort the player has to put in just to do things in character. The real issue is whether playing in your game world is an effort or not. Even if your game s worth the extra work, RPGs are a recreational activity, and as such, it's reasonable to ask if the effort of entry is worth the level of fun had.
The solution is not to eliminate interesting and involved game world background. However, the thing to remember as a referee is that background and immersion are your job to create, rather than putting the onus on players. Structure your early sessions to introduce and reinforce elements of the setting rather than expecting players to read your campaign handout. By giving meaningful demonstrations of the weirdness of the world in-game, then it becomes more real, more manageable, and less like homework.