Your vocabulary.
The active vocabulary consists of the words that you use. While you don't want to talk over everyone's heads, which is a classic form of poor communication, you need an ample supply of the right words for various situations in your active vocabulary. You also want to keep things interesting by stocking a variety of fun and evocative words.
Consider this: Maintain a passive vocabulary, too.
Even if you don't use certain words yourself, you must have them standing by in your passive vocabulary. People will use them on you, and you need to know what they're saying.
For example, fatuous is probably a word you wouldn't normally use yourself. But if you come to me and ask, "How was my presentation?" I might tell you, "It was the most fatuous thing I've ever heard." You quickly ransack your long-term memory and conclude that it's related to infatuated.* So you call your mother and brag about how I loved your presentation. Your mother, however, pulls the plug by looking fatuous up in the dictionary and finds that it means "completely idiotic."
The moral: It's sometimes dangerous to try to work out word meanings on the fly. If you want to work with educated people, you need a well-developed passive vocabulary.
There's no way to say how much vocabulary is enough. It's really a question of what you do with what you have. Just keep in mind that between the desert wastelands of ignorance and the lofty peaks of arrogance lie the vast, fertile plains of competence and style. Endeavor to get yourself out of the wastelands and resolve to stay off the peaks, and you should be fine.
__________
* Actually, you wouldn't be wrong. Infatuated relates to this idea that you fall so deeply for someone your brain quits working properly.