One of the things that impressed me about reading Tolkien for the first time (and even more so, years later) is how he crafted entire unique languages for all of the nations and peoples in Middle Earth. The same has been done (with varying levels of success) by many authors. Some have created languages that are memorable, others crafted ones that add some flavor but not much more than that, and still others create a mess that serves as a hindrance to the reader.
When someone goes about crafting a language, Tolkien is often held up as the example. Of course, he was a linguist, so he had some advantages. He knew and understood languages on a level which most authors don’t really have time to do. Don’t forget, that he spent years developing his languages.
So, other than spending years working on developing a language, how can an author produce something that adds value to their work? There are a number of techniques that I’ve seen and used myself. The first one is to “borrow” from other languages. Tolkien borrowed heavily from Welsh, Finnish, and Gaelic. This of course requires finding or knowing languages that sound or look right for the culture you’re dealing with, as well as some vague familiarity with the language.
Another great technique is using a few words to add a bit of flavor to your text. A greeting here, a curse word there, can give the reader a feel of a distant land and new people. A great example of this is the Firefly series, where they used Chinese greetings, exclamations, and curses. Never enough that a listener was confused, just enough to add some spice.
Crafting languages with other alphabets or runic symbols is another method to add a bit of variety. Where this becomes an issue is formatting, especially with ebooks. If you fancy yourself an artist, you can spend hours, weeks, or even years crafting a unique alphabet (or borrowing from existing ones), which may then only appear in cover art or pictures within the novel.
Pitfalls of writing with your own fantasy and science fiction languages are things most well-read readers have encountered.
By and large, most readers tend to avoid big blocks of text they can’t read or understand. Now, if your intention is to confuse the reader, having long sentences in your own created language can work, but otherwise, I’d advise against it. Unless you think you’re as good as Tolkien (and even then) you probably won’t have people spending hours or days figuring out how to read your invented language.
The above mentioned runic language is another pitfall. Even if you consider yourself a professional artist, take the time to make sure what you are putting into a novel is what you really want there. Even in traditionally published novels I’ve seen crudely drawn bits of runes that I’ve mistaken for doodling in the margins.
By and large, the most important aspect is when you edit your novel. You may have spent years developing your languages, but if your story flows better without those, then you’ll need to cut it. By all means, throw some things in there to make it a bit more exciting or exotic, but not enough to eject the reader from your story.