You are not married to your agent. But, if you are not well suited to one another, getting a "divorce" can be tricky, especially if your agent has already sold one of your books. A split-up can involve a custody battle: changing the terms of your contract, lawyers, hard feelings.
And there will be gossip. The publishing industry is more provincial than you might think. Editors love to gossip among themselves about authors, and those editors often become agents, who also do their fair share of gossiping. If you end up divorcing your agent, everyone will know about it.
Guess whose side they will be on?
So, before you leap into the arms of the first agent who is willing to get down on one knee, consider the following:
Does the agent have a good track record with authors? Ask around. Find some authors they have represented (you can even ask the agent for a list), and ask how happy they have been. Go to conferences, talk to writers. Try to get a sense of how the agent interacts with people. Google the agent's name and see what pops up on absolutewrite, a forum where writers talk about their experiences in the publishing world.
Does the agent adore your work? Agents can only sell work that thrills them. Does the agent stand behind your book 100%? Will the agent be willing to spend a couple of years, if that is required, to get a contract? Or will he/she dump you after a few tries?
Does the agent like you? It is important for agents to be professional, but it is equally important for them to take an interest in you. I am not talking about sharing "worst date" stories. You need to feel comfortable enough to be able to ask your agent important questions.
Can you trust your agent? A publishing contract is not the end of the road. Contract negotiations are nerve-fraying experiences. If your agent is curt, or doesn't respond to your questions (particularly if it is your first contract ), or if your agent does not explain things to your satisfaction, you may want to bail out of the relationship before your contract is finalized. (This happens a lot more than you may think.) Talk to the agent about how he/she handles contract negotiations. And listen to your gut. If you have doubts, there may be a good reason for them.
There are other considerations as well: How many clients does the agent have (too many, and they won't have time for you, too few and they aren't successful); How many publishing houses has the agent worked with (if their publishing contacts are limited to houses that also accept unagented manuscripts, it's not a good sign); What genres has the agent represented (YA fiction is all the rage right now, but if an agent has not represented YA authors before, he or she may not have the contacts you need).
Ideally, you want a long and happy relationship with your agent. If you stand back and ask yourself whether you and your prospective agent are a "good fit" right at the start, you will avoid many problems further down the road.