Who Actually Pays a Realtor's Commission?

I keep beating my head against the wall when it comes to real estate agents. Here's another.

We are looking to purchase a home.  I don't use a real estate agent, largely because at least half of the agents I meet fail to impress me, and the ones I've used in the past decreased the value of the transaction for me, not increased it. Many are good, smart, and hard-working people, but there are just too few of them for me to spend time weeding out the incompetents for what I need.

Additionally, given the technology available today, the services of an agent are largely unnecessary, especially since I know the areas where the homes I am interested in are located. I therefore don't need an agent to purchase a home.  

Consequently, I deduct 2.5 to 3% from any offer I make in addition to what I would have offered had I used an agent. (My offers are actually a bit more complex than this analysis, but I'm trying to keep it simple.) When discussing this with agents, I am invariably told that I don't pay the commission anyway - the seller does - so it's not fair if I offer less simply because I don't have an agent.

I explain to them that first, much like the incidence of a tax, who actually ends up paying the commission depends on market conditions: in a seller's market the buyer is normally stuck footing more, if not all, of the bill, and in a buyer's market it's the seller. Yes, either way the money is traditionally collected by the seller in the form of a higher price and paid out to the agents, if any, but the selling price varies based on the market, reflecting the incidence of the agents' commission. Ceteris paribus, in a buyer's market the price will be lower than it would be in a seller's market. 

Real estate agents usually stipulate in their listing contracts that buyer's and seller's agents will split the commission in some way, and if the buyer is sans agent, then the listing agent keeps it all. This is a crock. By not having an agent in this buyer's market I am not bringing another mouth to feed to the table. Therefore, additional charges are not being incurred predominantly, if not exclusively, by the seller and doled out to an agent working on my behalf. I am incurring the additional work, writing my own contracts, taking the risks, etc., and I want to be compensated for my efforts, not the listing agent.