Size matters, or does it? We are talking speakers here. And since we have already stated that audio is mostly a male pursuit, how much does it matter? Many, if not most, people would prefer to get live-quality sound from invisible speakers...and the fewer speakers the better.
A personal experience with this question:
A friend of a friend, living in Manhattan, was a true music lover. She had subscriptions to Metropolitan Opera, several series at Carnegie Hall and attended many local concerts at Julliard, at local churches and synagogues. Music was a big part of her life; but she had no music at home. She felt every sound system she thought worth listening to was too large for her small apartment.
When I described the Ohm M to her as being the smallest Ohm ever and about the size of a paperback book (they were usually matched with Ohm N subwoofers), she got excited. She was disappointed when I clarified that while the speaker had about the same front dimensions as the cover of a paperback, it was 4 inches deep. But, what you do for friends or even friends of friends, I was able to make her a custom subwoofer with a 6.5" driver and the story ends happily. This was 35 years ago, before the Bose Acoustimass systems popularized the sub/sat systems by hiding the subs in their ads and in showrooms.
Today, you can get speakers the size of a paperback book to use with mp3 players. Are these too small? You must decide what you want the speakers to achieve. The three main questions to answer are "How loudly do you want them to play?", "How big a space do you want them to fill?" and "How far away is your listening position from the speakers?" My reference sound level is the level I enjoy in a live setting, sitting in the center about 12-15 rows back in Carnegie Hall: be the performance a solo singer or full orchestra. These still are much lower levels than I have endured at rock concerts and even some Broadway musicals; but higher than some neighbors appreciate late at night.
At one extreme, ear-buds need to fill a space only as large as your ear canal and are very close to your ears. They can be very small and easily reach my reference levels. In fact, it is pretty easy to play music so loudly it damages your hearing with ear-buds. Headphones are also very small spaces and very near your ears; so, again, they can achieve reference levels fairly easily. At the other extreme are concerts in outdoor stadiums and parks; the distance can be far and the room size is so large, it is nearly impossible to determine. You need a really big speaker array to reach my reference levels far from the stage. Often, the size of the space for your seakers to fill with sound will lie somewhere between these two extremes. Automobiles, for instance, are on the smaller side; while home listening areas vary from small dens and bedrooms to large great-rooms in open-plan homes.
The sound level generally falls off as you move away from a speaker. If you are sitting close to the speakers --at a computer, for instance -- you can enjoy reference levels from fairly small speakers. If you sit fifteen feet back from your main speakers, you will need them to be much bigger. If you are in the great-room, you probably need a big subwoofer to provide the deep bass that will flow through the whole house.
And now for the technical stuff...
One size does not fit all. What speaker specification should you look for? In the deep bass, you need to know how large a space the speaker can fill while still reaching the levels you want. (When the wavelength is bigger than a room dimension, you get a boost in level because the walls are constraining the flow of sound.) When you are sitting within about one wavelength (13.75 feet at 80 Hz) the sound level is more related to the distance from the speakers since there is little room reinforcement and most speakers get softer at the square of the distance. In my experience, to achieve my desired reference level of music at 15 feet requires a 12" system if it is sealed or 8" if vented. If is a sub/sat system, the subwoofer should be these same sizes (or larger if used for home theater where atomic bombs require more bass than any music). But the satellite can be as little as a 6.5" driver. If you are closer, the drivers can be smaller, if further away, you will need larger drivers.
So, how small is too small? The only real answer is "There is no one answer; It depends on your personal situation."
Enjoy & Good Listening!