Who hasn't wished for a library that neatly solves the programming problem that is currently staring you in the face? When you're down in the dirt, gearing up to tackle some gnarly problem that seems like it should have a general solution, it's only natural to wish, or even expect there to be a library that addresses your need. Once you have a workable solution that you think is elegant and useful, don't you think about releasing it for the world, or at least your company, to fulfill other programmer's wishes for self-contained solutions to their problems? Be careful what you wish for.
This post is probably not going to be about what you're thinking right now. I'm not going to relate some harrowing tale about a software library I picked up off the street, hoping it was going to solve all of my problems, only to find that it was more trouble than it was worth. Honestly, I haven't had many experiences like that. Either I've had amazing luck finding exactly what I need, or I have low expectations and work through the problems with the libraries I find as they come up. No, this post is about the way the software library marketplace has grown exponentially over the years and how one type of complaint about libraries has morphed into another, seemingly without notice.
When I was first learning to program, the languages I learned had small standard libraries. C has a pretty manageable standard library. C++ also has a small standard library by today's standards, even though it includes most of the C standard library as well. You could also include the standard template library, still keep all of the C++ library functionality in your head, and recall it at will to solve your programming problems. Early Java also had a tractable standard library.
Admittedly, my early view of the world of programming languages was somewhat limited, but the world was definitely smaller back then. One of the most common complaints I heard was that the world was a little too small. We needed more good libraries and frameworks for building applications—prepackaged plug-and-play code that would solve all of our problems and make development easy and painless. Sure, there were some isolated examples of application development frameworks out there, like MFC for Windows development in C++. But we wanted more, much more, and we weren't afraid to ask for it.
Now I'm not in any way knocking the fact that we wished for all of these libraries and frameworks, or that we expected them to solve all of our problems. It was and is entirely reasonable, even virtuous, to not want to reinvent the wheel over and over again. We should package up solutions to solved problems and make them part of the foundation that we then build on top of. I wouldn't have expected or wanted it to happen any other way, but I don't think many programmers realized that it would happen the way it did.
Today Java 8 has over 4,200 classes in the standard library public API. The .NET 3.5 framework has nearly 10,000 public classes—the latest version I could find numbers for. The .NET 4.5 framework has well over that number, I'm sure. No single person could possibly know everything that's in the standard libraries of these languages, and that's not even counting all of the contributions available from open source libraries.
The point here is not to get an accurate measure of the number of libraries available for different languages, but to show that in any language you use, that number is absolutely massive. The game has fundamentally changed from knowing everything that's available and how to use most of it to searching for what you need when you need it and evaluating whether or not you can use it (90% of it is crap, as per Sturgeon's Law). This shift makes some people very uncomfortable.
You see, back when we were complaining about the lack of good libraries and frameworks for building awesome applications and solving all of the mundane problems of software development, this is not what we wanted. We wanted elegant, high-performance tools that perfectly fit the problems we were trying to solve, not a mountain of crap to sift through for a few choice pearls. I posit that the former was never going to happen, and the latter is the best we could have hoped for.
Part of the reason is purely because that's the nature of software development and the web. When anyone can post a library on the web for others to use, you're going to end up with a ton of stuff that nobody needs or is of decidedly low quality. To get the fraction of really good stuff, we need to put up with a lot of mediocre, useless stuff.
The other part of the reason is that perfect software libraries are one of Steve Yegge's Nonesuch Beasts. They don't exist. You can't build one. It's impossible, and here's why. Any library that's generally applicable to a wide range of projects necessarily needs to solve problems in a general way. On the other side of the coin, you likely require a specific solution to a narrow problem, and a large collection of projects like yours will have a wide range of narrow problems that are slightly different.
I count that as a good thing. If programming was simple, and all we had to do was plug together a bunch of libraries we found on GitHub, it would get unbearably boring. Besides, if we truly solved the software library problem, it would mean we were no longer innovating and pushing the frontier of software. Researching options and piecing them together to build novel solutions, basically creative problem solving, is an inherent part of our job. It is fundamental to what we do. Without that difficulty, most of us would probably be out of a job. We wished for more and better libraries. Now we're getting them, and we're wishing that we didn't have to think so hard about when and how to use them. We should be careful what we wish for. What if we actually get it?