Your views are very similar to those of John Taylor Gatto. Are you familiar with his work? Most libertarians I know consider him a hero which is interesting given his vociferous criticism of IQ testing
I am somewhat familiar with Gatto and I quoted him in my chapter on schooling and education in The Joy of Freedom: An Economist's Odyssey. Parenthetically, I wish school reformers, who include some of my colleagues at Hoover, would go in his direction rather than try to turn the Socialist Ship of State Schooling a few degrees.
Now to the testing issue. I'm not familiar enough with IQ testing to answer, but I reprint here my retelling, in the aforementioned chapter, of a story about educational testing told in 1995 by Charles Johnsen. Here's what I wrote:
The next story comes from an educational tester who, for reasons you will quickly understand, is no longer an educational tester. His name was Charles Johnsen, and when he wrote this story in 1995, he was a Lutheran preacher and a computer chip designer in Aurora, Colorado. Twenty years earlier, Johnsen worked on a proposed test for a testing firm in Chicago that had a contract with the Chicago Board of Education. One of the questions on the test was about the "el," the word used locally for Chicago's system of elevated trains. Johnsen put a photocopy of an actual el schedule in the test and asked a question like the following: "CJ has a job interview at 9:00 a.m. The company is a block away from the State Street station. What is the last train CJ can take to get to the interview on time?" Then he listed five different train times from the schedule, only one of them right.
The results surprised him. Students in the suburban schools did well on the other items but, in his words, "blew the item about the el big time." But the kids in the inner schools generally got the right answer on the el question even when, as was obvious from the rest of the test, they could barely read.
That was the first surprise. Then came his bigger surprise. His employer's software generated statistics about each test question. A "good" question did not discriminate between ignorance and knowledge, but instead discriminated between "good" students and "poor" students. No matter how important the question, if the obviously good students⎯the ones in the suburbs⎯got the answer wrong and the obviously bad students⎯the ones in the inner cities⎯got the answer right, the question was thrown out. I think of this story whenever I hear people say that the SAT does not discriminate against black people.