It's not about numbers. The Jewish Holocaust was a program of genocide of an entire people by one state, while the goal of slavery was not extermination. The deaths were a side-effect of enslavement, with no single group responsible for it or for slavery. Africans enslaved Africans; Arabs enslaved Africans; and whites enslaved them, etc.
Why is it that the Jews don't go on about how they were enslaved for thousands of years by the Egyptians, Babylonians, Romans, Spain, etc., and that many more millions of them died as a result than died in the Holocaust? Why isn't that a tragedy to the Jews, or the world, worse than the Holocaust? For the same reason that the Africans killed because of slavery isn't on the same level, whether more died or not. It would be like comparing all the Japanese civilian casualties during four years of war with the casualties at Hiroshima. It's not the same thing, even when the non-Nuke casualties over four years may have been more. It has a special moral dimension to it. So too the Holocaust.
You think it gets brought up way too much? Apparently, it doesn't get brought up enough when we have Americans and foreign governments, such as Iran, trying to disprove the Holocaust's existence! Are there any governments or anybody trying to disprove black slavery's existence?