It's pretty simple - black immigrants and their progeny should not be beneficiaries of affirmative action. Affirmative action is a remedial program to redress historical harms (not prospective harms). It's clear that AAs made a strategic mistake in defending affirmative action on the grounds of diversity.
Also, AAs really screwed up for pushing for (and achieving) immigration reform in 1965, which allowed black and "other" non-whites to immigrate to the country in MUCH larger numbers (the law removed certain quotas on non-white immigration). Most immigrants and their children have absolutely no idea of the law or its history (kindly note the year of the passage of the law - the same as the Civil Rights Act).
These strategic mistakes have resulted in AAs having greater competition for limited resources and very little (if any) political or economic benefit resulting from such immigration.