“We’re building something much smarter than us” understates the threat. After all we have people with IQ-140 co-existing happily with people with IQ-60 — and entities like corporations and markets which aggregate or tile this intelligence in various ways — what could a few more IQ points do?
Perhaps a phrasing that better communicates the intelligence explosion thesis is: we’re creating the technology that lets an IQ-140 being create an IQ 150 being.
… and the question is: how does this technology get “generalized”? Vaguely: - Does it let any IQ \(x\) being create an IQ \(x\cdot 15/14\) being? - Does it let any IQ \(x\) being create an IQ \(x+10\) being? - Does it let any IQ \(x\) being create an IQ 150 being?