The Veritas Capital-backed company behind Blackboard struggled to integrate its business lines and faced tough competition.