Would a cello sextet be allowed?

Yes. Your ensemble may be made up of similar instruments -  flute ensemble, a cello sextet, or a guitar ensemble, for instance.