It looks like Apple have tightened up on the Cert usage flags in OS X Server 5.
A Cert that worked in 4 may no longer be allowed
First check and remove any of YOUR cert entries from
/etc/certificates
Then make sure you key Extension usage matches the ones in the Self Signed Cert that Server makes.
In the case of server 5.0.15 these are :-
Key Usage ( 2.5.29.15 )
Critical – YES
Usage – Digital Signature, Key Encipherment, Data Encipherment, Key Cert Sign
Extended Key Usage ( 2.5.29.37 )
Critical – YES
Server Authentication ( 1.3.6.1.5.5.7.3.1 )