Not 100% sure if/how self-signed certs might differ from this, but we made our own certification authority for internal use using these instructions then it's possible convert the root certificate from X509 to DER format like this:
openssl x509 -in ROOTCERT.pem -outtype DER -out ROOTCERT.der
... and now it can be opened and installed on your Symbian phone.
It's necessary to specify the trust level of the certificate after install to say for purposes you want to trust the certificate (to authenticate secure web sites, email, application installs, etc.) The method of doing this will vary by phone but IIRC my S60v3 phone prompted me at the moment of install, whereas my 9300 Communicator required setting manually in the security settings in Control Panel.
If you want to serve DER files from a web server, they should be delivered with mime type application/x-x509-ca-cert.