Implementing custom security handler based on standard PDF security.

Q:

I've subclassed StdSecurityHandler and overridden FillEncryptDict() in
order to set our "Filter" in the encryption dictionary along with some
other data we need.

I provide a fixed 128 bit key in the User Password and encrypt a pdf
document.

When I open the document in Acrobat Reader it calls our own custom
security handler and attempts to open the document. I've hardcoded the
decryption key in our security handler to ensure it matches the one I
provided in ChangeUserPassword() to our StdSecurityHandler derived
class.

The document opens but the text is garbage.... i.e. it didn't decrypt
correctly.

In the past we wrote a custom security handler derived directly from
SecurityHandler rather than StdSecurityHandler. In this class we had
to implement all of the encryption routines.

I am basically trying to use StdSecurityHandler because it has the
encryption filters for RC4 and AES built in. I don't want to have to
implement my own encryption filters.
----

A:

It is possible to treat standard PDF security as a complete black box
(and we have several customers who implemented custom security filters
in this way).

Basically you would first use standard PDF security handler as normal:

1)
StdSecurityHandler new_handler = new StdSecurityHandler();
string my_password = "test";
new_handler.ChangeUserPassword(
  new System.Text.UTF8Encoding().GetBytes(my_password));
new_handler.SetPermission(SecurityHandler.Permission.e_extract_content,
false);
...
pdfdoc.SetSecurityHandler(new_handler);
pdfdoc.Save("my.pdf",...); // or save in memory

2) Then you would modify standard PDF encryption dictionary. For
example,

Doc doc = new Doc("my.pdf");
// Note: do not call doc.InitializeSecurityHalder() here.
Obj encrypt_dict = doc.GetTrailer().Get("Encrypt").Value();
encrypt_dict.Put("Filter", new Name("MySecurity"));
// encrypt encryption keys "O" and "P" using your own proprietary
encryption.
// you can modify key names, add additional key/value pairs etc.
...
// then save changes...
doc.Save("my.pdf", 0, 0);

So to encrypt a PDF document using custom security you don't even need
to derive from StdSecurityHandler.

The advantage of this approach is that you don't need to understand
all PDF encryption algorithms. You only need to edit encrypt
dictionary after (exporting) or before (opening) the document.

If you would like to implement a new security handler that is not
based on StdSecurityHandler, it is also possible by deriving from
SDF.SecurityHandler instead of SDF.StdSecurityHandler. If you want to
simulate all algorithms and all of the functionality from
SDF.StdSecurityHandler, it is also possible but is really lots of work.

Q:

After stage 2 the saved document is still encrypted however Acrobat
doesn't think it uses a security handler so doesn't load either the
standard security handler or the vincera one. So Acrobat opens the
document and all the content is garbled as it's still encrypted.
---

A:

It seems that /Encrypt dictionary is not present in the document (this
is most likely the reason that Acrobat is not calling your plug-in).

Try adding SDF::Doc::e_incremental flag to the second call in
doc.Save() method:
  doc.Save(" my.pdf", SDF::Doc::e_incremental, 0);