Custom errors won't turn off

[Agendum]

.NET Framework 3.5
Visual Studio 2008
C#

I implemented my own transport channel. It works to a point: the
client is capable of invoking a method on a server singleton. When
the method completes successfully it exits and back on the client I
receive:


System.Runtime.Remoting.RemotingException was unhandled
Message="Server encountered an internal error. For more information,
turn off customErrors in the server's .config file."
Source="mscorlib"
StackTrace:
Server stack trace:
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage
(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke
(MessageData& msgData, Int32 type)


After searching the internet, I found the main culprit being that I
needed to:

RemotingConfiguration.CustomErrorsMode = CustomErrorsModes.Off;

And the usual cause for the failure was:

SoapServerFormatterSinkProvider provider = new
SoapServerFormatterSinkProvider();
provider.TypeFilterLevel = TypeFilterLevel.Full;

However, this didn't solve the problem. Using configuration files
also did not help.

To discover the source of the problem, I examined the outgoing message
of my transport stream and I can see that the server is writing the
response as:


HEADER: Content-Type=text/xml; charset="utf-8"
HEADER: StreamMessageId=04:21:43.792#-2147483646
STREAM:
<SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-
instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-
ENC="http://schemas.xmlsoap.org
/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/
envelope/" xmlns:clr="http://schemas.microsoft.com/soap/encoding/clr/
1.0" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/
encoding/">
<SOAP-ENV:Body>
<SOAP-ENV:Fault id="ref-1">
<faultcode id="ref-2">SOAP-ENV:Server</faultcode>
<faultstring id="ref-3"> ****
System.Runtime.Remoting.RemotingException - Server encountered an
internal error. For more information, turn off customErrors in the
server&#39;s .config file.</faultstring>
<detail xsi:type="a1:ServerFault" xmlns:a1="http://
schemas.microsoft.com/clr/ns/System.Runtime.Serialization.Formatters">
<exceptionType xsi:null="1"/>
<message xsi:null="1"/>
<stackTrace xsi:null="1"/>
<exception href="#ref-4"/>
</detail>
</SOAP-ENV:Fault>
<a3:RemotingException id="ref-4" xmlns:a3="http://
schemas.microsoft.com/clr/ns/S
ystem.Runtime.Remoting">
<ClassName id="ref-5">System.Runtime.Remoting.RemotingException</
ClassName>
<Message id="ref-6">Server encountered an internal error. For more
information, turn off customErrors in the server&#39;s .config file.</
Message>
<Data xsi:null="1"/>
<InnerException xsi:null="1"/>
<HelpURL xsi:null="1"/>
<StackTraceString xsi:null="1"/>
<RemoteStackTraceString xsi:null="1"/>
<RemoteStackIndex>0</RemoteStackIndex>
<ExceptionMethod xsi:null="1"/>
<HResult>-2146233077</HResult>
<Source xsi:null="1"/>
</a3:RemotingException>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>


Thus the server is definitely the one causing the error, not the
client. To make sure my CustomErrors and TypeFilterLevel settings
were being set, I instrumented the pipeline to print the CustomErrors
and TypeFilterLevel when both reading a request and writing a response
in both the formatter provider and sink. In both cases CustomErrors
is OFF and TypeFilterLevel is FULL.

Therefore I can say without a doubt the problem is not these two
settings. I also examined the request message just to make sure the
request doesn't have anything magical which will cause the response to
have CustomErrors turned on. But it does not -- it is a typical
envelope with expected headers.

I've been tearing my hair out trying to get CustomErrors turned off so
that I can debug the real problem. Does anybody have any insight on
why custom errors won't turn off?

Thanks!

[Agendum]

I managed to resolve the problem. I used .NET Reflector to trace
where the exception is being set. The only place it is set is in the
ReturnMessage exception-based constructor:


public ReturnMessage(Exception e, IMethodCallMessage mcm)
{
this._e = IsCustomErrorEnabled() ? new RemotingException
(Environment.GetResourceString("Remoting_InternalError")) : e;
...


The implementation of IsCustomErrorEnabled is:


internal static bool IsCustomErrorEnabled()
{
object data = CallContext.GetData("__CustomErrorsEnabled");
return ((data != null) && ((bool) data));
}


So I searched for references to the "__CustomErrorsEnabled" string and
found the predefined formatter sinks do this in ProcessMessage:


bool data = true;
object obj2 = requestHeaders["__CustomErrorsEnabled"];
if ((obj2 != null) && (obj2 is bool))
{
data = (bool) obj2;
}
CallContext.SetData("__CustomErrorsEnabled", data);


So basically, even though I set RemotingConfiguration.CustomErrorsMode
to be OFF on both the server and client, .NET Remoting expects the
client to include this in the headers. Even though the predefined
transport channels (HTTP, IPC, TCP) do this, since I made a custom
transport channel I had to implement this /completely undocumented
logic/.

I have in front of me the books .NET Remoting (Scott McLean, James
Naftel, and Kim Williams), and Advanced .NET Remoting (Ingo Rammer,
and Mario Szpuszta), and neither books makes any reference to this
logic for custom transport channels. Maybe this logic was implemented
after these books, so a search on Google for "site:microsoft.com
__CustomErrorsEnabled" (Bing cuts off the underscores) reveals
nothing. Basically, this is me ranting on complaining because
Microsoft fails to document things and I am left to go above and
beyond to figure out their own intentions.

Thanks!