Difference between revisions of "DTLS Implementation Notes"

Revision as of 22:12, 19 January 2009

This page documents the experience of implementing SNMP over DTLS as described by documents being developed for the ISMS working group. A large section of this is relevant only to SNMP developers (Net-SNMP or otherwise) and some of this is relevant to anyone who is implementing a DTLS solution using OpenSSL. In particular, there are a number of tricks that need to be employed to make OpenSSL properly handle multiple clients. The implementation and this document were done by Wes Hardaker

Net-SNMP Background

Net-SNMP implements the transports over which SNMP messages can be sent using a pluggable architecture. This architecture defines hooks that allow implemented transports to handle opening, sending and receiving packets through "something or other".

To implement DTLS support within Net-SNMP a new transport had to be written that was responsible for sending and receiving packets. This new file (snmplib/snmpDTLSUDPDomain.c), though not yet checked into the SVN repository as of the time of this writing, is functional and has passed a number of tests. It should hopefully be ready for release by the Net-SNMP 5.5 release.

OpenSSL Background

OpenSSL's internal implementation architecture is well designed from a modular point of view. To some extent, however, this will come back to bite us as we'll see later on.

TLS/DTLS

Internally the TLS and DTLS implementations merely process the data they receive through "anything" and send responses back through the configured mechanism. These sending and receiving mechanisms are entirely separated from the TLS/DTLS implementations by a modular data sending and receiving framework called BIOs.

BIOs

BIOs in OpenSSL are merely code that knows how to send and receive data. You could easily think of them as a buffering layer between a data producer and consumer and where-ever-the-data-needs-to-go. They act in a very similar fashion to the way transports separate the Net-SNMP packet processing from the sending/receiving framework.

BIOs exist in OpenSSL to handle sending and receiving from many different directions. Probably the most common BIO is the one that attaches to a network socket like the TCP BIO. There are also BIOs that wrap around stdout, and there are memory BIOs that merely store data in a buffer. Starting with some version of OpenSSL there also was a datagram BIO that was designed to be used with DTLS and UDP. The datagram BIO is merely a wrapper around UDP sockets in the same way that the TCP BIO wrapped around TCP sockets.

TCP vs UDP Background

There is a fundamental difference, of course, between how UDP and TCP works. The biggest difference is in receiving packets. Normally for TCP you have to call accept() to allow a new connection to come through and get a new OS socket for sending and receiving on that socket to that peer. You have no idea yet how important that last statement is, so let me repeat it. In bold. TCP implementations provide consumers with one socket per connection.

Within the OpenSSL framework, when a new connection is requested from a new client the OpenSSL stack also creates a new BIO which is then attached to the new socket.

With UDP there is only one socket available to send and receive from (well, you could create multiple sockets with a different socket per client using a different UDP port per socket but you'd have to convince all your clients to send the traffic to the newly opened port just for them... In fact this is sort of what FTP does). With UDP a receiving server needs to check each packet that it's getting data from for the source address. Then if it needs to respond to the packet it needs to make sure that it remembers the address so it can send the response back to the right place. (TCP implementations, on the other hand remember it for you so there is less to do).

OpenSSL Context Background

OpenSSL needs to keep state with respect to ever TLS or DTLS session that it has established. It does this through the use of a SSL * pointer which is then attached to the sending and receiving BIO * pointer using a call as follows:

   SSL_set_bio(SSL *ptr, BIO *read_from_bio, BIO *write_to_bio);

Then, whenever stuff needs to be done (be it sending data or negotiating cryptography) the internal TLS implementation, when operating on traffic or data being sent in the context of the ptr function can simply send and receive traffic destined for it using the two "tied" BIOs. Normally, as I mentioned earlier, these BIOs are wrappers around TCP sockets for use with TLS. But wait, there's more...

Where the problem comes in

Problem 1: UDP is a single socket

The first problem is that UDP uses only a single socket for sending and receiving. OpenSSL has to send and receive from multiple locations. IE, it needs multiple SSL * / BIO * pairs. I thought, originally, that maybe I could trick openssl and manually tell it which SSL * pointer to use for each packet. IE, I would do a PEEK at the incoming packet and based on where it was coming from (the remote source address and source port) I could pick the right SSL * pointer and tell OpenSSL to process the incoming packet. That way I could have just one DGRAM based BIO (like all the OpenSSL examples use) and still handle multiple connections.

This actually almost worked, except for the next problem got in the way:

Problem 2: Packets Pile Up

When OpenSSL is operating, particularly when starting up a new TLS or DTLS session, it needs to read and write multiple packets in order to complete negotiations. Internally OpenSSL will attempt to read as much as it can from the BIO it was given in order to process as much as possible before returning control to the application (possibly with data).

In the case of TLS over TCP, this is perfectly safe since the the BIO is attached to a socket where all the data coming through the socket should be destined for that one TLS connection (and if it's not, it's safe to throw away).

However! This concept of reading as much as you want fails completely when using UDP. The first packet may certainly be destined for the current SSL * context (especially if the application handed it carefully to OpenSSL as described above). But there is no guarantee that the next packet will be from the same client or the same TLS session.

So, when OpenSSL is trying to be as proactive as it can and read as much as it can from a socket before returning data to the calling application, it may actually read multiple packets from the BIO attached to the UDP socket and assumes they're all destined for the SSL * context that it's currently handling. This is a completely invalid thing to do, unfortunately. And in fact when I launched two DTLS instrumented versions of snmpget against a single DTLS instrumented snmpd server one of the clients failed because it was suddenly trying to read too much data from the UDP socket and read packets for one SSL * context that were, in fact, destined for the other.

The Solution That Works

So, the solution that I've found that works is to take the sockets completely away from OpenSSL's view. Instead, I use memory BIOs to communicate to and from OpenSSL and my code personally copies stuff between the network's UDP socket and the input and output memory buffer. This works fairly well but requires a lot more processing by the client/server code that OpenSSL really should take care of for us.

The general set up is as follows:

THIS IS HALF PSEUDO-CODE AND WILL NOT WORK AS IS

Initialize things

   SSL_library_init();
   SSL_load_error_strings();
   ERR_load_BIO_strings();
   OpenSSL_add_all_algorithms();
   
   /* Open UDP socket */
   int sock = socket(PF_INET, SOCK_DGRAM, 0);

Receiving a Packet

After select() notices that there is new traffic on the socket, you should be able to do something like:

   struct sockaddr *from;
   rc = recvfrom(sock, buf, size, MSG_DONTWAIT, from, &fromlen);
   

Now, if this is a new connection (ie, from a new remote address and remote port) then we won't have anything in our cache and we need to create a new SSL * connection context for it and a new set of memory BIOs for handing stuff in and out.

So, only if it's from a new address/port:

   /* this assumes you have a valid SSL_CTX *ctx already created and available */
   SSL *con = SSL_new(ctx);
   
   /* set up the memory-buffer BIOs */
   BIO *for_reading = BIO_new(BIO_s_mem());
   BIO *for_writing = BIO_new(BIO_s_mem());
   BIO_set_mem_eof_return(for_reading, -1);
   BIO_set_mem_eof_return(for_writing, -1);
   
   /* bind them together */
   SSL_set_bio(con, for_reading, for_writing);
   /* if on the client: SSL_set_connect_state(con); */
   SSL_set_accept_state(cachep->con);

Remembering the con, for_reading and for_writing pointers and associating them with a remote address/port combination is left as an exercise for the reader. You can look at the caching system that is used within the snmpDTLSUDPDomain.c file when I get it checked into SVN.

Once the two BIOs and the SSL pointers are known, regardless of whether they are remembered from a cache or created fresh we continue on and actually read the data from the buf:

   /* write the received buffer from the UDP socket to the memory-based input bio */
   BIO_write(for_reading, buf, rc);
   
   /* Tell openssl to process the packet now stored in the memory bio */
   rc = SSL_read(con, buf, size);
   
   /* at this point buf will store the results (with a length of rc) */

What if OpenSSL is in negotiation? If so, it'll have an outgoing packet to send that it queued into the for_writing buffer. So you need to check whether the buffer contains data, and if it does send it out.

   /* this should be done even if rc == -1 and no consumable data was received*/
   if (BIO_ctrl_pending(for_writing) > 0) {
        /* Read the data out of the for_writing bio */
        outsize = BIO_read(for_writing, outbuf, sizeof(outbuf));
   
        /* send it out the udp port */
        rc2 = sendto(sock, outbuf, outsize, 0, &sockaddr, sizeof(struct sockaddr));
   }

Sending a Packet

When sending a packet, we need to see if we have an open connection. If we don't have one (ie, it's a new destination) then we do something similar as we did above so I won't repeat it here: create an input and output memory bio and attach them to a new SSL * pointer. For a client, we don't technically need to do it this way because it would be easy to use a new DGRAM bio per server to talk to (each with its own fresh socket). It's not as tricky as the server isde where we can't do that as easily because there is only one socket.

   /* call OpenSSL to write out our buffer of data. */
   /* reminder: this actually writes it out to a memory bio */
   rc = SSL_write(con, buf, size);
   
   /* Read the actual packet to be sent out of the for_writing bio */
   rc = BIO_read(for_writing, outbuf, sizeof(outbuf));
   
   /* and send it out the udp socket */
   rc = sendto(sock, outbuf, rc, 0, &sockaddr, sizeof(struct sockaddr));

Caveats

This completely breaks cookie handling within DTLS. Ok, it doesn't break it but it defeats the purpose. Now we're keeping state for OpenSSL but the whole point of (D)TLS cookies is to not keep any state at all until the client proves themselves.

Other possible solutions

Writing a new UDP BIO

In theory, a new dgram BIO could be written that would fake an accept call and auto-fork a new BIO for new incoming "connections" (defined just like above using a unique combination of remote IP address and port number). The new BIO could still read and write from a single socket, but each BIO would have to do a PEEK on the socket to make sure the next incoming dgram message was from the expected source/port pair (or else don't read from it).

The problem with this approach is that it still requires select() to hit a single socket and after the select is done, you still have to map that single socket to a number of different BIOs that are all using it. And the only way to do that is to do a PEEK on incoming packet for the source/port pair and select the right BIO. IE, we're right back where we started. It doesn't actually buy you that much.

Eventual Solution and Architecture Changes for OpenSSL

??