Re: Auto-update protocol



On Wed, 10 Mar 2010 23:56:44 -0700, D Yuniskis
<not.going.to.be@xxxxxxxx> wrote:

George Neuner wrote:
On Mon, 08 Mar 2010 12:34:53 -0700, D Yuniskis
<not.going.to.be@xxxxxxxx> wrote:

The projects that I am "practicing for" have much larger images
(1 - 4G) -- though generally have more resources to call on
at run time, so... <shrug>

I'm currently playing on a 100Mb wired network. But, I am
looking for a solution that will scale up to Gb and down to
"wireless" rates.

What is the overall structure? Does the new image reset the device or
is it a hot patch that needs to continue using existing data
structures?

Everything tries to run hot if at all possible. E.g., if
conditions coincidentally arise that would cause some piece
of code to rely on resources that aren't available during the
update, then things might not quite work as they would if the
update process was not active (but this would be unlikely).

Is it feasible to have the client load a new image concurrently with
the old and switch at some well defined point(s)?

Not the whole image. I am structuring the application so I
can swap out pieces of it and update it incrementally. E.g.,
update the RTOS, then switch it in; update parts of the
library, then switch them in; etc.

That's definitely the right approach, particularly if you want to
scale to low wireless rates ... on 100Mb WiFi a 1GB file takes minutes
to transfer even with a single client and no interference. You can
just forget about using something like 3-G cellular.


- The protocol can't be spoofed by "unfriendlies".

There's only one real solution to spoofing and/or "man in the middle"
attacks and that is to encrypt the image binaries. You want it to be
as hard as possible for someone to decompile your application and
figure out how to spoof it.

The sources for the application will be available.

So, shared secret/public key is the only way to do the encryption.
The problem there is it forces all devices to have the same
keys (or, I have to build special images for each device).

Multiple devices sharing the same key is not a problem. Although one
of the keys is "public" and the other "private", it actually doesn't
matter which is which: public(private(text)) == private(public(text)).

The point is to authenticate the origin of the image - the client's
"public" key can only decrypt a file encrypted by your private key -
successfully decryption is the authentication. (Of course you need to
verify by signature that the decryption was successful.)

You only need to worry about more complicated key handling if you are
creating programming tools for end users that themselves will need to
worry about spoofing. AFAIHS, your previous comments didn't indicate
that.

Regardless, it neither desirable nor necessary to use public key
encryption for the whole image (or its parts). Symmetric single key
encryption (like DES, AES, etc.) requires fewer resources and runs
faster. The only parts that require public key for secure
communication and authentication are the symmetric key(s) and image
signature(s).


The more realistic problem is to guard against something
messing with the image or process and effectively leaving
the device(s) in "half updated" states. So, it is even
more important that updates be supported "in pieces"
(otherwise the window of opportunity for screwing things up
is just too big)

Building on the idea of embedding the signature in a header, you could
also embed a decryption key for the image. The header itself should
be encrypted with a public key algorithm so that the client can verify
that _you_ created the image. (Obviously the whole image could be
public key encrypted, but it isn't necessary ... symmetric key
algorithms are more performant. Successfully completing the process
of decrypting the header, extracting the image decryption key,
decrypting and verifying the image signature proves that the whole
thing is legitimate.)

Yes, but it would have to be done for each "piece".

True, but signatures for all the pieces can be bundled for en-mass
checking. If the pieces need to be encrypted, you only need to
provide one decryption key for all of them.


And don't store the decrypted image on the device ... design the
update process so that the local image can be decrypted into RAM and
reuse it to do the startup.

Huh? I *think* you are assuming there is enough RAM to
support the decrypted image *and* my run-time data requirements?
If so, that's not the case -- far more code than data.

I only assumed that the device stored the program for cold start and
that you wanted to keep the code secret. However, since the source
will be available there is no particular reason to encrypt images at
all - runtime image encryption is typically only done to protect trade
secrets.

What I was saying was that the decryption mechanism could be reused in
the general program loader, enabling the image(s) to remain encrypted
on storage. That way, it would be much harder to gain access to and
reverse engineer the code.


I'm not too worried about people reverse engineering devices.
Rather, I am more concerned with someone having access to
the medium and screwing with (spoofing or otherwise) the
update "undetected" (consider the wireless case as well as
how easy it is to have a hostile client sitting "somewhere"
on a network...)

So then the only things that require authentication are the image
signatures. You don't need to encrypt the image binaries at all -
just the image signature bundle.


- The protocol should be as light as possible -- but
no lighter! ;-)

This feeds back to the size of the images. TFTP is about as simple as
you can get using a generic farm server, but TFTP does not have any
way to request a partial transfer (e.g., a "head" command). Of
course, for version checking, you can hack your client to start the
transfer and then abort after fetching the predetermined image header.

I was figuring I could just have the client request whatever
blocks it wants in whatever order, etc. E.g., TFTP doesn't
*force* me to request blocks in sequential order...

It has been a long time since I've read any of the relevant RFCs, so I
may be completely wrong here (and please feel free to correct me 8),
but AFAIK, none of the popular file copy/transfer protocols allow
random access.

My understanding is that protocols like FTP, TFTP, Kermit, etc. just
send file blocks in order and wait for the receiver to ACK/NAK them.
Some of the sliding window implementations allow OoO within the window
but, AFAIK, none allows arbitrary positioning of the window or
advancing the window beyond a missing block. The so-called
"restartable" protocols periodically checkpoint the file/window
position and restart from the last checkpoint if the same client
re-attaches and requests the same file within some timeout period.

If you want/need essentially random access without needing your own
server application you might be forced into a remote file system
protocol (like SMB, NFS, etc.). I have no experience implementing
such protocols ... I've only used them.


- Minimize unnecessary network traffic as well as
load on the server ...

What I am more concerned with are the apps that follow.
Those are so big that updates can be much more frequent.

True, but that situation is mitigated by piece wise updates.


The more pressing issue for those apps is turnaround time.
I.e., the update would want to start as soon as it was
available (on the server). So, checking "once a day" would
be inappropriate. Once minute might be nice as the expected
delay to update start would only he half that (recall that
those devices would need much longer to perform their entire
update).

Downloading a small header (up to 1 UDP block) once every couple of
minutes would not be too bad - at least for a limited number of
devices and images (I'm presuming here that "like" devices can use the
same image?).

Obviously it would be far better to broadcast/multicast that updates
are available and forgo client polling at all ... but that requires an
intelligent server.



The idea (mentioned elsewhere in this thread) of having
devices collaborate to:
- reduce the polling rate "per device" to keep the overall
polling rate (seen by the update server) constant
- share discovery of valid updates with other devices

Couple of possible solutions:

- have all the "like" devices on a network segment (i.e. those that
can share an image) cooperate. Use a non-routable protocol to elect
one to do the polling and have it tell the others when an update is
available.

- run the devices in promiscuous mode and have them "sniff" update
checks. This way when any device on the net polls, all devices on the
segment see the result.

- combine election and sniffing so only one device actually talks to
the server and the others eavesdrop on the conversation.


Sniffing might be more complicated than it's worth - ideally you'd
like to extend it to cover the whole update process. But it may be
worth doing for update checks because the file name, block number,
etc. could be fixed and so easy to search for in sniffed packets.



Use a "well known" public name and hide the file updates behind hard
links.

Changing a hard link is an atomic process so there will be no race
conditions with the update - a process trying to open the file through
the link will either get some version of the file or fail if the open
call sneaks between unlinking the old file and linking the new.

So updating the server then is a 3 step (scriptable) process. You
upload the new version alongside the old, change the well known hard
link to point to the new version, then delete the old version.

<frown> Requires someone do something "special" on the server.
When that someone (IT) screws up, The Boss doesn't see it as *his*
problem. The IT guy *claims* he did everything right and there
must be something wrong with the update software or the devices.
So, *vendor* gets a frantic call from an angry customer complaining
that the system has been "down" for "over an hour now"...

Yes. But the key is that it is scriptable. Bleary eyed, overworked
IT person or dumbass noob makes no difference ... it's hard to get
much simpler than "<script> filename". If you must have a GUI you can
use Javascript in a web browser so the (l)user only needs to specify
the file (or better yet drop it on the browser page).

In any event, the server update process can be stupid friendly.

George
.