Re: Mars Rover Not Responding - Update - NASA Makes Contact With Mars Rover

From: del cecchi (dcecchi_at_msn.com)
Date: 01/25/04 

Date: Sat, 24 Jan 2004 22:11:43 -0600

>From a website "spaceflightnow.com" (still sounds like software)

Rover project manager Peter Theisinger gave the following update on
activities to diagnose Spirit's ailment and return the craft to working
order:

"We made good progress overnight and the rover has been upgraded from
critical to serious. We have a working hypothesis we are pursuing that
is consistent with many of the observables and consistent with
operations that we performed on the vehicle last night. It involves the
flash memory on the vehicle and the software used to communicate with
that memory.

"The processor on Spirit has three kinds of memory:

"There is the random access memory just like the memory in your
computer, which is used basically in a real-time mode. And that memory
is volatile, so when we turn off the rover at night that memory goes
away.

"We have flash memory. That memory is just like the memory in a digital
camera. It can also be read to and written from easily. But it has
non-volatile characteristics -- the information that is stored there
stays overnight even if the vehicle is powered down.

"And then we have we call double EPROM, which is
electrically-programmable memory. That is more difficult to write to and
read from, and we use that to store part of the flight software image.

"The vehicle normally uses the flash memory mostly for the storage and
retrieval of engineering and the scientific telemetry. The software has
to communicate with the flash memory -- has to open up files, establish
file directories, close files in that flash memory. That process has to
work correctly.

"We are capable of operating the vehicle without going to the flash
memory in what we call a 'cripple mode.' That is name we have chosen --
you should not read too much into that. That basically tells the flight
software that when it boots up, it should operate with its file
directory out of the random access memory rather than the flash memory.
That would avoid any issues that we might have with either the flash
memory itself or the flight software that is used to write to it.

"Let me talk about the chronology of what happened in the last 24 hours.
If you recall when we last talked at yesterday's press conference, we
were attempting to shut down the vehicle because the batteries were
becoming depleted. We had an apparent inability to shut down the vehicle
last night, yesterday at the end of the Mars day, which is about the
time of the press conference at 9 a.m. Pacific. That was in fact
confirmed because later we had a UHF communications session with Odyssey
where we got 73 megabits of data, mostly fill or garbage data, although
we did get some fault data, some current, some 14 hours old.

"We did not know but thought we might go into low-power overnight if the
batteries were fully depleted. When came up this morning we looked for
the 9:30 Mars time communications window and it was not there,
indicating, we thought, that we had gone into low-power. That would
cause the vehicle to come up at 11 o'clock and tell us that.

"We, just prior to 11 o'clock, sent a command to the vehicle that said
'go into this cripple mode.' That is only done at the next reset, so
then we sent a command to the vehicle that said 'and reset' in order to
do that. And we timed it so that when the 11 o'clock session would
start, we would begin to get that session at 10 bits per second
indicating we'd gone into low-power. When the commands reached the
spacecraft light-time away, it would send us into cripple mode, reset
the computer and we would come up in the mode. And in fact that is
precisely what happened.

"At that point, we commanded a one-hour communications session at 120
bits per second. That communications session happened as planned.

"The progressive set of resets that we were getting into every hour did
not reoccur, leading us to conclude that our hypothesis is largely
correct -- that is there is something involved in the flight software
that talks to the flash memory that's causing this difficulty.

"Why that might cause us difficulty is because when the spacecraft first
wakes up it needs to communicate with the flash memory to establish a
file structure and when it goes to sleep and shuts down, just like you
shut down your computer at home, it needs to go out to that memory and
close all of those files and clean everything up. If it is unable to do
that, it would not complete those tasks appropriately and will basically
reset itself and not shut down.

"In the midst of that 120 bits per second, one-hour session, we decided
to shut down the vehicle in order to replenish the batteries. We
commanded the shutdown just prior to the end of the communications
session so that we would see the communications session terminate early
if we were able to operate it correctly. That happened. And we sent two
post-shutdown beeps, which we expect not to hear if it is asleep but we
would hear if it was not asleep, and we did not get those, once again
confirming that the vehicle to the best of our ability to determine is
now sleeping on Mars.

"We also yesterday as part of the command sessions during this period of
time terminated tonight's UHF passes and reset the uploss timer. The
uploss timer is a set of fault code that go into play when the vehicle
thinks it has a communications problem and causes some things to switch
state and we didn't want to get into that if we could avoid it. Both of
those were confirmed to be successful.

"So we have a vehicle that is stable now in power and thermal. We have a
working hypothesis that we have confirmed. The fault protection to the
best of our estimation has worked as designed. It took us a lot to
figure out what was going on, but we think everything has worked in the
fault protection as we expected it to do.

"We have a go-forward plan. The cripple mode, which we can use every
day, needs to be re-established every day because it loses the memory
that that's the way it should start up. So every morning we will need to
start up in cripple mode, so we need to establish an operational way of
doing that every day for the next few sols (Mars days).

"We need to establish a high-rate link in order to be able to get much
more data back, particularly if we want to read out the flash memory and
determine what has happened. What we will plan to do is likely to use
the Odyssey afternoon relay pass for that purpose prior to the afternoon
shutdown.

"We need to then establish the contents of flash to find out what
happened and then we need to move forward with the diagnosis and
recovery of the vehicle capability based upon what we find there.

"Remember that this was all kicked off by Sequence 2502. That was the
sequence that was using the elevation motor in the mast failing out,
failing to complete (on Wednesday). We still do not know the details of
why that happened and we need to do that.

"The mission consequences of this are uncertain at the present time. But
I think that we feel that we probably have more capability left in the
vehicle that we can establish than the worst-case scenarios by quite a
bit. We still see a couple of weeks to determine what had happened and
to rebuild our confidence into what is working on the vehicle and to get
back closer to routine operations. I think we are probably like three
weeks away from driving, I am guessing.

"The team will begin to go into double-shift operations probably a day
or so after Opportunity's landing where we have a re-plan period and
then an operational period as we begin to work through the forensics of
this.

"But this is a very good news. We've established an ability to
communicate with the vehicle reliably, we've established that in fact we
do have controllability of the vehicle, can establish a good power and
thermal state, our working hypothesis is one that we can work around
with significant measure if it turns out our working hypothesis is
correct. So a good day for an Opportunity landing."

Relevant Pages