This is just a PSA.  Because by most accounts, the HTC Vive is really picking up adoption.  And troubleshooting can be difficult because it’s new, and there isn’t much information out there.

There is a trend in a particular type of problem.  And there are many reddit/forum posts and such where people are figuring this out piece by piece.

I’m terming it Early Cyclops Syndrome.  I don’t have any data showing how prevalent it is. I just have the anecdotal evidence I found, trying to track down my problem.  And further, HTC’s own troubleshooting just doesn’t seem to cover it.

The problem seems to occur when one of the two sweeping lasers in a lighthouse unit goes dead.  The Vive comes with two lighthouse units.  Each of them has two spinning laser emitters.  One on a horizontal axis and one on a vertical axis.

There is also an array of LEDs in the unit.  Both the LEDs and the lasers are emitting Near Infra-Red (NIR) light.  You shouldn’t be able to see it.  Many people can see the LEDs dully with their eyes.  But very few if any people can see the laser with their eyes.

Anyhow, if one of the lasers in one of your light-house units stops working, then it has gone from two “eyes” to one.  Hence, “Cyclops” syndrome.  Yes, I understand the actual photo-sensors are not on the light-houses but on the HMD and controllers.  No, I don’t care. I’m still using the “eye” analogy.

The other commonality is that most accounts I’ve seen, have a similar temporal commonality.  Simply put:  The common story is that the Vive was working great.  And then two weeks in, it started having strange tracking problems.  This is why I coin it “Early” Cyclops Syndrome.  Obviously with the product designed as it is, one would expect the units to periodically go “cyclops” just because motors die.  Emitters fail.  But there seems to be something to this early, two week death that is common.  Much like hard-drives have a tendency to either die very early, or very late, but not so much in the middle. statistically speaking anyway.

The types of tracking problems vary.  Some people start drifting or flickering when they face certain directions.  Some people notice the lighthouse base-station that draws inside the VR environment is drawing in the wrong place.  There are very good techy geeky “structure from motion problem” answers to why all of this happens when the laser dies.  I’m not going to go into them.

All of these problems tend to move people toward “troubleshooting” the problem.  And eventually, if you go far enough, you wipe out the json configuration file in the steam directory.  Then, the system will be forced to admit it has no idea where that second basestation is at all.

If you pay special attention to the SteamVR app, it may tell you that the basestation isn’t tracking.

If you look in the logs, you may find an explicit statement that an axis appears to have failed.

But you can also directly observe the problem with a camera-phone.  This is because camera sensors are naturally very sensitive to NIR light.  And even though they have a IR Cut filter built into their optical stack, it’s not perfectly efficient.  As such, most camera phones are still somewhat NIR sensitive.  If you look carefully, you can see the lasers to the bottom and side of the LED array.  The following is a good lighthouse basestation with both lasers firing.

Bonus tip.  You can tell when your TV remote needs batteries because a camera phone can see the LED in the remote firing normally.  When the batteries are daed, you wont be able to see the LED firing codes anymore.  It’s an old, old trick.

Anyhow, to diagnose properly, I compared my lighthouses both in “A” mode.  Only one powered on at a time.  One lighthouse in “A” mode can track my HMD at my desk.  The other cannot.  When observed with my iPhone camera, the one that works reveals two functioning lasers as seen above.  The one that does not, only reveals a single functioning laser.  It’s a cyclops.

Poor thing.

So what’s wrong exactly?  I figured this out late on a Friday.  I’ve noticed others have had this issue and are in varying states of frustration, while googling around over the weekend.  As I write this, it’s Sunday.  HTC Tech Support isn’t open until Monday.  I suppose there’s still a chance that some obscure firmware update will get the dead axis spinning.  But I doubt it.  I’ve already gone through firmware update procedures documented elsewhere.

My best guess is that either the Emitter is dead.  Or the motor is dead.  Diagnosing further would involve cracking the case.  And since this thing is under warranty, I should not crack the case.  Nor should you dear reader, I’d assume.

My expectation is that HTC should RMA the unit and repair/replace it at no charge. Because it’s under warranty.   We’ll see how that goes.

Is this to be “cyclops-gate?”  I hope not.  Firstly, there is no hotel involved.  Second, faulty hardware happens.  Faulty components happen.  The only way this becomes a “scandal” is if HTC manages, as an organization, to make it into one by their response.  They’re a big consumer electronics company with a lot of resources.  They RMA hardware and handle bad suppliers all the time.  This should be no different.  They should be good at this.  But if I were a tech reporter, I’d keep an eye on it.

So, dear reader, if this helped you figure out what is or isn’t wrong with your Vive, I’m glad. Please keep in mind, there are a lot of troubleshooting steps to go through before you get here.  There are a lot of tracking problems that are not this problem.  But since you can actually deterministically take out your camera phone and diagnose this very real physical problem definitively, I though it was worth writing up.  Either this will confirm your problem. Or it will confirm you don’t have this problem. Either way, it’s useful to be able to know.

PSA complete.

Update 1

I held off on this update until I felt the matter was resolved.  I didn’t want to lead you, dear reader, down a needlessly complex path just to get answers.

The short version:  Two months later, I finally have a working lighthouse to replace my broken one.  HTC support was incompetent.  After multiple attempts to get them to RMA the hardware, I was forced to go through SteamVR support instead.

Valve was able to get HTC to do their jobs and RMA the hardware.  However, even Valve took weeks to successfully accomplish that feat.  They had to contact HTC multiple times because HTC wasn’t doing what they told Valve they would do.

After keeping on HTC further, I was able to force them through their RMA procedure.  And when they finally returned the “repaired” lighthouse, it was clear they sent me a new one.  The serial number on the “repaired” device is different than the one I sent in.

The takeaways follow.

To RMA or not to RMA?

Since I finally got my lighthouse returned, HTC has finally begun selling Vive accessories on their web-store.  These accessories include an extra lighthouse.  Honestly, if it’s going to take two months to get the thing repaired under warranty, I’d rather just take the loss and pay for the new lighthouse.  It’s not “right” to do that.  HTC should both honor their warranty and also be good at servicing that warranty. But as a software developer, I can’t afford to lose two months to HTC’s incompetence.  You, dear reader, will need to make your own decisions based on cost/opportunity/situation.

Valve or HTC?

It’s worth trying to get HTC to do their job first because that’s how it’s supposed to work. And evidence suggests that some people get lucky and get a competent rep.  But keep in mind, Valve is deeply involved in the Vive for the time being.  If you need someone to kick HTC around because they can’t manage the support correctly, you might want to submit support requests to Valve via the SteamVR product in Steam.  Steam’s support people are competent.  They give you ticket numbers.  You can follow up with them.  They seem to take pride in actually providing service and resolving issues.  Remember, you have an ally in Valve.  Because they, as a company, are just as frustrated with HTC as you are, if HTC is doing a bad job.

What’s wrong with HTC’s approach?

Optical tracking systems like this really have two levels of problem.  The first level is one of functionality.  The second is one of optimization.

If HTC understood their own product better, they’d realize that they need to separate these things.  Before one goes down the path of trying to help a user optimize their setup for bad tracking, one first needs to establish that the components are functioning.

Their support staff should of course be checking firmware versions and such.  And that can be done by dumping logs and uploading them.  HTC support asked for the logs, but then failed to execute based on them.  The logs showed everything was up to date.  There should have been no reason at that point, to demand that everything be uninstalled and re-installed unless further functional testing showed that despite what the logs said, the devices weren’t working.

And further, since the logs showed a hardware error that clearly stated what the problem was, they should have recognized it.  They didn’t use the log data at all it seemed.  They just had me give it to them and then they had me do a bunch of useless things, and failed to move in a useful or logical direction.  They failed to find the functional problem and moved into optimization troubleshooting.

Next though, assuming a clean and up to date log, you need to establish that each lighthouse is functioning, and that they can see the HMD.  This again, is a functionality test to be established before moving into optimization.

Placing a single lighthouse in ‘A’ mode directly facing the HMD about 5 feet away with a clear line of site is a great simple test to determine clearly and simply, that the lighthouse works and the HMD works and tracks.  Following up with the same test for the second lighthouse verifies both of them. Adding a controller verifies the controller. Etc.  Notice, you can troubleshoot all kinds of hardware issues from this very simple place.  Like say, fixing a USB bus problem by having the user plug into a USB 3 port.

Only after the baseline functionality of all the hardware in the system is established, should the script start to descend into the murky waters of optimization of room geometry, and reflections, and the height and mounting of the lighthouse, and all that wonderful stuff.

At least that way, you can identify bad hardware and service it.

Hopefully HTC will realize this and adjust accordingly.  Otherwise, they risk creating a bad experience for early adopters.  Which will damage their product-line far into the future.

Tricks

• If you go through Valve, and valve hands you off to HTC, make sure Valve gives you a ticket number from HTC.  Even if HTC says they’ll contact you.  Often, they’ll fail to do so.  But, if you have a ticket number, then you can contact HTC yourself with the ticket number and force them to address it.
• HTC will not reliably address Vive issues through any method other than their web-chat.  If you contact the main HTC support line via phone, with your ticket number, they can see the ticket, but they are not authorized to help you.  They will tell you to use the web chat.  HTC’s email is extremely spotty in both directions.  Things they send you will not show up.  Things you send them will not show up.  Use the web chat.  Force them to take info from you in web chat.  Request all information be given to you through web chat.  Write it all down.  The email they say will come, may never come.
• If you give them enough time to do a thing, and the thing hasn’t happened, get on web-chat with your ticket number and follow up.  They seem to drop threads of action constantly.  The next rep may actually get the job done.
• Once you get far enough down the RMA process over web chat, you will eventually have to depend on e-mail because HTC hands you off to some kind of regional partner, who isn’t actually HTC.  You can tell when this happens because you get a new ticket number, and suddenly, you can track your case in a separate web-portal.
• Try not to lose your cool.  I did lose my cool once.  I feel justified given how they were handling me.  But you probably should be better than I was.

Update 2

It’s been a while since this post originally went up.  And we’re no longer “early adopters.”  Many of these units are off warranty.

I was contacted by a reader, John Landrum about his experience trying to fix such a broken unit.  His experience is worth relaying:

I ordered a spare “new” lighthouse from e-bay which came with a dead emitter (whether or not the item was actually new or not I don’t have a clue.) So I replaced the faulty motor/emitter in my original station with the good motor/emitter from my ebay purchase and when I reassembled everything the same problem existed (motor spins, emitter fails to light.)

The possibilities were:
-I’d seated the replacement component improperly and it needed to be disassembled and reseated.
– There was never anything wrong with the original motor/emitter
– Both

The second possibility was interesting to me. As I’d not yet reassembled the unit I’d used to cannibalize the part for the original, I decided to put the component I believed to be faulty back into the spare unit so that it should have two faulty tracking lasers.

The lasers from the original faulty station were marked with marker and the new faulty station’s lasers were unmarked. Having now replaced the faulty horizontal tracking laser form the original unit with the good horizontal tracking laser from the new unit. I fired up the new unit which now theoretically had a faulty horizontal from the original system and a faulty vertical laser which was left alone.

Low and behold the faulty laser from the original lighthouse when placed in the working spot on the new lighthouse… worked.

So it appears that the problem causing the cyclops syndrome has nothing to do with the motor/laser itself but something on the controlling board.

I’m not sure I’ll have the time to track down the issue any further but I thought this might be of use to anyone encountering this problem with a unit that isn’t under warranty any longer.

What I’d add to John’s analysis is simply that there are likely more than a few causes involved here.  Many units have bad motors.  Many have bad emitters.  Many have bad assembly and quality control issues.  And John seems to have confirmed that some (or maybe many) have upstream issues on the main board.  My instincts tell me to check out the capacitors on the main board for example.  As bad/cheap capacitors are a common problem in electronics in general.

My earlier advice was of course, not to crack the case, and to rely upon the warranty service.  Even if said warranty service was hard to make happen correctly.  But as John points out, it’s been a while and many of these units may be off warranty.  The information he provides can be useful for troubleshooting.  And therefore, it’s worth adding to the pile of information.  Thanks John!