Better SPI Bus Design in 3 Steps

Most Arduino SPI tutorials show this simple but poor SPI bus design:

Better SPI bus design can prevent conflicts.  3 simple improvements are needed:

  1. Use pullup resistors on all chip select signals.

  2. Verify tri-state behavior on MISO: use a tri-state buffer chip if necessary.

  3. Protect bus access with SPI.beginTransaction(settings) and SPI.endTransaction().

Step 1: Pullup Resistors for Chip Select & Reset Signals

When multiple SPI devices are used, and especially when each is supported by its own library, pullup resistors are needed on the chip select pins.

Without a pullup resistor, the second device can “hear” and respond to the communication taking place on the first device, if that second device’s chip select pin is not pulled up.  This is easy to understand in hindsight, but it can be temendously confusing and frustrating to novice Arduino users who purchase shields or breakout boards without pullup resistors.  Each SPI device works when used alone, but they sometimes mysteriously fail when used together, only because both devices are hearing communication meant to initialize only the first device!

A simpe workaround for devices without pullup resistor involves adding code at the beginning of setup.

    void setup() {
      pinMode(4, OUTPUT);
      digitalWrite(4, HIGH);
      pinMode(10, OUTPUT);
      digitalWrite(10, HIGH);
      delay(1);
      // now it's safe to use SD.begin(4) and Ethernet.begin()
    }

Step 2: Proper MISO Tri-State Behavior

Most SPI chips will tri-state (effectively disconnect) their MISO pin when their chip select signal is high (inactive).

However, some chips do not have proper MISO tri-state behavior.  Fortunately, checking MISO tri-state is easy, especially when prototyping on a breadboard.  Just connect two 10K resistors to the MISO line, like this:

When all SPI chips are disabled, the MISO signal should “float” to approximately half the Vcc voltage.  If any device is still driving the MISO line, you’ll see a logic high (usually close to 3.3V or 5.0V) or logic low (close to zero volts).  This test is so easy, it should always be performed by designers of Arduino compatible products.

Arduino shields and breakout boards with poorly-behaved chips should always include a tri-state buffer.  Adafruit’s CC3000 breakout board is a good example:

Step 3: USB SPI Transactions in Software

Newer versions of Arduino’s SPI library support transactions.  Transactions give you 2 benefits:

  • Your SPI settings are used, even if other devices use different settings
  • Your device gains exclusive use of the SPI bus.  Others will not disturb you.

These improvements solve software conflicts, allowing multiple SPI devices to properly share the SPI bus.

A typical use of transactions looks like this:

    SPI.beginTransaction(SPISettings(14000000, MSBFIRST, SPI_MODE0));
    digitalWrite(chipSelectPin, LOW);
    SPI.transfer(mybyte1);
    SPI.transfer(mybyte2);
    digitalWrite(chipSelectPin, HIGH);
    SPI.endTransaction();

SPI.beginTransaction() takes a special SPISettings variable, which give the maximum clock speed, the data order, and clock polarity mode.  The speed is give as an ordinary number, expressing the maximum clock speed that device can use.  The SPI library will automatically select the fastest clock available which is equal or less than your number.  This allows your code to always use the best speed, even on board with different clock speeds.

If your code will ever call SPI library functions from within an interrupt (eg, from attachInterrupt), you must call SPI.usingInterrupt().  For example:

    SPI.begin();
    SPI.usingInterrupt(digitalPinToInterrupt(mypin));
    attachInterrupt(digitalPinToInterrupt(mypin), myFunction, LOW);

If you are developing a library that must be compatible with older versions of Arduino, which lack these SPI transaction functions, you can use SPI_HAS_TRANSACTION to check for the new version.  For example:

    #ifdef SPI_HAS_TRANSACTION
    SPI.beginTransaction(SPISettings(2000000, LSBFIRST, SPI_MODE1));
    #endif

Please Share and Use This Information

Many SPI-based products for Arduino do not work well together.  My hope is this information can help all makers of Arduino compatible devices to achieve much better compatibility.

Long-term, sharing of knowledge is needed.  Please share this information and ask makers of SPI devices and libraries to consider these suggestions.

This article may be shared and copied under the terms of the Creative Commons Attribution 4.0 International License.  Please, copy & share!  🙂

 

This article was originally published on the DorkbotPDX website, on November 24, 2014.  In late 2018, DorkbotPDX removed its blog section.  An archive of the original article is still available on the Internet Archive.  I am republishing this article here, in the hope it may continue to be found and used by everyone using SPI chips, and especially companies making SPI-based products for the Arduino community.

 

 

 

Control Voltage to 1.2V Analog Input Pin

Often the question is asked, what is the simplest way to get modular synth control voltage (CV) into an analog input pin?

This simple circuit using only 3 resistors and 1 capacitor converts the -5V to +5V CV signal range to the 0 to 1.2V ADC input range.

For Teensy 3.2, 3.5 and 3.6, you would use analogReference(INTERNAL) to configure for the 0 to 1.2V range.  Then you can read the CV signal at any time with analogRead().

While this circuit is the simplest and easiest way, it is not necessarily the best way.  The input impedance, or load placed on the CV signal, it 31K ohms, rather than the standard 100K impedance normally used for modular synthesis systems.  This forum discussion goes into the circuit’s limitations and suggests more advanced ways using opamps.

 

Originally this article was published on the DorkbotPDX website.  Since that time, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve this info.  A copy of the original can also be found at the internet archive.

Light Table For Web White Background Photos

Years ago, in my slow quest for better photography of electronic projects, I built a light table to eliminate shadows.  Most of the white background photos you see on the PJRC site are shot with this light table.

This is how it turned out.

The build used boards from OSH Park (then “Laen’s PCB group order”), materials from TAP Plastics, 15 white LEDs and parts I mostly had laying around.

This view is inside a cheap 2 foot sized light tent I purchased from some ebay vendor, and a couple bright lights outside the tent on both side.

The LEDs are Cree CLM3C-WKW-CWBYA453, which are supposedly the same 5500K color temperature as the CFL lights outside the tent. Maybe that matters, maybe not, but it seemed like a good idea.

The Cree LED is a surface mount part, but fortunately Lean’s PCB group order made it very easy to convert to something I can solder wires onto. All the PCBs mount with double sticky mounting tape.

As you can see in this LED photo, there’s a bit of shadow. It’s a soft shadow due to the light tent casting light from many directions, but it’s still very present.  This is the type of shadow I’m hoping just a little bit of under side lighting will eliminate.

This little board is a constant current regulator. It takes a 0 to 5 volt input and regulates a 0 to 20 mA output current to a string of 5 LEDs. I wanted to make sure the current was perfectly constant since the camera might choose a quick shutter time.

Here’s the schematic for that circuit. At the time, it seemed like a good idea to sense the current using a resistor between the NPN transistor’s collector and the LEDs. The idea was any small change in ground potential between the board 0-5V control signal wouldn’t matter, if I ran separate signal and power ground lines.

But I didn’t consider the current draw though those resistors around the upper opamp. As you can see in the schematic, I change the values to about as high as I reasonably could. It still have a tiny bit of the lowest part of the range where the LEDs won’t completely turn off.

The 0 to 5 volt signal just comes from this potentiometer on the front panel.  Because it’s driving only the inputs to opamps, it doesn’t have any substantial load.  I still used a 1K pot anyways, though a higher value would have consumed a little less current.  I guess I didn’t care about an extra 5 mA.

The power for everything comes from this simple little power supply, which is just (approx) 24 volts unregulated, and a 5 volt regulated output from the pot, which is from a 7805 regulator.  Simple.

Of course, the opamp circuit isn’t perfect. After putting this together, I decided to try a different approach, sensing on the emitter side, and no current sensing path to add to the LED current! I also included 4.7K resistors on the feedback looks, and the positive inputs see about a 1K impedance. Any errors from the opamp’s input (PNP) bias currents should be small, and should be more on the negative than positive, so hopefully any tiny error will tend to reduce the LED current, not increase.

Then again, the original boards might work out ok, but Laen’s PCB group order makes it so very easy to just quickly draw up a (small) board.  Because the cost is so low, it’s easy to just send it off without all the worry the goes into a normal board order.

I still haven’t actually put this thing to use… the top plastic cover turned out to be just a bit too small, so I need to go shave it down to size on my table saw (which currently has a bunch of other project stuff piled on top of it). But soon, with a little luck, I’ll be able to take pictures of electronic stuff and adjust the light table to null out or at least soften away most of the little shadows that I still get, even with the light tent.

 

Originally this article was published on the DorkbotPDX website.  Since that time, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve this info.  A copy of the original can also be found at the internet archive.

Measuring Microamps & Milliamps at 3 MHz Bandwidth

Some time ago, I needed to actually “see” a current waveform in the 100 uA to 5 mA range with at least a couple MHz bandwidth.  This extremely expensive probe would have been perfect, but instead I built something similar for about $30 using the amazing Analog Devices AD8428 amplifier.

The first step was cutting the power trace and adding a resistor.  I used two 1 ohm resistors in parallel.

At 5 mA, this makes only 2.5 mV.  My scope’s supposed resolution is 1 mV, but the truth is there’s plenty of noise down in the 1 mV range.  That’s pretty common for most scopes, even pretty spendy ones.  So it’s just not feasible to measure this signal directly (not to mention using 2 probes and subtracting them in the scope).

That incredibly expensive Agilent probe probably has a couple really nice amplifiers inside…. so I went searching for an amplifier.  After a bit of searching, I found the AD8428.  It has a fixed gain of 2000 and a bandwidth of 3.5 MHz.  That’s a gain-bandwidth product of 7 GHz !!!  It’s also an extremely well matched instrumentation amp with an amazing CMRR of 140 dB.  So it gets rid of the power supply voltage and outputs the amplified signal referenced to ground.

The AD8428 is perfect.  It’s so very easy!  Of course, such amazing performance costs money: about $20.  Here’s that expensive little amplifier, and a 5V to +/- 15V power supply (about $10) to power it.

The one trick with measuring such tiny voltages is twisting the 2 sense wires together.  Honestly, I didn’t try it running them separately, but since this thing is getting voltages in only the microvoltage range for the lower measurements, I didn’t want to risk picking up noise.  I also put a 100 ohm resistor on the output, just in case I accidentally short the output or do something clumsy that might blow that little $20 part.

Here’s a scope screenshot using this little amplifier to “see” the current (the blue waveform).

In this test, the microcontroller is running in its slowest mode at only 10 kHz, drawing about 120 uA.  Then when the chip’s internal oscillator is started, the current jumps to about 600 uA.  Later, the CPU switches to actually clocking from that oscillator.  There’s an on-chip clock divider which is switched in and increased gradually.

The bottom trace (red) is the voltage on the chip’s 1.8V linear regulator.  It turns out that sudden jumps in current cause pretty substantial downward spikes on the regulated voltage.  This more gradual startup approach really helps.  This sort of thing is impossible to see with a slow multimeter, but with a reasonably good bandwidth measurement of the current, it’s easy to see what various code actually does to the current.

I tried connecting my multimeter to the amp output.  Sometimes it’s just a lot more convenient to look at a single number on a meter than fiddle with the scope.  I had been using the current mode on the meter before building this.  One thing I was surprised to find it my little meter updates its screen much faster while measuring about 125 mA than it does when measuring 125 uA.

Another interesting thing I’ve been noticing is patterns within the blue current waveform.  This Agilent scope has a “digital phosphor” rendering of the huge amount of data it collected.  This static screenshot can’t really capture the interactive experience of adjusting the waveform intensity, where various regions within the data change brightness differently, indicating there’s something interesting/different going on.  Even so, you can see several areas in the screenshot where interesting things are happening once the CPU is up and running.  It’s interesting how the current waveform changes as different code executes.

I know this isn’t anything terribly impressive… basically just buy a high-end amplifier and use it with a series resistor.  Maybe it even reads like an Analog Devices ad?  I’m not affiliated with Analog Devices… I just bought this part at normal qty 1 pricing from Digikey.

Still, this is the first time I’ve ever really looked at such low microcontroller currents with a few MHz bandwidth, and I’m guessing not many people have ever bothered to really measure such currents, so I thought I’d share.

 

Years ago I originally posted this article to the DorkbotPDX site.  Hackday published an article about it at the time.  Since then, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve the info… for the next time I or others might need to do this sort of current measurement on a budget.

 

 

 

 

 

 

How To Get Tech Help From Strangers On The Internet

Online tech forums can feel intimidating.  Three simple things can greatly improve your experience & odds for useful help, regardless of your skill level.

Good First Impression

“You get 1 chance to make a good first impression” is timeless wisdom. Strangers will quickly form an opinion of you, based only on the words, images or video in your message.  Make your words count!

Showing you’ve made an effort, more than any other factor, will influence people to help.

But if you’re an absolute beginner, what sort of effort can reasonably be expected?  Anyone can try a Google search using the words from their question.  Many experts have long forgotten how difficult finding relevant info can be when you don’t know the proper terms, or unrelated tech has since started using those words.  Just explaining the search you tried and confusing or off-topic info you’ve seen can go a long way towards helping seasoned experts to understand your struggle.  The results aren’t important, your personal effort is what matters!

Genuine interest to learn also tends to make a good impression.  Even a few words can really communicate your attitude.  Frustrated but determined to learn from a trying experience can make a great impression.  Very smart people who can help the most tend to appreciate genuine & candid communication.

Of course if you have started a project or done work, mention it.  Or better, show what you have tried.  Screenshots, photos or even a quick video can powerfully demonstrate what you’ve done, and cover the other 2 aspects of a successful forum question.

Even just a few extra words, which show you are making a sincere effort and sincerely wish to learn, can make an improvement in the response you’ll likely receive.

Context Brings Understanding

Humans have amazing capability to genuinely understand.  Experts can sometimes apply their tremendous knowledge towards creative ideas to solve your problem… when they actually understand what you really need.

Providing context in your question is key.  What are your larger goals?  How does the question you’re asking fit into your project?  What do you ideally hope to accomplish?  And why?

When working on a technical matter, it’s easy to focus on only the issue at hand.  Remember, good people with vast knowledge & experience regularly read forums, just because they enjoy helping others.  When composing your forum question, don’t forget to give context.  It can make the difference between answers that are at best technically correct and answers that are truly insightful.

Details Are Essential

Modern technical issues can involve a mind boggling number of details.  How much raw information should your question include?  Before posting, try asking these questions:

  1. Can readers SEE or EXPERIENCE your problem?
  2. Can readers REPRODUCE your problem?

It’s easy to joke “I saw an error, but clicked OK without reading”.  But seriously, a basic level of detail allows everyone reading your message to see the problem.  A screenshot or exact copy of an error, plus specific information about software, hardware, and steps taken are a baseline for your readers to merely attempt to see the problem as you did.

Ideally, people reading your message would be able to reproduce the problem.  Providing this level of detail is not always practical.  But if you can give enough info, odds of someone solving the problem are vastly improved.

Software programming questions should include complete source code, not merely the small fragments.  Often code issues involve subtle details, such as declarations of variables elsewhere in the code.  Some experts can spot these sorts of problems with amazing speed, if you show complete code!  Don’t forget to be specific about any extra libraries used.

Electronic hardware issues usually need photos or accurate wiring diagrams, and links or part numbers of materials used.

It’s common to feel nervous or self-conscious about showing your project details on a forum.  Don’t be shy.  Experts who regularly read the forum genuinely want to help.  Show enough detail, give context and demonstrate you’ve made some effort and odds very good they will help!

Common Pitfalls

Forums don’t always work ideally.  Usually the 3 steps above will lead to good results, but there are a number of common problems to keep in mind.

Too Much Diagnosis

Technical problem solving involves observing the effects of an issue and trying to deduce the cause.  A very natural human tendency writing while in this mindset is to focus moreso on the presumed causes than objectively sharing the observed effects.

Usually a short break between an intense debugging session and composing your forum post can make a tremendous difference.  When writing, try to think of your reader’s perspective.  Are you asking people to help diagnose the problem, or merely asking them to nod “yup, you’re right” in agreement with your existing conclusions?

However, sharing your thought process can avoid people needlessly covering the same ground you’ve already investigated.  There is no one perfect way.  Just keep in mind that too much diagnosis can shut people out of helping you troubleshoot.

Venting Frustration

Let’s face it, some technical problems are really hard, even intensely frustrating.  When you’re annoyed, venting frustration is a very natural human tendency.

Before you post on a forum, creating that all-important first impression with strangers, review your message.  Some level of expressed frustration is normal and maybe even helpful.  It can show you’ve really tried.  Just beware the common trap, where your forum message ends up being perceived as a rant rather than a question or request for help.

Nobody Replies

Even under ideal conditions, sometimes forum posts get no replies.  This can feel disappointing & disheartening.  Try not to let that emotion take control as you compose a reply to “bump” the thread.  You can do much better.

Effort matters.  If days or weeks have elapsed since your first post, presumably you’ve put more effort into the project?  Even if fruitless, the personal effort you’ve invested since your prior post is something you can share that tends to draw people in.  Photos or code or other clear signs of work on your part can be highly effective.

Sometimes very difficult questions, with plenty of detail, get no replies simply because nobody knows the answer.  Or nobody knows with certainty.  Asking for opinions or ideas for directions to try can really help in these cases, turning what started as a solid question into a more fluid conversation.  Again, apparent effort is the key factor that attracts people to contribute towards helping solve a tough problem.

Most forums discourage or even ban posting duplicate copies of your question, or “cross posting”.  Posting the exact same message in multiple places at the same time is almost never a good idea.  Experts who regularly read the forums, the people most likely to help you, will almost certainly notice.  You get one chance to make a good first impression.  However, after a period with no answers, sometimes duplicate posting can be done gracefully, usually with an explanation & link to the prior unanswered question.

Academically Dishonest Students

Every tech forum gets lazy students asking people to do their homework.  Experts who regularly read forums are inundated by these low-quality questions.  Like an email inbox with spam, the last thing you want is your message to be mistaken for more junk & quickly dismissed.

If you are doing a student project, usually the best approach is to be honest about the academic nature (and deadline) and to make sure your sincere effort shines through.  Earnest effort and genuine desire to learn is what distinguishes good students in the minds of experts who will actually help!

Proprietary Projects

When you can’t share source code or other relevant technical details, technical forum help is rarely effective.

Usually you must invest extra effort to separate just the one piece that is problematic from the rest of your project, so it can be shared.  If you have not done this, consider that asking “has anyone seen a problem like XYX” is essentially asking humans to function as a search engine.  Sometimes the results are good (better than you could do with Google), but difficult technical problems without sharing relevant details are rarely solved by blind guessing.

If your employer or organization absolutely will not allow any code or details shared, no matter how trivial, perhaps using “enterprise” products which come with support contracts, or hiring a private consulting firm is more appropriate than asking on a public forum.

After Your Question Is Answered

The best way to say “Thank You” is to acknowledge who had the right answer.  Many experts pour countless unpaid hours into helping strangers on forums, simply because it feels good to be helpful.  Being recognized as supplying the correct answer is a nice reward.

When your problem is solved, please consider Google and other search engines may send people with similar technical questions to your messages for years to come.  The very best thing you can do after a technical problem is resolved is a quick but clear message confirming the solution.

If you posted on multiple forums, please take a few minutes to follow up on every thread with a link to the message with best answer.

About This Article (and Me)

Over the last 6 years, I’ve written 18620 answers our forum here at PJRC, and many more on numerous other forums.  During this time, we’ve managed to grow a pretty good forum community and help many thousands of people with projects.

I watched and tried to learn which what has worked and how we might improve.  One thing that’s become very clear is so many people need a little guidance on how to best ask their questions.  It’s not the technical nature, but these subtle human factors that matter most.

My hope is to see all tech forums improve.  Please, share this article.

-Paul Stoffregen

 

USB Hub Bug Hunting & Lessons Learned

This tale begins with a customer reporting this cute little 2 port USB hub wasn’t working with Teensy 3.6.

In this article I’ll show how a protocol analyzer is used, how my instincts turned out to be very wrong, and along the way dive into arcane USB details you probably won’t see explained anywhere else.

First Instincts: Fail

This article is being written from the perspective of hindsight.  You can much of the process as it actually happened on the forum thread where this problem was reported.

Most of my first troubleshooting instincts revolved around these 2 questions:

1: This hub works on Linux, Mac & Windows.  What are they doing that Teensy isn’t?

2: All other hubs work on Teensy, so what’s different about this particular hub?

I believe most people would start with these 2 questions.  In the end, both turned out to involve very wrong assumptions that were ultimately a distraction from finding the real problem.

Protocol Analyzer: Fail (Software Crash)

Usually the first step in debugging USB problems is to hook up the protocol analyzer.  I have this one, the Beagle480 from Total Phase.

The way this works is pretty simple.  On the left side where I drew the 2 blue arrows, your USB communication passes through.  To whatever you’re monitoring, it just looks like a longer cable.  But it makes a copy of all communication happening in either direction and sends it to the port on the right (green arrow).

If you’re just making a USB device that plugs into your PC, usually you can run software to capture the data from your PC’s driver.  But when you’re making an embedded USB host like Teensy 3.6’s 2nd USB port, this is the only way to actually see what’s really happening.  The downside is the cost.  Total Phase sells this one for $1200.

Here’s the test setup, with the Beagle480 inserted between the Teensy 3.6 and 2-port hub.  Usually quite a clutter of cables is involved, but this little hub plugs right into the side of the Beagle480.  The hub has 1 keyboard connected.  A Mac laptop is watching the USB communication, and a Linux desktop is running the Arduino IDE to upload code to Teensy.

Despite the high price, it’s not made of magic.  Inside it has a fairly small buffer memory.  It has to send a copy everything on that right side port, which can be problematic during periods of sustained fast data flow.

This case turned out to be one of those problematic situations.  Unfortunately Total Phase’s software does not handle this situation well.  The Linux version simply crashes.

The Mac version is better.  It hangs with the infamous Mac beachball.  A Force quit is needed.

Fortunately having the software lock up means at least something can be seen on the screen, even if I could not scroll up to look at any other data.   Whatever was going wrong with this hub was causing a SETUP token to be repeatedly tried.  The timestamps show it’s happening about every 8 to 12 microseconds!

What’s Different About This Hub?

With the protocol analyzer not helping much, I decided to first focus on question #2.  All the other hubs work.  What’s different about this one?

In USBHost_t36.h on line 59 is an option to turn on lots of verbose debug printing.  But it doesn’t print much about the USB descriptors.  I decided this was a good time to fix that (rather than just plug into a PC and run a program to view), optimistically hoping the info would show some stark difference between this bad hub and all the good ones.  I spent a few hours adding this code to print descriptor info.

Here is the configuration descriptor for this 2 port hub.

Configuration Descriptor:
  09 02 29 00 01 01 00 E0 01 
    NumInterfaces = 1
    ConfigurationValue = 1
  09 04 00 00 01 09 00 01 00 
    Interface = 0
    Number of endpoints = 1
    Class/Subclass/Protocol = 9(Hub) / 0 / 1(Single-TT)
  07 05 81 03 01 00 0C 
    Endpoint = 1 IN
    Type = Interrupt
    Max Size = 1
    Polling Interval = 12
  09 04 00 01 01 09 00 02 00 
    Interface = 0
    Number of endpoints = 1
    Class/Subclass/Protocol = 9(Hub) / 0 / 2(Multi-TT)
  07 05 81 03 01 00 0C 
    Endpoint = 1 IN
    Type = Interrupt
    Max Size = 1
    Polling Interval = 12

Turns out this little 2 port hub is a Multi-TT type.  I’ll talk more about the transaction translators soon.  Most hubs have only a single TT, but 2 of my test hubs are Multi-TT.  Here’s one of their descriptors.

Configuration Descriptor:
  09 02 29 00 01 01 00 E0 32 
    NumInterfaces = 1
    ConfigurationValue = 1
  09 04 00 00 01 09 00 01 00 
    Interface = 0
    Number of endpoints = 1
    Class/Subclass/Protocol = 9(Hub) / 0 / 1(Single-TT)
  07 05 81 03 01 00 0C 
    Endpoint = 1 IN
    Type = Interrupt
    Max Size = 1
    Polling Interval = 12
  09 04 00 01 01 09 00 02 00 
    Interface = 0
    Number of endpoints = 1
 Class/Subclass/Protocol = 9(Hub) / 0 / 2(Multi-TT)
  07 05 81 03 01 00 0C 
    Endpoint = 1 IN
    Type = Interrupt
    Max Size = 1
    Polling Interval = 12

Turns out they’re an exact binary match, except for the last byte in the configuration descriptor.  I can’t emphasize enough how this process involves looking stuff up in the USB 2.0 PDF, in this case page 266.  This byte is just the hub telling the PC how much power is might draw.  The 2 port hub say 01, meaning only 2 mA which can’t be right.  But I know nothing in code on Teensy (which I wrote) ever look at this byte.  Maybe someday we’ll detect power issues, but for now it’s ignored.

Hubs also have a special descriptor for their capability.  It’s documented on page 417-418 of the USB 2.0 PDF.  Here the 3 different hubs did differ.

2 port (not working)
09 29 02 09 00 32 01 00 FF


4 port, ioGear, GUH274 (works)
09 29 04 89 00 32 64 00 FF


4 port, no-name blue color, UH-BSC4-US (works)
09 29 04 E0 00 32 64 00 FF

While I now know these differences are meaningless, and really only the 3rd byte (the number of ports) matters, at the moment I found these it seemed like this just had to explain the problem.  That 2 port hub is different.

Turns out the 4th byte describes features like which LEDs, over-current protection and other features the hub has.  The 7th byte says how long we’re supposed to wait between enabling power and accessing a port, but the bad 2-port hub says only 1 ms, much less than the other 2 working hubs.

What Would Linux Do?

With nothing apparently different between the hub, I returned to the idea that Teensy must be missing some initialization or other step that hasn’t mattered for all the other hubs, but does for this one.

It’s easy to look at what Linux does, since nothing goes wrong that crashes the Beagle480’s software.  Here’s most of the hub’s enumeration process on Linux:

I quickly noticed Linux is sending this Set Interface control transfer.  I did testing with all 7 of the hubs I have.  This doesn’t happen on the USB 1.1 hub or the 3 that are Single-TT, but Linux does do it for all the Multi-TT hubs.

Adding this to the USBHost_t36 library took time.  The hub driver was only looking at the first 16 bytes, to see if the interface and endpoint are compatible.  I had to rewrite the hub driver to look for all the possible interfaces.  Then when it was able to detect if 2 or more exist and parse which is the best to use, I added code to actually send that Set Interface command.

It didn’t make any difference.  I’d poured a few more hours into coding, which in the long term was probably worthwhile, but still no closer to solving the problem!

Working Around Data Capture Software Lockup

I seriously considered resigning this problem to my low-priority bug list, since all the other hubs work.  I considered restructuring the enumeration code to even more closely match Linux’s approach.  I looked over the descriptor data yet again, hoping to find the answer, but no.

Clearly I needed to get around the problem with Total Phase’s software locking up.  At first I just added a “while (1) ; // die here” right before calling new_Device() to start the enumeration process after the hub detects the keyboard.  But that didn’t reveal anything useful.

I needed to let the problem happen, but then shut off the USB port before the Beagle480’s buffer could overfill (or whatever other problem causes the Total Phase software to lock up).  Ultimately I did this:

    if (state == PORT_RECOVERY) {
        port_doing_reset = 0;
        print("PORT_RECOVERY, port=", port);
        println(", addr=", device->address);
static IntervalTimer stoptimer;
stoptimer.begin(panic, 1000);
        // begin enumeration process
        uint8_t speed = port_doing_reset_speed;
        devicelist[port-1] = new_Device(speed, device->address, port);

When I put in terrible debug code, I like to give it very odd indenting so I’ll see it when/if I later forget to remove it!  In this case, a hardware timer is started to generate an interrupt in 1 millisecond, which calls panic().

The panic() function looked like this:

void panic()
{
    GPIOE_PCOR = (1<<6); // turn off USB host power
    Serial.printf("Panic stop\n");
    while (1) ;
}

With this I was finally able to see the beginning of the problem.

Somehow the bad 2 port hub was indeed working, at least for the first control transfer which reads 8 bytes of the device descriptor.  But then everything goes wrong and nothing works.  The host controller keeps trying to resend the Set Address command… at least for 1 millisecond, until the panic() function abruptly shuts off the USB port’s power.

Realizing What’s Really Wrong

I wish I could tell you there’s a reliable process to go from actually being able to observe the problem to understanding it.  There isn’t!

In this case I spent several hours re-reading the USB 2.0 spec, staring at that screenful of info, and basically just guessing.  For a while I considered there might be some sort of electrical interference problem unique to this hub, somehow triggered by completing that first transfer.

It literally took hours to focus on this 1 byte which turned out to be the cause of so much trouble.

To understand this byte, you need to know about USB split transactions, which are probably among the most arcane of USB features.  This next part will be quite technical, so perhaps skip to the end if you’re not in the mood for intensely low-level USB protocol details.

Split transfers are used only between USB hosts and hubs.  If you’re making a USB device, you’ll never see this type of USB communication.  They’re used only when a host at 480 Mbit speed needs to communicate with devices at 12 or 1.5 Mbit.

When USB 3.x is used, all the 5+ Gbps communication happens on dedicated wires.  The cables & hubs have a redundant 480 Mbit path. When you use a 12 or 1.5 Mbit device on a USB 3.0 hub, that slow communicate is done using split transactions on the 480 Mbit line.  Slow speeds are never translated to the 5 Gbps lines.

USB has a lot of very specific terminology.  From smallest to largest, communication is in Tokens, then Transactions, then Transfers.  I’m not going into details here, other than it’s important to remember we’re dealing with 3 starts-with-T junks of data.  Tokens are the smallest.  Transactions are bigger, made up of Tokens.  Transfers are the largest, maybe of of Transactions & Tokens.

These SPLIT tokens communicate with a Transaction Translator (TT) inside the hub.  When the host wants to send a transaction at 12 or 1.5 Mbit, it sends a SPLIT-START token to the hub’s TT.  If the TT isn’t busy, it acknowledges and begins working on the slow communication.  That sequence of tokens is a transaction, also called split start (maybe confusing if you don’t know the context).  Later the host sends SPLIT-COMPLETE.  If the TT has finished, it replies with ACK, and if it’s still busy it replies with NYET.

Here’s the diagram from the USB 2.0 spec on page 202 showing this for OUT transactions.  Sadly there isn’t a diagram specifically for SETUP transactions, but my guess is this should be pretty close.  Did I mention this sort of work is all about guesswork and re-reading technical specs over and over?

Total Phase’s software is very nice.  It normally shows you one Transfer per line.  If you want to see the Transactions that made up that transfer, you click the little triangle to expand.  Likewise if you want to drill down to see the actual tokens, you can.  Here’s the first successful transfer.  Hopefully you can see this more-or-less corresponds to Figure 8-9, except SETUP instead of OUT.  Of course, we only see the stuff from circles #1 & #3 since we’re connected between Teensy and the hub.

In this case, we can see Teensy sent 58 SPLIT-COMPLETE tokens while the hub slowly send that SETUP token at 1.5 Mbit to the keyboard, where the hub replied with NYET to the first 57, then finally an ACK.

This is important, because the process of figuring out why something isn’t working almost always involves first looking at the deeper details of what it’s supposed to do when it does work.

With that in mind, here’s an expanded view of the first Set Address transfer that fails.

Remember the software is showing us nesting of Tokens inside Transactions inside Transfers.  So the 3rd line is actually the first real data.  This badness begins with the bytes 78 01 82 B4.

Earlier 78 01 82 10 was considered ok.  But 78 01 82 B4.  So what does that last byte mean?  Again the answer is digging into the nitty gritty details in the USB 2.0 document.  It happens to be on the same page as that diagram above.

Even though going from 10 to B4 looks like a big change, in fact the top 5 bits are a CRC check which is expected to be totally different if any of the checked bits changed.  So really this means just one of the “ET” bits flipped from a 0 to 1.

The meaning of those ET bits is documented a few pages later.

These bits were 00 for Control, but somehow became 10 for Bulk.  That’s definitely not right!  We’re sending a SETUP token, and that token has the endpoint number encoded within (also documented elsewhere in chapter 8….)

Where This 1 Bit Went So Wrong

The good news is this last part, going from an understanding of what’s wrong to finding the mistake causing it, is relatively easy.  It took only about an hour.  While I didn’t know exactly what controlled those ET bits, I was pretty sure it was something in the EHCI controller’s QH or qTD data structures.

USB controllers implementing EHCI are very unlike traditional embedded peripherals, where a handful of fixed register addresses control everything.  EHCI gets almost everything from main system RAM, where it’s regularly using DMA to read linked lists and binary tree data structures to find out what work you want it to perform.

For each pipe/endpoint each USB device, you create a QH structure (except complexities for isochronous… but trying to keep this simple).  Here’s the EHCI diagram for the QH structure.

The first field is used for the linked lists or inverted tree pointers.  The next two are where you actually configure how the hardware will communicate.  The big yellow area is written by the hardware, coping other info there temporarily as it works on the Transfers you want.  (Yes, the hardware deals in Transfers and automatically does the Transactions & Tokens)

While there’s a lot of small fields in those 64 bits, the “C” bit turned out to be the key.  The EHCI spec documents it as:

Control Endpoint Flag (C). If the QH.EPS field indicates the endpoint is not a high-speed device, and the endpoint is an control endpoint, then software must set this bit to a one. Otherwise it should always set this bit to a zero.

Fortunately the code uses C structs to with distinctive names to represent the QH and other EHCI data structures.  A quick grep of all the code turned up everything that accesses this field.  Only a few of them actually write to it.

Ultimately the bug turned out to be just this 1 line function called pipe_set_maxlen()

pipe->qh.capabilities[0] = (pipe->qh.capabilities[0] & 0x8000FFFF) | (maxlen << 16);

Of course if you look at the 2nd line in the QH documentation, this is incorrectly zeroing too many bits, destroying the C bit and part of the RL field.  It should be:

pipe->qh.capabilities[0] = (pipe->qh.capabilities[0] & 0xF800FFFF) | (maxlen << 16);

Indeed applying this fix made the “bad” 2-port hub work quite nicely.

Headslap Moment

After I found the fix, I looked one last time at the question “All other hubs work on Teensy, so what’s different about this particular hub?”

Here’s the communication with that keyboard through one of the “good” hubs!

Turns out the code has been sending improper SPLIT-START tokens all along, but every hub I own and the numerous hubs other people have used all were able to work anyway!  The different was this particular hub was actually rejecting the bad tokens.

In all this time, developing this library earlier this year and all this recent work to track down this bug, apparently I never actually connected the protocol analyzer and watched the “good” hubs really communicating with low speed devices!  Could have saved a *lot* of work.  Now I know, in hindsight.

Why APA102 LEDs Have Trouble At 24 MHz

It’s well known long APA102 LEDs strips have have trouble at 24 MHz, usually after 150 to 250 LEDs.  But why?  Here’s my attempt to investigate.

One tempting explanation is signal quality problems due to horribly messy unterminated wiring carrying high speed signals, as in this photo!  But it turned out to be a more fundamental timing problem.

In this test, a Teensy 3.2 runs the FastLED “Cylon” example with this line:

 LEDS.addLeds<APA102,11,13,RGB,DATA_RATE_MHZ(24)>(leds,NUM_LEDS);

NUM_LEDs was set to 160, and I connected a strip of 144.  The oscilloscope traces are the signals which arrive at the end of the 144 LED strip.

First, I set the clock speed to only 2 MHz to see the “normal” waveforms output by the last APA102 LED.

The main thing to observe here is the APA102 output changes its data line (blue) at the falling edge of the clock (red).  You might notice a slight delay from the falling edge of the clock to the change in data, but it’s tiny relative to the slow 2 MHz clock cycle.

At 24 MHz, the delay is much more significant.  In this case I measured approximately 15 ns delay from clock to the data changing.

You might also notice the red trace doesn’t look like the 50% duty cycle SPI clock signal.  I believe this, together with the data delay, is the main cause of APA102 issues on long LED strips.

Here’s another measurement of the clock.

Each APA102 LED is supposed to regenerate the clock signal.  Ideally this is supposed to allow a very long LED strip.  But it appears each APA102 lengthens the clock high time and shortens the clock low time slightly.  This might be internal to the APA102 control chip, or it could be simply due to the clock output driver having a faster fall time than rise time, causing the following APA102 to receive a slightly different clock high time.  Perhaps the APA102 controller chip has a better N-channel transistor for pulling the clock output low than the P-channel transistor for driving it high?

After 144 LEDs, the clock low time on this strip has shrunk from 20.83 ns to approximately 18 ns.

With the data output delayed 15 ns after the falling edge, this leaves only 3 ns before the next APA102 LED captures the data on the rising edge of the clock.  As the strip gets longer, each APA102 reduces the clock low time, until it’s shorter than the clock to data delay.

FastLED defaults to 12 MHz SPI clock for APA102 LEDs on Teensy 3.x, which should allow for several hundred LEDs before this clock duty cycle change becomes a problem.

This test was just one APA102 strip I purchased about a year ago.  The Chinese semiconductor manufacturers making these LEDs have a history of changing the silicon without any notice.  I also only tested at room temperature, using only 1 example program which doesn’t drive the LEDs anywhere near 100% duty cycle (more heating).  I powered the 144 LEDs with 5V from both ends, but didn’t make any measurements of the voltage near the middle of the strip.  Power supply voltage might matter.  In other words, your mileage may vary.

But hopefully this helps with understanding what’s really going on, why short APA102 LED strips work so well with the fast clock speeds, but fail when using very long strips, even though the LEDs are supposed to regenerate the clock and data as they pass it down the strip.

Fast pulse counting with interrupts and why nested priority really helps

A question was recently asked on the forum, how fast can attachInterrupt count pulses.  I did some testing to find out, and made this video.

Turns out Teensy 3.2 can run about 1.25 million interrupts per second.  Teensy 3.6 can do about 2.55 million per second.  But these depend on assigning a top priority to the interrupt, as explained in the video.

Of course, both boards can use a timer to count pulses at very high rates, at least 30 MHz.