Recently I’ve been working to improve the Arduino Serial Monitor.  Here it is running with Teensyduino 1.48-beta1.

Previously if a board sent data this fast (as Teensy 4.0 can), Java would run out of memory and the Arduino IDE crashes.

Teensy 4.0’s USB code is not yet fully optimized, so we can expect even greater speeds later this year.  The Arduino Serial Monitor needs improvement to handle these faster data rates!

Deja Vu From 2014

This isn’t the first time Teensy has crashed Arduino by sending too rapidly to the Serial Monitor.  Back in 2014, this same problem existed with Teensy 3.1.  Serial.print() without delay on Teensy 3.1 would cause Java to run out of memory and crash the Arduino IDE.

Arduino Due was also capable of crashing the Arduino IDE this way.  The Arduino developers had tried in October 2014 to solve it by limiting the buffered data size, which helped, but still Java would eventually run out of memory and lock up.

On December 6, 2014, I finally managed to work around the problem well enough for Arduino to handle sustained USB full speed (12 Mbit/sec) incoming data.  My solution worked around the terrible slowness of adding and removing data from the JTextArea component by collecting incoming data to a buffer and using a timer to add data at only 30 Hz rate.  It also limited the rate of removal to only once every 150 adds, and removed by the number of characters rather than the number of lines.  4 days later, the Arduino developers adapted my solution and merged it into Arduino.  This code as been in every version of Arduino since 1.6.0.

At the time, I wrote this explanation of the details and rant about Java performance.  Back then I wrote “Java is pretty horrible”.  Now with the benefit of hindsight, I realize I was equating Swing’s JTextArea and JTextComponent classes (and the complicated data storage infrastructure lurking behind them) with Java in general.  I also wrote “if dramatically faster hardware is made … in the future, this buffer might need to grow”.

Now with Teensy 4.0 bringing that dramatically faster hardware, and some hindsight, I’ve learned how so much more than merely increasing the size of an intermediate buffer is needed to support sustained data transfer at such speed.

What’s Really Using So Much Memory?

I quickly discovered the terrible slowness inside JTextArea & JTextComponent scaled up (or “down”) rapidly with data size.  Keeping the same 30 updates per second but with larger data would not work.  It failed spectacularly.  Under the load of Teensy 4.0 printing without delay, the Arduino IDE would run slowly for a matter of seconds, then on Windows and Mac start throwing OutOfMemoryError exceptions and ultimately lock up.  On Linux, it would keep running, but unusably slow and consume many gigabytes of memory.  Not good.

To start digging into the problem, I ran VisualVM, which is a Java profiler.  It’s one of the programs bundled with the Java SDK.  If you have the SDK and a JAVA_HOME environment variable (the usual setup for compiling Arduino from source), it can be run from the command line with “$JAVA_HOME/bin/jvisualvm”.

VisualVM is very easy to use.  Every Java-based program running on your machine shows up in the “Local” group.  Arduino appears as “processing.app.Base (pid [number])”.  Clicking it connects the profiler to the running Arduino IDE.  Then clicking the “Profiler” tab lets you see which Java classes are using so much memory.

This screenshot shows the memory use after only several seconds of Teensy 4.0 printing rapidly to the Serial Monitor.  While 100 megabytes is used by raw character data, the really startling result is nearly 2 million live instances of GapContent$MarkData, GapContent$UnfoPosRef, AbstractDocument$LeafElement, and GapContent$StickyPosition.  Yikes!

Even on Linux, with the extra burden of the VisualVM profiler, Java quickly crashes under the strain of Teensy 4.0 printing without delays.  But the profiler served its purpose, so shine light on what’s consuming such an insane amount of memory.  GapContent appeared to the culprit.

Flexible or Bloated, A Matter of Perspective?

Java has a pretty amazing amount of good documentation.  Google searches always turn up Oracle’s reference material.  Usually searches turn up many nice Java tutorials and well answered questions on sites like Stack Overflow.  But from the lack of non-reference material, not even unanswered questions (other than people trying to use JTextArea or JTextComponent as a terminal or live log file display and hitting these same memory use problems), it seems this part of Java is a seldom traveled path.  That’s much of the reason I’m taking some time to write this lengthy blog article, to share with you what I’ve learned on this optimization journey.

I spent a lot of time reading the Java reference pages.  A lot of time…

Internally a number of Java classes are used in a rather modular way.  This modularity could be been seen a highly flexibly or highly bloated system, either a blessing or a curse, depending on your perspective.  In the end, the modular nature turned out to be quite useful.  But first, let’s look at how it’s structured.

This diagram from the JTextComponent reference best sums up the way things really work under the hood.

If you compare with Oracle’s page, you’ll see I’ve add the GapContent part to this diagram.  It turns out the Document class actually outsources all the data storage to GapContent.  At one point I had imagined just replacing GapContext with something more efficient.  But sadly, the GapContent API is designed around the assumption that the data size grows to any arbitrary size.  I wanted to replace all the storage with a fixed-size FIFO circular buffer and avoid *any* use dynamically allocating classes or large data on the heap during the sustained processing of data.

FifoDocument Class

GapContent had to go, so I started work on a new FifoDocument class which would hold all the Serial Monitor text in a fixed size array.

The idea was simple.  Since we only new add lines at the end, and delete the oldest lines from the beginning, this ought to be simple, right?

Elements & Events

At first I did not understand the purpose of the Element class.  I imagined just sending a DocumentEvent output with a single Element representing all the text.  If you read only that Element reference, perhaps you can see how it seems to imply that might work?  At least that’s what I incorrectly assumed.

The Document reference is the only other page (which I found) describing Elements.  But it only describes how they might be used in a generic way.  This Element structure image is completely wrong for the use case of JTextComponent.

It turns out JTextComponent expects a single top-level Element as a container for the entire document, which has child Elements representing each line.  Near the end of the explanation on that Document reference page is a dead link to “see The Swing Connection and most particularly the article, The Element Interface“.  Every indication is Oracle deleted “The Swing Connection” blog many years ago, and dead links automatically redirect to the generic Java page.

Fortunately I did find a copy of The Element Interface article archived at an academic site.  This article is essential to understand what Element structure the various Java Document classes actually use, if you want to craft your own custom Document class to replace on of them.  For the Arduino Serial Monitor case, it’s the PlainDocument structure.

My initial hope to use a single Element was replaced by adding a large, fixed size array of FifoElementLine instances which keep track of where the individual lines of data are located within the big FIFO circular buffer.

With this addition and many trial-and-error tests to figure out which functions actually get called, *finally* the Serial Monitor window started responding to the DocumentEvent notifications and displayed the text.

An early experiment also showed that FifoDocument could delete data without receiving any input from JTextArea.  By simply sending an event to notify JTextArea that data has been deleted, indeed the Serial Monitor would update properly.  This highly flexible event-based design that could be seen as bloat turned out to be very useful for implementing an efficient FIFO that automatically discards old data.

Later this ability for the Document to notify the GUI of changes (which it didn’t initiate or participate in any way) turned out to be even more useful for directly adding data into the FIFO.

Bugs & Thread Safety

Without the help of several great people on the PJRC Forum, especially Tim (Defragster), FifoDocment probably never would have reached a usable state.  The DocumentEvent interface involves many complex requirements which are only scantly documented.  Many tries were needed to get everything right.  Defragster found pretty much every bug very quickly.

Even after the “easy” bugs where fixed, thorny problems with threads remained.  I ended up making almost all the public methods of FifoDocument synchronized.  The thread which adds new data into FifoDocument is also called using SwingUtilities.invokeAndWait().  These are less than optimal.  Perhaps later even better performance could be possible?

Direct Write Into FIFO Memory

Even with all these optimizations, Java would still run out of memory on some Macintosh machines.  Arduino’s traditional implementation of the serial monitor makes multiple copies of incoming data before it finally ends up stored in FifoDocument (or GapContent).

First, the Serial class has an event handler which receives incoming characters into a buffer, which is allocates on the heap.  Then that buffer is converted to a String and passed to a message() function, which is the abstraction allowing different classes to receive data.  The contents of that String are then copied into a StringBuffer instance, which is the improvement I contributed 5 years ago.  At a rate of 30 Hz, then StringBuffer is then copied another String instances, which is passed to JTextArea, which then passes it to the Document storing the data.

I replaced all this copying of data with path directly from arriving characters into the FifoDocument character buffer.

Unfortunately this means overriding the usual path data takes between the abstract serial monitor classes.  Instead, a single loop waits for data to arrive.  When data is ready to read, it requests the maximum number of bytes FifoDocument can accept, and the offset where that data goes inside FifoDocument’s fixed size buffer.  It then reads incoming characters directly from the input stream to FifoDocument’s buffer.  No extra copies are made in other buffers, or String instances to pass the data between abstraction layers.

With this final optimization, even older Macs could continuously receive from Teensy 4.0 without running out of memory or locking up.

However, Java’s InputStreamReader class is still used to convert the raw bytes from UTF8 format to Java’s internal handling of all characters, and the Document event API still uses heap-based temporary allocations for some features.  These do still cause Java’s memory usage to grow gradually, then suddenly shrink what Java’s garbage collection runs.  But at least initial testing appears as if this overhead is acceptable.

Auto Scroll Behavior

While developing the FifoDocument with a truly fixed size buffer, I was forced to make some hard decisions about how to handle the Serial Monitor’s auto-scroll checkbox.

Since Arduino 1.6.0, the Serial Monitor has used a target size of 4,000,000 characters for its buffer (half of the maxChars size in the TextAreaFIFO class).  But this only checked every 150 updates, which can happen no faster than 30 Hz.  Regardless of the auto-scroll checkbox, if the stored data has grown longer than 4,000,000 characters, the oldest data is deleted so only 4,000,000 characters remain.

FifoDocument has a fixed size buffer, rather than the flexible storage of GapContent, so I was not able to preserve this functionality.  I’m also not convinced this existing approach is really correct, since rapidly arriving data can cause whatever the user is trying to read to be deleted.  The TextAreaFIFO class as a flag to tell whether to trim the oldest data, but in all modern versions of Arduino it is never used or changed.

For FifoDocument, I implemented a buffer management policy where the buffer is allowed to fill to 60% capacity while scrolling.  During sustained scrolling, 40% of the buffer remains free.

When auto-scroll is disabled, FifoDocument allows new data to completely fill the buffer.  Once the buffer is 100% full, FifoDocument discards new data.  This may be a controversial decision.  The idea is to allow the user to read anything already within the buffer, and everything new which arrives until the buffer becomes 100% full.  The Serial Monitor window can’t jump to go blank if too much new data pours in while the user is reading.  But once the fixed size buffer is full, nothing more can be captured until auto-scroll is turned back on.

FifoDocument currently implements a 10,000,000 character buffer, and a maximum of 1,000,000 lines.  This allows more data to be captured, but the behavior is not exactly the same as current versions of Arduino when auto-scroll is turned off.

Performance

Despite my frustrated words 5 years ago, indeed Java is capable of implementing the serial monitor at these higher speeds.  But the pervasive approach of most Java code, allocating new objects while passing data between many abstraction layers, puts far too much burden on Java’s memory management and garbage collection.  Using a fixed size buffer, with incoming data stored directly to the buffer without allocating copies on Java’s heap is indeed fast enough.

In a somewhat surprising result, the most efficient system for this use is Microsoft Windows 10.

The Java virtual machine appears to run Java code more efficiently on Windows than it does on Macintosh and Linux.  Perhaps Oracle (or Sun) invested more work to optimize the JRE for Windows?

In this screenshot, you can see the “teensy_serialmon” program running, also with low CPU usage.  This helper program talks to Teensy using Windows native WIN32 functions and then passes the data to Java using stdin & stdout streams.  On Windows 10, both the built in serial driver and the lightweight anonymous pipes used for stdin/stdout appear to be very efficient when accessed with native WIN32 functions.

While Linux is not far behind Windows, unfortunately Macintosh appears to have considerable CPU overhead for accessing serial devices.

This screenshot was taken on the same Macbook Air as the Windows 10 test above (running natively, via Bootcamp dual boot).

However, the Macintosh USB drivers allow Teensy 4.0 to transmit about 60% faster than the drivers on Windows and Linux.  When running the USB serial print speed benchmark, Linux and Windows usually sustain about 200,000 lines/second.  Macintosh usually runs just over 300,0000 lines/second.

These differences are very likely an artifact caused by less-than-optimal USB driver code on Teensy 4.0, interacting with subtle timing differences in the USB host drivers on each system.  I have been delaying work for USB optimizations on Teensy 4.0 until the Serial Monitor is capable of handling the incoming speed without crashing the Arduino IDE.  Now, with these improvements, I can start to focus my effort on optimizing the Teensy side!

Arduino Contribution

As always, my intention is to contribute Teensy-inspired improvements to the Arduino IDE back to the Arduino project.  Several weeks ago I exchanged a few emails with the Arduino developers about this Serial Monitor optimization work, so they are aware of my effort.  Part of the reason to write this lengthy article is to document this work from a “high level” perspective which isn’t really possible by the comments in the code which give more detail.

All of this source code is published on Github.  These are the files:

• FifoDocument.java – All the FIFO and Document code
• TeensyPipeMonitor.java – Teensy’s “Pluggable” Serial Monitor using FifoDocument.  This file creates the listener threads which receive stdin data directly into FifoDocument’s buffer, and parse stderr for status updates.
• FifoEvent.java – The event info FifoDocument sends to JTextComponent.  Methods just call the actual implementation in FifoDocument.
• FifoElementLine.java – The Element instances representing individual lines of text.  Methods just call the actual implementation in FifoDocument.
• FifoElementRoot.java – The top-level Element required by JTextComponent.
• FifoPosition.java – The Position interface which JTextComponent requires for users to select text and copy to clipboard.  Positions are actually implemented with a 64 bit total-historic offset, as explained in this comment.

This work could be considered 2 separate contributions, even though their code is now pretty tightly integrated.  The other work could be called “Pluggable Serial Monitor”, similar in concept to Pluggable Discovery.  The concept, like with discovering ports, is a board can provide a program like “teensy_serialmon” which does the actual communication and makes it available to the Arduino IDE using stdin & stdout streams.  Teensy has used this approach in beta testing since early 2017, and released in Teensyduino 1.42  June 2018).  The port discovery portion was accepted by Arduino around that time.  The serial monitor part will hopefully become an official Arduino feature at some point in the future.

Whether this serial monitor performance improvement may ever accepted by Arduino is unclear.  Arduino has announced at Maker Faire and Arduino Day (March 2018) a next-gen Arduino IDE which will no longer use Java.  How they feel about merging such a large change to the existing Java code is unclear, especially when Teensy 4.0 is (probably) the only board available today which can transmit at these speeds which crash the existing IDE.

Still, my hope is this code may eventually find its way into an official Arduino release.  Eventually more microcontrollers will appear on the market with 480 Mbit or faster USB, and fast enough processors and USB code capable of sustained printing at these speeds and perhaps much faster.

When anyone is later interested in this Serial Monitor optimization, hopefully this lengthy article and the code on github will help.

Normally sine waves are generated on microcontrollers using a table lookup.

Lookup tables are perfect when wavelength happens to be an exact multiple of the sample rate, because you never actually need to know the values in between the table’s stored points.

But if you want to generate waveforms at any frequency without changing your sample rate, you end up needing points on the waveform that are between two entries in the table.  Four approaches are possible.

1. Use the prior point, even if the next point would have been better
2. Use the prior or next point, whichever is closer
3. Use both nearest points with linear interpolation
4. Use 3 or more nearest points, with spline or other non-linear interpolation

With any of these, larger lookup tables give better accuracy.  Since sine waves have symmetry, some programmers choose to store only 1/4 of the waveform and add slight overhead to map the other 3 quadrants onto the smaller table.

The Teensy Audio Library uses approach #3 for normal sine wave synthesis.  The vast majority of sine wave examples in the Arduino ecosystem use approach #1.

One nice advantage of cutting off the Taylor series with on of the subtracted powers (3, 7, 11, etc) is the tiny remaining error will always be slightly less than the true ideal sine value.  This means the final result does not need be checked for greater than 1.00000 and rounded down to fit into the maximum value of an integer.

// High accuracy 11th order Taylor Series Approximation
// input is 0 to 0xFFFFFFFF, representing 0 to 360 degree phase
// output is 32 bit signed integer, top 25 bits should be very good
static int32_t taylor(uint32_t ph)
{
        int32_t angle, sum, p1, p2, p3, p5, p7, p9, p11;

        if (ph >= 0xC0000000 || ph < 0x40000000) {
                angle = (int32_t)ph; // valid from -90 to +90 degrees
        } else {
                angle = (int32_t)(0x80000000u - ph);
        }
        p1 =  multiply_32x32_rshift32_rounded(angle << 1, 1686629713);
        p2 =  multiply_32x32_rshift32_rounded(p1, p1) << 3;
        p3 =  multiply_32x32_rshift32_rounded(p2, p1) << 3;
        sum = multiply_subtract_32x32_rshift32_rounded(p1 << 1, p3, 1431655765);
        p5 =  multiply_32x32_rshift32_rounded(p3, p2) << 1;
        sum = multiply_accumulate_32x32_rshift32_rounded(sum, p5, 286331153);
        p7 =  multiply_32x32_rshift32_rounded(p5, p2);
        sum = multiply_subtract_32x32_rshift32_rounded(sum, p7, 54539267);
        p9 =  multiply_32x32_rshift32_rounded(p7, p2);
        sum = multiply_accumulate_32x32_rshift32_rounded(sum, p9, 6059919);
        p11 = multiply_32x32_rshift32_rounded(p9, p2);
        sum = multiply_subtract_32x32_rshift32_rounded(sum, p11, 440721);
        return sum <<= 1;
}

UPDATE: Josy Boelen mentioned alternate forms for Taylor series approximation which require fewer multiplies.  Whether these could also be optimized with the M4 DSP extension instructions (not keeping full 64 bit resolution at every step) could be a really interesting future project…

This article was originally published in January 2016 (archive.org link) on the DorkbotPDX site.  Since then, the DorkbotPDX blog section has vanished.  I’m reposting it here with slight edits and a couple waveform plots, to preserve the info, and also because Michael Field recently asked for an article about these sorts of numerical approximations (which are rarely given as highly optimized fixed-point source code).

Arduino 1.8.9 has a new Pluggable Discovery feature, which began as a contribution from Teensy.  Ports can now be customized to support any protocol.

Already Omlzo is using Pluggable Discovery with CAN bus.

In this article, I’ll dive into the details needed to use Pluggable Discovery.  But first…

A Brief History

Since the first Arduino software in 2005, the main way to upload code to boards has been by serial ports.  In 2008, support for in-circuit programmer hardware was added, where you select a type of programmer  rather than a port.  In 2013, port detection and selection in the Ports menu was expanded to include network ports for Arduino Yun.  Around that time, Arduino also became highly configurable by recipes in platform.txt, based on a major contribution from Mark Sproul & Rick Anderson.   But until now, Arduino has supported detecting only 2 types of ports, either serial or network (using Yun’s specific protocol), with the only way to customize being dedicated programmer hardware.  Protocols other than serial & Yun’s network just weren’t supported for the Ports menu.

Teensy has used HID protocol, not Serial, for code uploading since its launch in 2008.  Using Teensy with Arduino was somewhat awkward.  When running a program implementing USB Serial, Teensy would appear in Arduino’s list of serial ports.  But with HID or other non-serial protocols, no port would be shown.  Teensyduino’s upload process could automatically detect Teensy in those modes, so uploading would “just work” even though the ports menu indicated no ports available.  When things went wrong, like this infamous Windows 7 driver bug, lack of reliable ports display made confusing situations hard to understand.

In early 2017, I started work to improve how Teensyduino integrates with Arduino.  A dedicated “teensy_ports” utility watches for Teensy’s USB devices to be added & removed.  Inside the Arduino IDE, I added a 3rd type of port to the DiscoveryManager class, using this utility’s info.  Finally, Teensy could be seen in the Ports menu even with running in a non-Serial mode.  You could choose exactly which board to use for upload, rather than hoping the automatic detection would find the right one.

In a June 2018 conversation with Massimo Banzi, Cristian Maglie and other Arduino developers, it was decided that I would contribute this code to Arduino.  A flexible JSON format would replace the original single-line text, so anyone could use it, and Arduino can adapt the format in the future.  After a lengthy process, where Cristian rewrote and greatly improved the JSON parsing, in March 2019 Arduino 1.8.9 was finally released with this new “Pluggable Discovery” feature.

JSON Data Format

The first step to using Pluggable Discovery is to write a “discoverer” utility which sends information to the Arduino IDE as your board is connected and disconnected.  Data is sent to Arduino on stdout, and Arduino may send you commands on stdin.

Here is a sample of the JSON data “teensy_ports” sends to Arduino 1.8.9 when running on Linux.

{
  "eventType": "add",
  "port": {
    "address": "/sys/devices/pci0000:00/0000:00:14.0/usb4/4-14/4-14.1/4-14.1.4",
    "protocol": "Teensy",
    "protocolLabel": "Teensy Ports",
    "boardName": "Teensy 3.6",
    "label": "/dev/hidraw5 Audio",
    "identificationPrefs": {
    },
    "prefs": {
      "vendorId": "16C0",
      "productId": "04D2",
      "serialNumber": "0"
    },
    "online": "true"
  }
}

Here is a quick attempt to document these fields.  Please understand these fields may change meaning, or others may be added by future Arduino releases.  Hopefully in the future Arduino will add official documentation for this JSON format.

• eventType: Either “add” when a device is detected and Arduino should begin showing it in the Ports menu, or “remove” when disconnected and the Ports menu should no longer show it.  If information changes, or more information becomes available, multiple “add” messages may be sent to cause Arduino to update the Ports menu.  The eventType “error” is also supported.  See Discoverer Commands below for details.
• port: A container for all of the information about the port.
• address: A unique identifier for this port.  The address is not shows in the Ports menu, but does appear in the status bar on the bottom of the Arduino IDE windows when the port has been selected.  The address is the primary info you would give to your uploader utility.  See “Integrating With Your Uploader” below for details.
• protocol: A unique (or at least distinctive) name for the communication protocol the port uses.  Currently Arduino 1.8.9 ignores this field, unless it is “serial” or “network”.  This name can be passed to your uploader utility.
• protocolLabel: The name of your type of port, as shown in the title of a sub-section of the Ports menu.  Currently there is no way to adapt this field to different languages.  This info can also be passed to your uploader utility, though typically the “protocol” field would be used rather than this name meant for display to the user.
• boardName: The name of the board.  This should probably match one of the names you define in boards.txt, though Arduino 1.8.9 does not currently enforce any requirements.
• label: The human readable name for the port, to be shown as a selectable item in the Ports menu.
• identificationPrefs: Arduino 1.8.9 does not make any use of this field.
• prefs: This is meant to collect info about the port, such as the info shown by “Get Board Info”.  However, Arduino 1.8.9 does not make use of this field for non-serial ports.  Hopefully future versions will fix this.
• online: While not used by Arduino 1.8.9 for non-serial ports, this should normally be “true” when eventType is “add” and “false” when eventType is “remove”.  With serial ports, the port’s online status can be momentarily false when Arduino is attempting to send a command to the board to automatically reboot into its uploading or bootloader mode.

When a port should be deleted, only the eventType and address fields are required.  Here is a sample of the JSON remove message teensy_ports transmits.

{
  "eventType": "remove",
  "port": {
    "address": "/sys/devices/pci0000:00/0000:00:14.0/usb4/4-14/4-14.1/4-14.1.4",
    "online": "false"
  }
}

Like all programs which print to stdout, buffering may exist between your output and Arduino’s reception.  Composing the entire JSON message in a buffer, then using a single write and flush to stdout is the most efficient way.  Arduino is using the Jackson library to parse your JSON messages, so sending piecemeal, with or without newlines or other optional white space should also work.

Discoverer Commands

The Arduino IDE will also send your discoverer commands on its stdin stream.  These are not JSON.  They are plain text commands, terminated by a single newline character.

• START_SYNC – Begin running in sync mode, where your program transmits JSON messages automatically as the device change events occur.
• START – Begin running in polled mode, where Arduino will send a “LIST” command at regular intervals.
• LIST – In polled mode, this command is meant to prompt your program to scan for changes.
• STOP – Stop running.

Future versions of Arduino may behave differently, so please understand the following behavior is specific to version 1.8.9.

When Arduino runs your discoverer, it will send a single START_SYNC command.  If you do not need polled mode, no response is needed or expected.  Your program simply transmits JSON messages as it detects ports, starting with “add” eventType messages all the ports currently available, and then more “add” or “remove” messages as available ports change.

If you require polling mode, you would transmit this JSON message.

{
  "eventType": "error",
  "message": "START_SYNC not supported"
}

Upon receiving this message, Arduino should transmit “START” and then send “LIST” at regular intervals.  If your discoverer is unable to detect hardware changes as they occur, you can design your code to wait for “LIST” and then check whether ports have changed.

With polling, typically you would need to design your discoverer to keep in memory a list of the ports it has previously transmitted to Arduino.  When you poll which ports are actually available, your discoverer is responsible for detecting when any port you have previously transmitted an “add” eventType is no longer present.  You must transmit the “remove” eventType, to cause the Arduino IDE to no longer offer that port to users in the Ports menu.

A list of previously detected ports can also help you avoid repeatedly sending the same info as your discoverer checks which ports currently exist.  But this is not required.  As long as the “address” field remains the same, Arduino Pluggable Discovery treats a redundant “add” eventType as a request to update the port’s info.  Sending the same info repeatedly is inefficient, but harmless.

Polling mode may not be working properly in Arduino 1.8.9.  Most testing was done with sync mode.

Arduino 1.8.9 transmits STOP only when the user closes the entire IDE, just before your program is terminated.  There is no guarantee your discoverer will continue running long enough to actually receive the STOP message.

Future Arduino software may send START_SYNC, START and STOP at different times, but with 1.8.9 the start messages are used only as the Arduino IDE starts up, and STOP is sent only when the user quits the software.

Running Your Discoverer

Like everything else customizable in modern Arduino, you create a recipe in your platform.txt file which tells Arduino to run you discoverer.  Here is a recipe for “teensy_ports”.

## Teensy Ports Discovery (Arduino 1.8.9)
discovery.teensy.pattern="{runtime.ide.path}/hardware/tools/teensy_ports" -J2

This recipe assumes the teensy_ports utility is installed in the hardware folder, inside the Arduino software.  Because the recipe starts with “discovery”, Arduino 1.8.9 detects it is a command for a pluggable discovery program.  It will be run as Arduino starts up.  Use of {runtime.ide.path} only works if your discover is copied to a locating inside Arduino’s folder.

The “-J2” is parameter given to “teensy_ports”, the same as if you had typed “./teensy_ports -J2” on the command line.  While unlikely to be needed, you can build almost any command line your discoverer might need.  In this case, the JSON format changed during development, so “-J2” tells teensy_ports with JSON dialect to output.

If your board is installed by Arduino’s Boards Manager, your discover is meant to be defined and installed as a tool and referenced with {runtime.tools.TOOLNAME.path}.  See this comment from Cristian for more detail.

Integrating With Your Uploader

Now that your discover can find your board’s custom port, and the user can select it in the Ports menu, you need a way to actually pass the detected info to your board’s uploader program.

Fortunately this part is easy.  The macro {serial.port} in your uploader recipe in platform.txt will be replaced by whatever text your discoverer provided in the JSON “address” field.

tools.yourtoolname.upload.pattern="{runtime.hardware.path}/upload_program_name" "{build.path}/{build.project_name}" "{serial.port}"

Yes, it’s still called “serial.port” in these substitution patterns, even though we can now (theoretically) extend the Arduino IDE any sort of protocol.

While your uploader probably only needs the unique address field and the pathname to find the code Arduino compiled, you can also pass some of the other JSON fields to your uploader.  Here is the recipe Teensy is currently using.

tools.teensyloader.upload.pattern="{cmd.path}/teensy_post_compile" "-file={build.project_name}" "-path={build.path}" "-tools={cmd.path}" "-board={build.board}" -reboot "-port={serial.port}" "-portlabel={serial.port.label}" "-portprotocol={serial.port.protocol}"

Hopefully Arduino will eventually document the mapping from JSON fields to recipe substitution names.  These were found by reading the Java source code and some experimentation.

Troubleshooting

What could possibly go wrong?!

Arduino 1.8.9 has a fair amount of debugging code built in to help you see how Arduino is running your discoverer, and what it’s receiving.  To enable the debug output, add this line to your preferences.txt file.

discovery.debug=true

If you’re not sure where your preferences.txt file is located, click File > Preferences and look at the info in the button part of the Preferences dialog.

Minor Issues In Arduino 1.8.9

Polling mode may not work.

The substitution macro {runtime.hardware.path} very often used in platform.txt recipes is not yet supported.  Teensyduino adds a patch to Arduino 1.8.9 to support this in discovery recipes.  If you prefer {runtime.hardware.path}, installing Teensyduino may be a short-term solution.  Hopefully this patch will be merged into future Arduino versions.

Other substitution issues may exist in Arduino 1.8.9.  Pluggable discovery is brand new.

Pluggable Discovery does not yet exist in Arduino’s Command Line Interface software.  Using Pluggable Discovery today means your board’s package can not work with the CLI.  Cristian started work on Pluggable Discovery for CLI months ago, but since January there appears to be no progress.  Hopefully as more people use Pluggable Discovery to support non-Serial protocols, we’ll see progress on bringing it to CLI as well.

Serial Monitor

Of course the other key Arduino IDE feature missing to truly support boards with any type of port is the serial monitor window.  And also the serial plotter.

Teensyduino currently adds several patches to the Arduino IDE to install a customized serial monitor.  It communicates, via stdin/stdout/stderr with a “teensy_serialmon” utility program.  Hopefully in the future, these patches can eventually be contributed to Arduino as as “Pluggable Serial Monitor” feature which complements Pluggable Discovery.

But at least for now, if you are interested in extending the Arduino IDE to work with boards using completely different communication protocols, hopefully the Pluggable Discovery contribution and this article can help.

Most Arduino SPI tutorials show this simple but poor SPI bus design:

Better SPI bus design can prevent conflicts.  3 simple improvements are needed:

1. Use pullup resistors on all chip select signals.

2. Verify tri-state behavior on MISO: use a tri-state buffer chip if necessary.

3. Protect bus access with SPI.beginTransaction(settings) and SPI.endTransaction().

Step 1: Pullup Resistors for Chip Select & Reset Signals

When multiple SPI devices are used, and especially when each is supported by its own library, pullup resistors are needed on the chip select pins.

Without a pullup resistor, the second device can “hear” and respond to the communication taking place on the first device, if that second device’s chip select pin is not pulled up.  This is easy to understand in hindsight, but it can be temendously confusing and frustrating to novice Arduino users who purchase shields or breakout boards without pullup resistors.  Each SPI device works when used alone, but they sometimes mysteriously fail when used together, only because both devices are hearing communication meant to initialize only the first device!

A simpe workaround for devices without pullup resistor involves adding code at the beginning of setup.

    void setup() {
      pinMode(4, OUTPUT);
      digitalWrite(4, HIGH);
      pinMode(10, OUTPUT);
      digitalWrite(10, HIGH);
      delay(1);
      // now it's safe to use SD.begin(4) and Ethernet.begin()
    }

Step 2: Proper MISO Tri-State Behavior

Most SPI chips will tri-state (effectively disconnect) their MISO pin when their chip select signal is high (inactive).

However, some chips do not have proper MISO tri-state behavior.  Fortunately, checking MISO tri-state is easy, especially when prototyping on a breadboard.  Just connect two 10K resistors to the MISO line, like this:

When all SPI chips are disabled, the MISO signal should “float” to approximately half the Vcc voltage.  If any device is still driving the MISO line, you’ll see a logic high (usually close to 3.3V or 5.0V) or logic low (close to zero volts).  This test is so easy, it should always be performed by designers of Arduino compatible products.

Arduino shields and breakout boards with poorly-behaved chips should always include a tri-state buffer.  Adafruit’s CC3000 breakout board is a good example:

Step 3: USB SPI Transactions in Software

Newer versions of Arduino’s SPI library support transactions.  Transactions give you 2 benefits:

• Your SPI settings are used, even if other devices use different settings
• Your device gains exclusive use of the SPI bus.  Others will not disturb you.

These improvements solve software conflicts, allowing multiple SPI devices to properly share the SPI bus.

A typical use of transactions looks like this:

    SPI.beginTransaction(SPISettings(14000000, MSBFIRST, SPI_MODE0));
    digitalWrite(chipSelectPin, LOW);
    SPI.transfer(mybyte1);
    SPI.transfer(mybyte2);
    digitalWrite(chipSelectPin, HIGH);
    SPI.endTransaction();

SPI.beginTransaction() takes a special SPISettings variable, which give the maximum clock speed, the data order, and clock polarity mode.  The speed is give as an ordinary number, expressing the maximum clock speed that device can use.  The SPI library will automatically select the fastest clock available which is equal or less than your number.  This allows your code to always use the best speed, even on board with different clock speeds.

If your code will ever call SPI library functions from within an interrupt (eg, from attachInterrupt), you must call SPI.usingInterrupt().  For example:

    SPI.begin();
    SPI.usingInterrupt(digitalPinToInterrupt(mypin));
    attachInterrupt(digitalPinToInterrupt(mypin), myFunction, LOW);

If you are developing a library that must be compatible with older versions of Arduino, which lack these SPI transaction functions, you can use SPI_HAS_TRANSACTION to check for the new version.  For example:

    #ifdef SPI_HAS_TRANSACTION
    SPI.beginTransaction(SPISettings(2000000, LSBFIRST, SPI_MODE1));
    #endif

Please Share and Use This Information

Many SPI-based products for Arduino do not work well together.  My hope is this information can help all makers of Arduino compatible devices to achieve much better compatibility.

Long-term, sharing of knowledge is needed.  Please share this information and ask makers of SPI devices and libraries to consider these suggestions.

This article may be shared and copied under the terms of the Creative Commons Attribution 4.0 International License.  Please, copy & share!  🙂

This article was originally published on the DorkbotPDX website, on November 24, 2014.  In late 2018, DorkbotPDX removed its blog section.  An archive of the original article is still available on the Internet Archive.  I am republishing this article here, in the hope it may continue to be found and used by everyone using SPI chips, and especially companies making SPI-based products for the Arduino community.

Often the question is asked, what is the simplest way to get modular synth control voltage (CV) into an analog input pin?

This simple circuit using only 3 resistors and 1 capacitor converts the -5V to +5V CV signal range to the 0 to 1.2V ADC input range.

For Teensy 3.2, 3.5 and 3.6, you would use analogReference(INTERNAL) to configure for the 0 to 1.2V range.  Then you can read the CV signal at any time with analogRead().

While this circuit is the simplest and easiest way, it is not necessarily the best way.  The input impedance, or load placed on the CV signal, it 31K ohms, rather than the standard 100K impedance normally used for modular synthesis systems.  This forum discussion goes into the circuit’s limitations and suggests more advanced ways using opamps.

Originally this article was published on the DorkbotPDX website.  Since that time, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve this info.  A copy of the original can also be found at the internet archive.

Years ago, in my slow quest for better photography of electronic projects, I built a light table to eliminate shadows.  Most of the white background photos you see on the PJRC site are shot with this light table.

This is how it turned out.

The build used boards from OSH Park (then “Laen’s PCB group order”), materials from TAP Plastics, 15 white LEDs and parts I mostly had laying around.

This view is inside a cheap 2 foot sized light tent I purchased from some ebay vendor, and a couple bright lights outside the tent on both side.

The LEDs are Cree CLM3C-WKW-CWBYA453, which are supposedly the same 5500K color temperature as the CFL lights outside the tent. Maybe that matters, maybe not, but it seemed like a good idea.

The Cree LED is a surface mount part, but fortunately Lean’s PCB group order made it very easy to convert to something I can solder wires onto. All the PCBs mount with double sticky mounting tape.

As you can see in this LED photo, there’s a bit of shadow. It’s a soft shadow due to the light tent casting light from many directions, but it’s still very present.  This is the type of shadow I’m hoping just a little bit of under side lighting will eliminate.

This little board is a constant current regulator. It takes a 0 to 5 volt input and regulates a 0 to 20 mA output current to a string of 5 LEDs. I wanted to make sure the current was perfectly constant since the camera might choose a quick shutter time.

Here’s the schematic for that circuit. At the time, it seemed like a good idea to sense the current using a resistor between the NPN transistor’s collector and the LEDs. The idea was any small change in ground potential between the board 0-5V control signal wouldn’t matter, if I ran separate signal and power ground lines.

But I didn’t consider the current draw though those resistors around the upper opamp. As you can see in the schematic, I change the values to about as high as I reasonably could. It still have a tiny bit of the lowest part of the range where the LEDs won’t completely turn off.

The 0 to 5 volt signal just comes from this potentiometer on the front panel.  Because it’s driving only the inputs to opamps, it doesn’t have any substantial load.  I still used a 1K pot anyways, though a higher value would have consumed a little less current.  I guess I didn’t care about an extra 5 mA.

The power for everything comes from this simple little power supply, which is just (approx) 24 volts unregulated, and a 5 volt regulated output from the pot, which is from a 7805 regulator.  Simple.

Of course, the opamp circuit isn’t perfect. After putting this together, I decided to try a different approach, sensing on the emitter side, and no current sensing path to add to the LED current! I also included 4.7K resistors on the feedback looks, and the positive inputs see about a 1K impedance. Any errors from the opamp’s input (PNP) bias currents should be small, and should be more on the negative than positive, so hopefully any tiny error will tend to reduce the LED current, not increase.

Then again, the original boards might work out ok, but Laen’s PCB group order makes it so very easy to just quickly draw up a (small) board.  Because the cost is so low, it’s easy to just send it off without all the worry the goes into a normal board order.

I still haven’t actually put this thing to use… the top plastic cover turned out to be just a bit too small, so I need to go shave it down to size on my table saw (which currently has a bunch of other project stuff piled on top of it). But soon, with a little luck, I’ll be able to take pictures of electronic stuff and adjust the light table to null out or at least soften away most of the little shadows that I still get, even with the light tent.

Originally this article was published on the DorkbotPDX website.  Since that time, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve this info.  A copy of the original can also be found at the internet archive.

Some time ago, I needed to actually “see” a current waveform in the 100 uA to 5 mA range with at least a couple MHz bandwidth.  This extremely expensive probe would have been perfect, but instead I built something similar for about $30 using the amazing Analog Devices AD8428 amplifier.

The first step was cutting the power trace and adding a resistor.  I used two 1 ohm resistors in parallel.

At 5 mA, this makes only 2.5 mV.  My scope’s supposed resolution is 1 mV, but the truth is there’s plenty of noise down in the 1 mV range.  That’s pretty common for most scopes, even pretty spendy ones.  So it’s just not feasible to measure this signal directly (not to mention using 2 probes and subtracting them in the scope).

That incredibly expensive Agilent probe probably has a couple really nice amplifiers inside…. so I went searching for an amplifier.  After a bit of searching, I found the AD8428.  It has a fixed gain of 2000 and a bandwidth of 3.5 MHz.  That’s a gain-bandwidth product of 7 GHz !!!  It’s also an extremely well matched instrumentation amp with an amazing CMRR of 140 dB.  So it gets rid of the power supply voltage and outputs the amplified signal referenced to ground.

The AD8428 is perfect.  It’s so very easy!  Of course, such amazing performance costs money: about $20.  Here’s that expensive little amplifier, and a 5V to +/- 15V power supply (about $10) to power it.

The one trick with measuring such tiny voltages is twisting the 2 sense wires together.  Honestly, I didn’t try it running them separately, but since this thing is getting voltages in only the microvoltage range for the lower measurements, I didn’t want to risk picking up noise.  I also put a 100 ohm resistor on the output, just in case I accidentally short the output or do something clumsy that might blow that little $20 part.

Here’s a scope screenshot using this little amplifier to “see” the current (the blue waveform).

In this test, the microcontroller is running in its slowest mode at only 10 kHz, drawing about 120 uA.  Then when the chip’s internal oscillator is started, the current jumps to about 600 uA.  Later, the CPU switches to actually clocking from that oscillator.  There’s an on-chip clock divider which is switched in and increased gradually.

The bottom trace (red) is the voltage on the chip’s 1.8V linear regulator.  It turns out that sudden jumps in current cause pretty substantial downward spikes on the regulated voltage.  This more gradual startup approach really helps.  This sort of thing is impossible to see with a slow multimeter, but with a reasonably good bandwidth measurement of the current, it’s easy to see what various code actually does to the current.

I tried connecting my multimeter to the amp output.  Sometimes it’s just a lot more convenient to look at a single number on a meter than fiddle with the scope.  I had been using the current mode on the meter before building this.  One thing I was surprised to find it my little meter updates its screen much faster while measuring about 125 mA than it does when measuring 125 uA.

Another interesting thing I’ve been noticing is patterns within the blue current waveform.  This Agilent scope has a “digital phosphor” rendering of the huge amount of data it collected.  This static screenshot can’t really capture the interactive experience of adjusting the waveform intensity, where various regions within the data change brightness differently, indicating there’s something interesting/different going on.  Even so, you can see several areas in the screenshot where interesting things are happening once the CPU is up and running.  It’s interesting how the current waveform changes as different code executes.

I know this isn’t anything terribly impressive… basically just buy a high-end amplifier and use it with a series resistor.  Maybe it even reads like an Analog Devices ad?  I’m not affiliated with Analog Devices… I just bought this part at normal qty 1 pricing from Digikey.

Still, this is the first time I’ve ever really looked at such low microcontroller currents with a few MHz bandwidth, and I’m guessing not many people have ever bothered to really measure such currents, so I thought I’d share.

Years ago I originally posted this article to the DorkbotPDX site.  Hackday published an article about it at the time.  Since then, the DorkbotPDX blog section has vanished.  I’m reposting it here, to preserve the info… for the next time I or others might need to do this sort of current measurement on a budget.

Online tech forums can feel intimidating.  Three simple things can greatly improve your experience & odds for useful help, regardless of your skill level.

Good First Impression

“You get 1 chance to make a good first impression” is timeless wisdom. Strangers will quickly form an opinion of you, based only on the words, images or video in your message.  Make your words count!

Showing you’ve made an effort, more than any other factor, will influence people to help.

But if you’re an absolute beginner, what sort of effort can reasonably be expected?  Anyone can try a Google search using the words from their question.  Many experts have long forgotten how difficult finding relevant info can be when you don’t know the proper terms, or unrelated tech has since started using those words.  Just explaining the search you tried and confusing or off-topic info you’ve seen can go a long way towards helping seasoned experts to understand your struggle.  The results aren’t important, your personal effort is what matters!

Genuine interest to learn also tends to make a good impression.  Even a few words can really communicate your attitude.  Frustrated but determined to learn from a trying experience can make a great impression.  Very smart people who can help the most tend to appreciate genuine & candid communication.

Of course if you have started a project or done work, mention it.  Or better, show what you have tried.  Screenshots, photos or even a quick video can powerfully demonstrate what you’ve done, and cover the other 2 aspects of a successful forum question.

Even just a few extra words, which show you are making a sincere effort and sincerely wish to learn, can make an improvement in the response you’ll likely receive.

Context Brings Understanding

Humans have amazing capability to genuinely understand.  Experts can sometimes apply their tremendous knowledge towards creative ideas to solve your problem… when they actually understand what you really need.

Providing context in your question is key.  What are your larger goals?  How does the question you’re asking fit into your project?  What do you ideally hope to accomplish?  And why?

When working on a technical matter, it’s easy to focus on only the issue at hand.  Remember, good people with vast knowledge & experience regularly read forums, just because they enjoy helping others.  When composing your forum question, don’t forget to give context.  It can make the difference between answers that are at best technically correct and answers that are truly insightful.

Details Are Essential

Modern technical issues can involve a mind boggling number of details.  How much raw information should your question include?  Before posting, try asking these questions:

1. Can readers SEE or EXPERIENCE your problem?
2. Can readers REPRODUCE your problem?

It’s easy to joke “I saw an error, but clicked OK without reading”.  But seriously, a basic level of detail allows everyone reading your message to see the problem.  A screenshot or exact copy of an error, plus specific information about software, hardware, and steps taken are a baseline for your readers to merely attempt to see the problem as you did.

Ideally, people reading your message would be able to reproduce the problem.  Providing this level of detail is not always practical.  But if you can give enough info, odds of someone solving the problem are vastly improved.

Software programming questions should include complete source code, not merely the small fragments.  Often code issues involve subtle details, such as declarations of variables elsewhere in the code.  Some experts can spot these sorts of problems with amazing speed, if you show complete code!  Don’t forget to be specific about any extra libraries used.

Electronic hardware issues usually need photos or accurate wiring diagrams, and links or part numbers of materials used.

It’s common to feel nervous or self-conscious about showing your project details on a forum.  Don’t be shy.  Experts who regularly read the forum genuinely want to help.  Show enough detail, give context and demonstrate you’ve made some effort and odds very good they will help!

Common Pitfalls

Forums don’t always work ideally.  Usually the 3 steps above will lead to good results, but there are a number of common problems to keep in mind.

Too Much Diagnosis

Technical problem solving involves observing the effects of an issue and trying to deduce the cause.  A very natural human tendency writing while in this mindset is to focus moreso on the presumed causes than objectively sharing the observed effects.

Usually a short break between an intense debugging session and composing your forum post can make a tremendous difference.  When writing, try to think of your reader’s perspective.  Are you asking people to help diagnose the problem, or merely asking them to nod “yup, you’re right” in agreement with your existing conclusions?

However, sharing your thought process can avoid people needlessly covering the same ground you’ve already investigated.  There is no one perfect way.  Just keep in mind that too much diagnosis can shut people out of helping you troubleshoot.

Venting Frustration

Let’s face it, some technical problems are really hard, even intensely frustrating.  When you’re annoyed, venting frustration is a very natural human tendency.

Before you post on a forum, creating that all-important first impression with strangers, review your message.  Some level of expressed frustration is normal and maybe even helpful.  It can show you’ve really tried.  Just beware the common trap, where your forum message ends up being perceived as a rant rather than a question or request for help.

Nobody Replies

Even under ideal conditions, sometimes forum posts get no replies.  This can feel disappointing & disheartening.  Try not to let that emotion take control as you compose a reply to “bump” the thread.  You can do much better.

Effort matters.  If days or weeks have elapsed since your first post, presumably you’ve put more effort into the project?  Even if fruitless, the personal effort you’ve invested since your prior post is something you can share that tends to draw people in.  Photos or code or other clear signs of work on your part can be highly effective.

Sometimes very difficult questions, with plenty of detail, get no replies simply because nobody knows the answer.  Or nobody knows with certainty.  Asking for opinions or ideas for directions to try can really help in these cases, turning what started as a solid question into a more fluid conversation.  Again, apparent effort is the key factor that attracts people to contribute towards helping solve a tough problem.

Most forums discourage or even ban posting duplicate copies of your question, or “cross posting”.  Posting the exact same message in multiple places at the same time is almost never a good idea.  Experts who regularly read the forums, the people most likely to help you, will almost certainly notice.  You get one chance to make a good first impression.  However, after a period with no answers, sometimes duplicate posting can be done gracefully, usually with an explanation & link to the prior unanswered question.

Academically Dishonest Students

Every tech forum gets lazy students asking people to do their homework.  Experts who regularly read forums are inundated by these low-quality questions.  Like an email inbox with spam, the last thing you want is your message to be mistaken for more junk & quickly dismissed.

If you are doing a student project, usually the best approach is to be honest about the academic nature (and deadline) and to make sure your sincere effort shines through.  Earnest effort and genuine desire to learn is what distinguishes good students in the minds of experts who will actually help!

Proprietary Projects

When you can’t share source code or other relevant technical details, technical forum help is rarely effective.

Usually you must invest extra effort to separate just the one piece that is problematic from the rest of your project, so it can be shared.  If you have not done this, consider that asking “has anyone seen a problem like XYX” is essentially asking humans to function as a search engine.  Sometimes the results are good (better than you could do with Google), but difficult technical problems without sharing relevant details are rarely solved by blind guessing.

If your employer or organization absolutely will not allow any code or details shared, no matter how trivial, perhaps using “enterprise” products which come with support contracts, or hiring a private consulting firm is more appropriate than asking on a public forum.

After Your Question Is Answered

The best way to say “Thank You” is to acknowledge who had the right answer.  Many experts pour countless unpaid hours into helping strangers on forums, simply because it feels good to be helpful.  Being recognized as supplying the correct answer is a nice reward.

When your problem is solved, please consider Google and other search engines may send people with similar technical questions to your messages for years to come.  The very best thing you can do after a technical problem is resolved is a quick but clear message confirming the solution.

If you posted on multiple forums, please take a few minutes to follow up on every thread with a link to the message with best answer.

About This Article (and Me)

Over the last 6 years, I’ve written 18620 answers our forum here at PJRC, and many more on numerous other forums.  During this time, we’ve managed to grow a pretty good forum community and help many thousands of people with projects.

I watched and tried to learn which what has worked and how we might improve.  One thing that’s become very clear is so many people need a little guidance on how to best ask their questions.  It’s not the technical nature, but these subtle human factors that matter most.

My hope is to see all tech forums improve.  Please, share this article.

-Paul Stoffregen