StreamUtils: Power-ups for Arduino Streams

The stream is an essential abstraction in Arduino; we find it in many places:

• HardwareSerial
• SoftwareSerial
• File
• EthernetClient
• WiFiClient
• Wire
• and many others...

This library provides some helper classes and functions for dealing with streams.

For example, with this library, you can:

• speed up your program by buffering the data it reads from a file
• reduce the number of packets sent over WiFi by buffering the data you send
• improve the reliability of a serial connection by adding error correction codes
• debug your program more easily by logging what it sends to a Web service
• send large data with the Wire library
• use a String, EEPROM, or PROGMEM with a stream interface
• decode HTTP chunks

Read on to see how StreamUtils can help you!

How to add buffering to a Stream?

Buffering read operations

Sometimes, you can significantly improve performance by reading many bytes at once. For example, according to SPIFFS's wiki, reading files in chunks of 64 bytes is much faster than reading them one byte at a time.

To buffer the input, decorate the original Stream with ReadBufferingStream. For example, suppose your program reads a JSON document from SPIFFS like this:

File file = SPIFFS.open("example.json", "r");
deserializeJson(doc, file);

Then you only need to insert one line to greatly improve the reading speed:

File file = SPIFFS.open("example.json", "r");
ReadBufferingStream bufferedFile{file, 64};  // <- HERE
deserializeJson(doc, bufferedFile);

Unfortunately, this optimization is only possible if:

1. Stream.readBytes() is declared virtual in your Arduino Code (as it's the case for ESP8266), and
2. the derived class has an optimized implementation of readBytes() (as it's the case for SPIFFS' File).

When possible, prefer ReadBufferingClient to ReadBufferingStream because Client defines a read() method similar to readBytes(), except this one is virtual on all platforms.

If memory allocation fails, ReadBufferingStream behaves as if no buffer was used: it forwards all calls to the upstream Stream.

Adding a buffer only makes sense for unbuffered streams. For example, there is no benefit to adding a buffer to serial ports because they already include an internal buffer.

Buffering write operations

Similarly, you can improve performance significantly by writing many bytes at once. For example, writing to WiFiClient one byte at a time is very slow; it's much faster if you send large chunks.

To add a buffer, decorate the original Stream with WriteBufferingStream. For example, if your program sends a JSON document via WiFiClient like this:

serializeJson(doc, wifiClient);

Rewrite it like this:

WriteBufferingStream bufferedWifiClient{wifiClient, 64};
serializeJson(doc, bufferedWifiClient);
bufferedWifiClient.flush();

flush() sends the remaining data; if you forget to call it, the end of the message will be missing. The destructor of WriteBufferingStream calls flush(), so you can remove this line if you destroy the decorator immediately.

If memory allocation fails, WriteBufferingStream behaves as if no buffer was used: it forwards all calls to the upstream Stream.

Adding a buffer only makes sense for unbuffered streams. For example, there is no benefit to adding a buffer to serial ports because they already include an internal buffer.

How to add logging to a stream?

Logging write operations

When debugging a program that makes HTTP requests, you first want to check whether the request is correct. With this library, you can decorate the EthernetClient or the WiFiClient to log everything to the serial.

For example, if your program is:

client.println("GET / HTTP/1.1");
client.println("User-Agent: Arduino");
// ...

Then, create the decorator and update the calls to println():

WriteLoggingStream loggingClient(client, Serial);
loggingClient.println("GET / HTTP/1.1");
loggingClient.println("User-Agent: Arduino");
// ...

Everything you write to loggingClient is written to client and logged to Serial.

Logging read operations

Similarly, you often want to see what the HTTP server sent back. With this library, you can decorate the EthernetClient or the WiFiClient to log everything to the serial.

For example, if your program is:

char response[256];
client.readBytes(response, 256);

Then, create the decorator and update the calls to readBytes():

ReadLoggingStream loggingClient(client, Serial);
char response[256];
loggingClient.readBytes(response, 256);
// ...

loggingClient forwards all operations to client and logs read operations to Serial.

⚠ WARNING ⚠
If your program receives data from one serial port and logs to another, ensure the latter runs at a much higher baud rate. Logging must be at least ten times faster, or it will slow down the receiving port, which may drop incoming bytes.

Logging read and write operations

Of course, you could log read and write operations by combining ReadLoggingStream and WriteLoggingStream, but there is a simpler solution: LoggingStream.

As usual, if your program is:

client.println("GET / HTTP/1.1");
client.println("User-Agent: Arduino");

char response[256];
client.readBytes(response, 256);

Then, decorate client and replace the calls:

LoggingStream loggingClient(client, Serial);

loggingClient.println("GET / HTTP/1.1");
loggingClient.println("User-Agent: Arduino");

char response[256];
loggingClient.readBytes(response, 256);

How to use error-correction codes (ECC)?

StreamUtils supports the Hamming(7, 4) error-correction code, which encodes 4 bits of data into 7 bits by adding three parity bits. These extra bits increase the amount of traffic but allow correcting any one-bit error within the 7 bits.

If you use this encoding on an 8-bit channel, it effectively doubles the amount of traffic. However, if you use an HardwareSerial instance (like Serial, Serial1...), you can slightly reduce the overhead by configuring the ports as a 7-bit channel, like so:

// Initialize serial port with 9600 bauds, 7 bits of data, no parity, and one stop bit
Serial1.begin(9600, SERIAL_7N1);

Adding parity bits

The class HammingEncodingStream<7, 4> decorates an existing Stream to include parity bits in every write operation.

You can use this class like so:

HammingEncodingStream<7, 4> eccSerial(Serial1);

eccSerial.println("Hello world!");

Like every Stream decorator in this library, HammingEncodingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

Correcting errors

The class HammingDecodingStream<7, 4> decorates an existing Stream to decode parity bits in every read operation.

You can use this class like so:

HammingDecodingStream<7, 4> eccSerial(Serial1);

char buffer[256];
size_t n = eccSerial.readBytes(buffer, n);

Like every Stream decorator in this library, HammingDecodingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

Encoding and decoding in both directions

The class HammingStream<7, 4> combines the features of HammingEncodingStream<7, 4> and HammingDecodingStream<7, 4>, which is very useful when you do two-way communication.

You can use this class like so:

HammingStream<7, 4> eccSerial(Serial1);

eccSerial.println("Hello world!");

char buffer[256];
size_t n = eccSerial.readBytes(buffer, n);

Like every Stream decorator in this library, HammingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

How to retry write operations?

Sometimes, a stream is limited to the capacity of its internal buffer. In that case, you must wait before sending more data. To solve this problem, StreamUtils provides the WriteWaitingStream decorator:

This function repeatedly waits and retries until it times out. You can customize the wait() function; by default, it's yield().

For example, if you want to send more than 32 bytes with the Wire library, you can do the following:

WriteWaitingStream wireStream(Wire, [](){
  Wire.endTransmission(false); // <- don't forget this argument
  Wire.beginTransmission(address);
});

Wire.beginTransmission(address); 
wireStream.print("This is a very very long message that I'm sending!");
Wire.endTransmission();

As you can see, we use the wait() function as a hook to flush the Wire transmission buffer. Notice that we pass false to endTransmission() so that it sends the data but doesn't actually stop the transmission.

How to use a String as a stream?

Writing to a String

Sometimes, you use a piece of code that expects a Print instance (like ReadLoggingStream), but you want the output in a String instead of a regular Stream. In that case, use the StringPrint class. It wraps a String within a Print implementation.

Here is how you can use it:

StringPrint stream;

stream.print("Temperature = ");
stream.print(22.3);
stream.print(" °C");

String result = stream.str();

At the end of this snippet, the string result contains:

Temperature = 22.30 °C

Reading from a String

Similarly, there are cases where you have a String, but you need to pass a Stream to some other piece of code. In that case, use StringStream; it's similar to StrintPrint, except you can also read from it.

How to use EEPROM as a stream?

SteamUtils also allows using EEPROM as a stream. Create an instance of EepromStream and specify the start address and the size of the region you want to expose.

For example, it allows you to save a JSON document in EEPROM:

EepromStream eepromStream(0, 128);
serializeJson(doc, eepromStream);
eepromStream.flush();  // <- calls EEPROM.commit() on ESP (optional)

In the same way, you can read a JSON document from EEPROM:

EepromStream eepromStream(0, 128);
deserializeJson(doc, eepromStream);

How to use PROGMEM as a stream?

SteamUtils also allows reading PROGMEM buffers with a Stream interface.

Create an instance of ProgmemStream and pass the pointer to the PROGMEM buffer.

const char buffer[] PROGMEM = "This string is in program memory"
ProgmemStream stream{buffer};
Serial.println(stream.readString());

ProgmemStream's constructor also supports const __FlashStringHelper* (the type returned by the F() macro) and an optional second argument to specify the size of the buffer.

How to decode HTTP chunks?

HTTP servers can send their response in multiple parts using Chunked Transfer Encoding. Clients using HTTP 1.1 must support this encoding as it's not optional and is dictated by the server.

ChunkDecodingStream and ChunkDecodingClient are decorators that decode the chunks and make the response available as a regular stream.

Here is an example using HTTPClient:

// Initialize HTTPClient
HTTPClient http;
http.begin(client, url);

// Tell HTTPClient to collect the Transfer-Encoding header
// (by default HTTPClient discards the response headers)
const char *keys[] = {"Transfer-Encoding"};
http.collectHeaders(keys, 1);

// Send the request
int status = http.GET();
if (status != 200) return;

// Create the raw and decoded stream
Stream& rawStream = http.getStream();
ChunkDecodingStream decodedStream(http.getStream());

// Choose the stream based on the Transfer-Encoding header
Stream& response = http.header("Transfer-Encoding") == "chunked" ? decodedStream : rawStream;

// Read the response
JsonDocument doc;
deserializeJson(doc, response);

// Close the connection
http.end();

Note that HTTPClient already performs chunk decoding if you use getString(), but you might want to use getStream() to avoid buffering the entire response in memory.

Also, you can avoid chunked transfer encoding by downgrading the HTTP version to 1.0. HTTPClient allows you to do that by calling useHTTP10(true) before sending the request.

Summary

Some of the decorators are also available for the Print and Client classes. See the equivalence table below.

Purpose	Client	Stream	Print
Log write operations	WriteLoggingClient	WriteLoggingStream	LoggingPrint
Log read operations	ReadLoggingClient	ReadLoggingStream
Log read and write op.	LoggingClient	LoggingStream
Buffer write operations	WriteBufferingClient	WriteBufferingStream	BufferingPrint
Buffer read operations	ReadBufferingClient	ReadBufferingStream
Repeat write operations	WriteWaitingClient	WriteWaitingStream	WaitingPrint
Use String as a stream		StringStream	StringPrint
Use EEPROM as a stream		EepromStream
Use PROGMEM as a stream		ProgmemStream
Error correction (decode only)	HammingDecodingClient	HammingDecodingStream
Error correction (encode only)	HammingEncodingClient	HammingEncodingStream	HammingPrint
Error correction (encode & decode)	HammingClient	HammingStream
Decode HTTP chunks	ChunkDecodingClient	ChunkDecodingStream

Prefer XxxClient to XxxStream because, unlike Stream::readBytes(), Client::read() is virtual on all cores and therefore allows optimized implementations.

Portability

This library relies on Client, Print, and Stream definitions, which unfortunately differ from one core to another.

It has been tested on the following cores:

• AVR
• DxCore
• ESP32
• ESP8266
• mbed
• megaAVR
• nRF52
• RP2040
• SAMD
• STM32 Official
• STM32 Roger's Core (no EEPROM support)
• Teensy

If your core is not supported, please open an issue. Thank you for your understanding.