embed.dart

Code generation for embedding arbitrary file content into Dart code

Explore the docs »

Pub.dev · Report Bug · Request Feature

Motivation

Occasionally there are situations where we want to read non-Dart files for some reason, such as reading configuration values, or reading a test HTTP response for unit testing. A common way to do this is to load the file at runtime using File. However, since such files are usually bundled in the package, it would be nice to be able to read their contents directly from within the dart code without worrying about runtime errors and async I/O processing.

There are several ways to embed structured data into dart code. We can use multi-line string literal to embed a long text content, or Map literal to embed a structured data, or further, we can use the records to create a static structured data tree in a type-safe manner. However, there are still situations where reading a non-Dart file is required, because the file is downloaded from the Internet, or automatically generated by a script, or shared with another package written in a programming language other than Dart, etc. This is where embed comes in. The package solves this problem by generating code that allows to embed the contents of non-Dart files directly into the source file as literals.

Some of the other languages have a similar feature to this package, such as include_str macro from Rust, embed package from Go, and Javascript/Typescript's ability to directly import static JSON files as typed objects. Also, the C language has a similar feature: the #include delective. What the #include "header.h" actually means is that it tells the C preprocessor that "please replace me with the entire content of header.h", and interestingly, the #include delective can literally include any file other than *.h files as text. In fact, the following code works fine (might not be an intended use, but actually works fine):

// text.txt
"Hello world\n"

// main.c
#include <stdio.h>
int main(void) {
    const message =
    #include "text.txt"
    ; // ^^^^^^^^^^^^^ This line will be replaced with "Hello world\n"
    printf(message); // Displays "Hello world"
}

Index

• Motivation
• Index
• Installation
• Quickstart
• Examples
• How to use
  • Embed a text content as a string literal
  • Embed contents as binary
  • Embed a structured data as a Dart object
    • Preprocessing
    • How is the data type determined?
    • How to restrict the structure of data to be embedded?
• Troubleshooting Guide
  • I edited my json file to embed, but the generated code doesn't update even when I run build_runner again
• Roadmap
• Contributing
• Support
• Thanks
• Links

Installation

Run the following command:

flutter pub add embed_annotation dev:embed dev:build_runner

For a Dart project:

dart pub add embed_annotation dev:embed dev:build_runner

This command installs three packages:

• embed : the code generator
• embed_annotation : a package exposing annotations for embed
• build_runner : a tool to run code generators, published by the Dart team

Quickstart

Here's an example of embedding the content of the pubspec.yaml in the Dart code as an object:

// This file is 'main.dart'

// Import annotations
import 'package:embed_annotation/embed_annotation.dart';

// Like other code generation packages, you need to add this line
part 'main.g.dart';

// Annotate a top-level variable specifing the location of a content file to embed
@EmbedLiteral("../pubspec.yaml")
const pubspec = _$pubspec;

Then, run the code generator:

dart run build_runner build

If your are working in a Flutter project, you can also run the generator by:

flutter pub build_runner build

Finally, you should see the main.g.dart is generated in the same directory as main.dart.

// This is 'main.g.dart'

// GENERATED CODE - DO NOT MODIFY BY HAND

part of 'main.dart';

// **************************************************************************
// EmbedGenerator
// **************************************************************************

const _$pubspec = (
  name: "example_app",
  publishTo: "none",
  environment: (sdk: "^3.0.5"),
  dependencies: (embedAnnotation: (path: "../embed_annotation")),
  devDependencies: (
    buildRunner: "^2.4.6",
    lints: "^2.0.0",
    embed: (path: "../embed")
  ),
  dependencyOverrides: (embedAnnotation: (path: "../embed_annotation"))
);

You can see the content of the pubspec.yaml is embedded as a record object in the generated file. Let's print your package name to the console using this embedding:

print(pubspec.name); // This should display "example_app"

After modifying the original file, the pubspec.yaml in this case, you need to run the code generator again to update the embedded content. It is recommended to clear the cache before running the build_runner to avoid a code generation problem (see the troubleshooting guide for more information), as follows:

flutter pub run build_runner clean

Examples

You can find many more examples in the following resources:

• embed/test/literal/literal_embedding_generator_test_src.dart
• embed/test/str/str_embedding_generator_test_src.dart
• embed/test/binary/binary_embedding_generator_test_src.dart
• example/lib/example.dart

How to use

Currently, there are 3 types of embedding methods:

• Embed a text content as a String literal
• Embed contents as binary
• Embed a structured data as a Dart object

What content to embed and how to embed it can be described using predefined annotations. For example, you can use the EmbedStr annotation to embed text content as a string literal. Note that only top-level variables can be annotated, as shown below:

@EmbedStr(...) // This is OK
const topLevelVariable = ...;

class SomeClass {
  @EmbedStr(...) // This is invalid!
  static const classVariable = ...;
}

Each annotation needs at least one parameter, the location of the content file. There are 2 ways to specify the file location, a relative path and an absolute path. If you specify a relative path, it will be treated as relative to the parent directory of the source file where the annotated variable is defined. For example, suppose we have a simple Flutter project, typically structured as:

project_root
  |- lib
  |    |- main.dart
  |- pubspec.yaml 

In this scenario, we can refer the pubspec.yaml from the lib/main.dart using a relative path like ../pubspec.yaml :

@EmbedStr("../pubspec.yaml")
const pubspec = _$pubspec;

Depending on the structure of your project, it may be more intuitive to use an absolute path rather than a relative path:

@EmbedStr("/pubspec.yaml")
const pubspec = _$pubspec;

If you specify the content file path as an absolute path, as in the snippet above, it is treated as relative to the project root directory. In this example, the absolute path /pubspec.yam is interpreted by the code generator as /path/to/project/root/pubspec.yaml. Both methods can be used with all annotations, so choose one that suits your project structure.

Embed a text content as a string literal

Use EmbedStr to embed an arbitary file content in a source file as a string literal, as-is.

// main.dart
@EmbedStr("useful_text.txt")
const usefulText = _$usefulText;

By default, it embeds the text content as a raw string:

// main.g.dart
const _$usefulText = r'''
This is a useful text for you.
''';

If this doesn't work well with your text content, you can disable this behavior by setting EmbedStr.raw to false:

@EmbedStr("useful_text.txt", raw: false)

This will generates a regular string literal:

const _$usefulText = '''
This is a useful text for you.
''';

Embed contents as binary

Use EmbedBinary to embed a content file as a binary data.

@EmbedBinary("/assets/avator.png")
const avator = _$avator;

By default, the content is embedded as a List<int> literal.

const _$avator = [137, 88, 234, 85, ..., 13];

If you want to embed the content as a Base64 string literal, set EmbedBinary.bse64 to true.

@EmbedBinary("avator.png", base64: true)

The code generator will then have the following output:

const _$avator = 'iVBORw0KGgoAA...Sp8AAAAASUVORK5CYII=';

Embed a structured data as a Dart object

Use EmbedLiteral to convert a structured data file such as JSON to a dart object and embed it in a source file. This is useful when you want to read a non-Dart file bundled into your package in a type-safe way, without worrying about runtime errors and asynchronous I/O operations. Currently EmbedLiteral supports JSON, TOML and YAML files.

// main.dart
@EmbedLiteral("config.json")
const config = _$config;

If the config.json is like:

// This is just an example, don't care about the meaning of the content :)
{
  "url": "https://api.example.com",
  "api_key": "AJFKEl04i9jlsLJFXS9w09",
  "default": 2,
}

Then, the code generator will dump the following code:

// main.g.dart
const _$config = (
  url: "https://api.example.com",
  apiKey: "AJFKEl04i9jlsLJFXS9w09",
  $default: 2,
);

You can see that the given JSON data is converted as a record object. And if you take a closer look at the output, you may notice that some JSON keys are converted to camelCase. This is because it is the recommended style for record type field names.

One more thing, when a reserved keyword like if is used as a JSON key, the code generator automatically adds a $ sign at the beginning of the key; for example, in the above example, a JSON key default is converted to $default in the dart code.

Preprocessing

In the previous example, all JSON keys are converted to camelCase, and if any reserved Dart keywords are used as JSON keys, they are prefixed with a $ sign to avoid syntax errors. This processing is done by Preprocessors. You can specify preprocessors to be applied to the content in the constructor of EmbedLiteral.

@EmbedLiteral(
  "config.json", 
	preprocessors = [
    Preprocessor.recase, // e.g. converts 'snake_case' to 'snakeCase'
    Preprocessor.escapeReservedKeywords, // e.g. converts 'if' to '$if'
    Preprocessor.replace("#", "0x"), // e.g. converts "#fff" to "0xfff"
  ],
)
const config = _$config;

These preprocessors are applied recursively to all elements in the content, in the order specified. By default, Recase and EscapeReservedKeywords are applied, but you can disable this behavior by explicitly specifying an empty list to the preprocessors parameter:

@EmbedLiteral("config.json", preprocessors = const [])
const config = _$config;

How is the data type determined?

The code generator tries to represent map-like data as records rather than Maps whenever possible. For example, the following JSON file is converted to a record because the all the keys have a valid format as record field names:

{
  "snake_case": 0,
  "camelCase": "text",
  "PascalCase": true,
}

On the other hand, the next JSON will be converted as a Map<String, Object> because at least one of the keys has an invalid format as a record field name:

{
  "snake_case": 0, // This is fine
  "0_starts_with_number": "text", // BAD
  "contians *invalid* characters!": true, // BAD
}

In this case, the output code will be a Map literal:

// main.g.dart
const _$config = {
  "snake_case": 0,
  "0_starts_with_number": "text",
  "contians *invalid* characters!": true,
};

This rule is applied recursively if the input file contains nestd data structure, from root to leaf objects each time a map-like structure is converted to a literal representation.

How to restrict the structure of data to be embedded?

You can restrict the types of generated dart objects by specifying concrete types to annotated top level variables.

// Suppose you are only interested in the 'name' and 'publish_to' fields in the pubspec.yaml
typedef Pubspec = ({ String name, String publishTo });

@EmbedLiteral("/pubspec.yaml")
const Pubspec pubspec = _$pubspec; // Expects `_$pubspec` to be of type `Pubspec`

// Or if you prefer a Map to a Record
@EmbedLiteral("/pubspec.yaml")
const Map pubspecMap = _$pubspecMap;

Then, the build_runner will generates the following:

const _$pubspec = (name: "ExampleApp", publishTo: "none");
const _$pubspecMap = {"name": "ExampleApp", "publishTo": "none", "version": ... };

Troubleshooting Guide

I edited my json file to embed, but the generated code doesn't update even when I run build_runner again

It seems that the build_runner caches the previous output and if a source file has not changed from the previous one, it will not regenerate the code for that file. Since the source file does not change before and after modifinyg the json file, the updates are not reflected.

To avoid this problem, try removing the cache before running the build_runner as follows (replace flutter with dart if you are working in a Dart project):

flutter pub run build_runner clean && flutter pub run build_runner build

If you are still having the problem, also try this:

flutter clean && flutter pub run build_runner build

Roadmap

• Restrict the type of dart object to embed by giving the corresponding variable a concrete type ➡️ Available from v1.1.0

Contributors

Thanks to all the contributors!

• @NicolaVerbeeck: Support embedding binary files (#13)
• @bramp: Bumped the toml and lint dependencies to their latest(#17)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Support

Please give me a star on GitHub if you like this package. It will motivate me!

Thanks

• Best-README-Template by @othneildrew

Links

• API Documentation
• pub.dev (embed, embed_annotation)
• GitHub repository