A Tour Inside Arduino Core: Source Files, How to Make A New Core and Arduino Building Steps

Introduction

Diving into an open-source project code/design brings a lot of benefits to the diver especially if it’s a global and collaborative project which means that the diver will be exposed to different mindsets and smart hacks. Definitely, Arduino project is one of these projects. What adds even more uniqueness to it, is the new way of modularity and abstraction proposed in Arduino for Hardware (MCU world particularly) which was uncommon in the past. Arduino wants the developer’s energy to be focused  on solving the problem rather than reinventing the wheel by doing some low-level programming and writing a lot of boilerplate code to control the MCU’s hardware.

A Lot of developers hate Arduino style, and hate how a novice who doesn’t even know how registers work can program an MCU. Arduino made it very easy to make a full project without dealing with one single register. For developers intended to be experts or students from engineering background,  it’s not good to keep their eyes blind about low-level details because neither low-level bugs nor the application can be done effectively without knowing and mastering the details.

Personally, my old journey with Arduino began without knowing any low-level details but I had a personal eager to know the background magic behind the scenes. While my electronics experience was growing I could understand the Hardware well , and dealing with MCUs with bare-metal C in addition to  dealing with more than one MCU family made understanding the software (Arduino core for MCU) possible.

Arduino core is the place where all built-in functions like: pinMode(), digitalWrite(), analogRead(), Serial.begin() …etc are defined. This article is an attempt to understand the Arduino core structure, and even building a clone. Moreover, it also discusses the building process of Arduino sketches.

A Tour inside Arduino Core

As we said, Arduino core is the place where all built-in functions source files can be found. These tables summarize the most glaring things inside most of core source files grouped according to the functionality.

Generic Source Files

Source File	What Is Inside?
Arduino.h	1- Declaration of most built-in functions of Arduino like pinMode(), digitalWrite(), ...etc. 2- Macros of some constants like HIGH, LOW, INPUT, OUTPUT, ..etc. 3- Macro functions for bitwise operations and some other general operations like: min(), rand(), ..etc. 4- Declaration of pin and port mapping arrays(check our micro-blog about mapping arrays in Arduino) in flash memory to make a map between Arduino pin number and the physical number. I.e. mapping pin 13 to the according port and pin in the MCU registers. The value of these arrays can be found here: hardware\arduino\avr\variants\standard\pins_arduino.h
main.cpp	Here where the basic structure of Arduino program is declared. Here is the actual program: int main(void) { init(); initVariant(); setup(); for (;;) { loop(); } return 0; }

Source Files Related to Digital and Analog Ports

Source File	What Is Inside?
wiring_digital.c	Defines digitalWrite(), digitalRead(), pinMode(), ..etc.
wiring_analog.c	Defines analogRead(), analogWrite() ..etc.
wiring_shift.c	Two functions to read in/out serial bits-streams.
wiring_pulse.c	Two functions to read input PWM signals
WInterrupts.c	Defines some functions to handle external interrupts.

Source Files Related to Timing and Timers

Source File	What Is Inside?
wiring.c	1- Defines timing functions like millis(),delay() ..etc. 2- Timer0 overflow ISR 3- Initialization function used to initialize some peripherals before calling setup function in main.cpp.

Source Files Related to Hardware Serial (mostly)

Source File	What Is Inside?
Stream.h, Stream.cpp	Defines the Stream class which its methods are used (by inheritance) in HardwareSerial class like readUntil(), find(), ..etc.
Print.cpp, Print.h	Defines the Print class with methods to print messages (print & println). This is inherited by the Stream class.
Printable.h	This header defines a class that can be inherited by others to make any new object type printable by print() and println(). Check our micro-blog about Printable class.
HardwareSerial.h, HardwareSerial.cpp, HardwareSerial_private.h	1- Defines the HardwareSerial class. 2- Other methods like print(), find(), ..etc are inherited from Stream class.
HardwareSerialx.cpp	1- Defines UARTx ISR. 2- Declares the class object HardwareSerial Serialx.

Miscellaneous Source Files

Source File	What Is Inside?
New.cpp and new.h	New definitions for dynamic memory allocation functions.
binary.h	Macros of Bxxx notation in Arduino
Tone.cpp	Some functions to generate tones on some pins.
WCharacter.h	Defines some simple functions to process characters like: toAscii() using some function from ctype header.
Wmath.cpp	Some math functions like map() function.
wiring_private.h	Some scattered declarations.
WString.cpp & WString.h	Defines the String class in Arduino.

Files System Tour

The tour will focus on the part related to Arduino core only and the files related to the building process.

First, all source file of Arduino core is placed inside the following directory \Arduino\hardware\arduino\{the MCU architecture}\cores\arduino. Another place for Arduino third party cores (i.e. ESP8266 core) is in \Users\{username}\AppData\Local\Arduino15\packages and these cores are the ones downloaded from board manager (Tools>Boards>Board Manager). Tool-chain tools (and even avr-gcc header files) are located in Arduino\hardware\tools\{the MCU architecture} and here Users\{username}\AppData\Local\Arduino15\packages\arduino\tools.

How to Make a New Arduino Core

As an example, a clone for avr core will be made. To make a new Arduino core you need two main things:

1- JSON file to describe the new core: its name, versions, boards, tools dependencies, URLs to download files … etc.

2- Core source files packed in .zip, .tar.bz2 or .tar.gz file.
First, The JSON file name should be like “package_x_index.json” (or any other descriptive name) and contains the following template:

{
    "packages": [
        {
            "name": "aaa",
            "maintainer": "bbb",
            "websiteURL": "http://bbb.com/",
            "email": "aaa@bbb.com",
            "help": {
                "online": "http://bbb.com/"
            },
            "platforms": [
                {
                    "name": "aaa Boards",
                    "architecture": "avr",
                    "version": "1.0.0",
                    "category": "aaa",
                    "url": "http://bbb.com/aaa.zip",
                    "archiveFileName": "aaa.zip",
                    "checksum": "SHA-256:x",
                    "size": "x",
                    "help": {
                        "online": "http://bbb.com/"
                    },
                    "boards": [
                        {
                            "name": "Arduino/Genuino Uno"
                        }
                    ],
                    "toolsDependencies": [
                    {
                      "packager": "aaa",
                      "name": "avr-gcc",
                      "version": "4.9.2-atmel3.5.3-arduino2"
                    },
                    {
                      "packager": "aaa",
                      "name": "avrdude",
                      "version": "6.3.0-arduino6"
                    }
                            ]
                        }
                    ],
            "tools": []
        }
    ]
}

Now, change the JSON file information according to new core details. There are many online tools to calculate the SHA-256 checksum of core zip file like this one. Finally, Try to check other cores json files like esp8266 core JSON file.

The next step is to make the zip file. It should include one folder with the core name (aaa) and the core “template” file system inside (bootloaders, cores, doc, libraries, tools and variants folders and boards.txt, platform.txt and programmers.txt files).

As this article is about an overview not an in depth about making new core, you can check the job of each one of these files from the formal documentation.

To sum up, we will make a template clone called “atadait”. The steps:

1- A change in platform.txt:

name=Atadiat AVR Boards

version=1.0

2- A change in boards.txt to keep only one board in the new core (Arduino UNO).

# See: http://code.google.com/p/arduino/wiki/Platforms


menu.cpu=Processor


##############################################################


uno.name=Arduino/Genuino Uno


uno.vid.0=0x2341

uno.pid.0=0x0043

uno.vid.1=0x2341

uno.pid.1=0x0001

uno.vid.2=0x2A03

uno.pid.2=0x0043

uno.vid.3=0x2341

uno.pid.3=0x0243


uno.upload.tool=avrdude

uno.upload.protocol=arduino

uno.upload.maximum_size=32256

uno.upload.maximum_data_size=2048

uno.upload.speed=115200


uno.bootloader.tool=avrdude

uno.bootloader.low_fuses=0xFF

uno.bootloader.high_fuses=0xDE

uno.bootloader.extended_fuses=0xFD

uno.bootloader.unlock_bits=0x3F

uno.bootloader.lock_bits=0x0F

uno.bootloader.file=optiboot/optiboot_atmega328.hex


uno.build.mcu=atmega328p

uno.build.f_cpu=16000000L

uno.build.board=AVR_UNO

uno.build.core=arduino

uno.build.variant=standard


##############################################################

3- Making the zip file “atadiat.zip” and calculating the checksum using SHA online generator.

4- Making the JSON file according to the mentioned template.

5- To test this new core, Add JSON URL to Arduino IDE (https://atadiat.com/wp-content/uploads/package_atadiat_index.json)

Then go to board manager and download the new core.

Adding New Arduino Core to IDE Offline

If there is a need to add a core without hosting the core and JSON file on a website. The easy way to do that is to create a file with the core name in the following directory Documents\Arduino\hardware. The full directory would be like:

Documents\Arduino\hardware \{core name }\{the MCU architecture}

Arduino Building Process

Arduino native IDE has cryptic compiling-and-linking process which means that you can’t find a makefile defines the whole process of building code. Instead, the platform.txt file can give the developer the ability to change the building process a little, but some stages of building process will still be out of the developer control. Keep in mind that some information are retrieved from board.txt and used while the building process.

Building process in Arduino (for AVR architecture for instance) consists of the following stages in order ( to double check that, enable the output messages during compilation):

• The IDE detects the used board and core according to user choice and retrieves the corresponding information form board.txt.
• A temp directory is created in the following path Users\{username}\AppData\Local\Temp\arduino_build_x to store all compiled and temporary files in it. Object files for all Arduino core source code files and used libraries files will be there beside the output files of compilation like HEX file which will be uploaded later to the MCU by the programmer.

• Compiling sketch.
• Compiling the libraries source files.
• Compiling Arduino core source files with the following steps:
  • Compiling core C files.
  • Compiling core C++ files.
  • Compiling core assembly files.
• Combining everything in an archive file (.a) called core.a (it’s like putting all core object files in one file). What is archive file? Check the answer on Stackoverflow.
• Linking everything together to generate the output files like HEX file.

In a nutshell, what Arduino allows you to do with platform.txt is:

• Defining  your own compilation/building flags/variables to add them later to building commands.
• Changing the “pattern” of each stage mentioned previously.
• Changing building commands.
• Adding hooks to the building process.

Have a look how Arduino building patterns look like:

## Compile c files
recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.c.flags} -mmcu={build.mcu} -DF_CPU={build.f_cpu} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} {compiler.c.extra_flags} {build.extra_flags} {includes} "{source_file}" -o "{object_file}"
## Compile c++ files
recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpp.flags} -mmcu={build.mcu} -DF_CPU={build.f_cpu} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} {compiler.cpp.extra_flags} {build.extra_flags} {includes} "{source_file}" -o "{object_file}"
## Compile S files
recipe.S.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.S.flags} -mmcu={build.mcu} -DF_CPU={build.f_cpu} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} {compiler.S.extra_flags} {build.extra_flags} {includes} "{source_file}" -o "{object_file}"

These patterns are converted to repeated building commands

Thankfully, Arduino team has added since IDE 1.6.5 a way to add building steps as a hook to the predefined building process called pre and post build hooks.

As an example we will add a hook after core compilation. First, we will add some variables (variables name can be anything else):

compiler.hook.cmd = avr-gcc-nm

compiler.hook.cmd.flag = -g

The hook here is to list symbols inside an object file using a tool from GCC called nm. Second, we will add the hook:

recipe.hooks.core.postbuild.1.pattern = "{compiler.path}{compiler.hook.cmd}" {compiler.hook.cmd.flag} "{build.path}/core/wiring_digital.c.o"

Note: Multiple hooks can be added to the same stage by increasing the hook number.

Finally, If the developer needs even more control to the building process, then projects like Arduino-makefile should be considered. Arduino-makefile defines entire workflows for compiling code that can be customized.

Read More

• To know more about the internal definitions of some of Arduino API functions then consider reading bits & pieces chapters of this amazing Arduino in-depth book Embedded Controllers Using C and Arduino by James M. Fiore.
• Arduino 3rd party hardware specification
• Arduino core package json format specification
• Arduino Build-Process