ArticlesEmbedded C & MCU

A Tour Inside Arduino Core: Source Files, How to Make A New Core and Arduino Building Steps

Introduction

Diving into an open-source project code/design brings a lot of benefits to the diver especially if it’s a global and collaborative project which means that the diver will be exposed to different mindsets and smart hacks. Definitely, Arduino project is one of these projects. What adds even more uniqueness to it, is the new way of modularity and abstraction proposed in Arduino for Hardware (MCU world particularly) which was uncommon in the past. Arduino wants the developer’s energy to be focused  on solving the problem rather than reinventing the wheel by doing some low-level programming and writing a lot of boilerplate code to control the MCU’s hardware.

A Lot of developers hate Arduino style, and hate how a novice who doesn’t even know how registers work can program an MCU. Arduino made it very easy to make a full project without dealing with one single register. For developers intended to be experts or students from engineering background,  it’s not good to keep their eyes blind about low-level details because neither low-level bugs nor the application can be done effectively without knowing and mastering the details.

Personally, my old journey with Arduino began without knowing any low-level details but I had a personal eager to know the background magic behind the scenes. While my electronics experience was growing I could understand the Hardware well , and dealing with MCUs with bare-metal C in addition to  dealing with more than one MCU family made understanding the software (Arduino core for MCU) possible.

Arduino core is the place where all built-in functions like: pinMode(), digitalWrite(), analogRead(), Serial.begin() …etc are defined. This article is an attempt to understand the Arduino core structure, and even building a clone. Moreover, it also discusses the building process of Arduino sketches.

A Tour inside Arduino Core

As we said, Arduino core is the place where all built-in functions source files can be found. These tables summarize the most glaring things inside most of core source files grouped according to the functionality.

Generic Source Files

Source File
What Is Inside?
Arduino.h
1- Declaration of most built-in functions of Arduino like pinMode(), digitalWrite(), ...etc.
2- Macros of some constants like HIGH, LOW, INPUT, OUTPUT, ..etc.
3- Macro functions for bitwise operations and some other general operations like: min(), rand(), ..etc.
4- Declaration of pin and port mapping arrays(check our micro-blog about mapping arrays in Arduino) in flash memory to make a map between Arduino pin number and the physical number. I.e. mapping pin 13 to the according port and pin in the MCU registers. The value of these arrays can be found here: hardware\arduino\avr\variants\standard\pins_arduino.h
main.cppHere where the basic structure of Arduino program is declared. Here is the actual program:
int main(void)
{
init();
initVariant();
setup();
for (;;) {
loop(); }
return 0; }

Source Files Related to Digital and Analog Ports

Source File
What Is Inside?
wiring_digital.c
Defines digitalWrite(), digitalRead(), pinMode(), ..etc.
wiring_analog.c
Defines analogRead(), analogWrite() ..etc.
wiring_shift.c
Two functions to read in/out serial bits-streams.
wiring_pulse.c
Two functions to read input PWM signals
WInterrupts.c
Defines some functions to handle external interrupts.

Source Files Related to Timing and Timers

Source File
What Is Inside?
wiring.c
1- Defines timing functions like millis(),delay() ..etc.
2- Timer0 overflow ISR
3- Initialization function used to initialize some peripherals before calling setup function in main.cpp.

Source Files Related to Hardware Serial (mostly)

Source File
What Is Inside?
Stream.h, Stream.cpp
Defines the Stream class which its methods are used (by inheritance) in HardwareSerial class like readUntil(), find(), ..etc.
Print.cpp, Print.h
Defines the Print class with methods to print messages (print & println). This is inherited by the Stream class.
Printable.h
This header defines a class that can be inherited by others to make any new object type printable by print() and println(). Check our micro-blog about Printable class.
HardwareSerial.h, HardwareSerial.cpp,
HardwareSerial_private.h
1- Defines the HardwareSerial class.
2- Other methods like print(), find(), ..etc are inherited from Stream class.
HardwareSerialx.cpp
1- Defines UARTx ISR.
2- Declares the class object HardwareSerial Serialx.

Miscellaneous Source Files

Source File
What Is Inside?
New.cpp and new.h
New definitions for dynamic memory allocation functions.
binary.h
Macros of Bxxx notation in Arduino
Tone.cpp
Some functions to generate tones on some pins.
WCharacter.h
Defines some simple functions to process characters like: toAscii() using some function from ctype header.
Wmath.cpp
Some math functions like map() function.
wiring_private.h
Some scattered declarations.
WString.cpp & WString.h
Defines the String class in Arduino.

Files System Tour

The tour will focus on the part related to Arduino core only and the files related to the building process.

First, all source file of Arduino core is placed inside the following directory \Arduino\hardware\arduino\{the MCU architecture}\cores\arduino. Another place for Arduino third party cores (i.e. ESP8266 core) is in \Users\{username}\AppData\Local\Arduino15\packages and these cores are the ones downloaded from board manager (Tools>Boards>Board Manager). Tool-chain tools (and even avr-gcc header files) are located in Arduino\hardware\tools\{the MCU architecture} and here Users\{username}\AppData\Local\Arduino15\packages\arduino\tools.

How to Make a New Arduino Core

As an example, a clone for avr core will be made. To make a new Arduino core you need two main things:

1- JSON file to describe the new core: its name, versions, boards, tools dependencies, URLs to download files … etc.

2- Core source files packed in .zip, .tar.bz2 or .tar.gz file.
First, The JSON file name should be like “package_x_index.json” (or any other descriptive name) and contains the following template:

Now, change the JSON file information according to new core details. There are many online tools to calculate the SHA-256 checksum of core zip file like this one. Finally, Try to check other cores json files like esp8266 core JSON file.

The next step is to make the zip file. It should include one folder with the core name (aaa) and the core “template” file system inside (bootloaders, cores, doc, libraries, tools and variants folders and boards.txt, platform.txt and programmers.txt files).

Arduino core folder

As this article is about an overview not an in depth about making new core, you can check the job of each one of these files from the formal documentation.

platform.txt contains definitions for the CPU architecture used (compiler, build process parameters, tools used for upload, etc.) boards.txt contains definitions for the boards (board name, parameters for building and uploading sketches, etc.) programmers.txt contains definitions for external programmers (typically used to burn bootloaders or sketches on a blank CPU/board)

To sum up, we will make a template clone called “atadait”. The steps:

1- A change in platform.txt:

2- A change in boards.txt to keep only one board in the new core (Arduino UNO).

3- Making the zip file “atadiat.zip” and calculating the checksum using SHA online generator.

4- Making the JSON file according to the mentioned template.

5- To test this new core, Add JSON URL to Arduino IDE (https://atadiat.com/wp-content/uploads/package_atadiat_index.json)

arduino preferences json

Then go to board manager and download the new core.

arduino board manager

Adding New Arduino Core to IDE Offline

If there is a need to add a core without hosting the core and JSON file on a website. The easy way to do that is to create a file with the core name in the following directory Documents\Arduino\hardware. The full directory would be like:

Documents\Arduino\hardware \{core name }\{the MCU architecture}

Arduino Building Process

Arduino native IDE has cryptic compiling-and-linking process which means that you can’t find a makefile defines the whole process of building code. Instead, the platform.txt file can give the developer the ability to change the building process a little, but some stages of building process will still be out of the developer control. Keep in mind that some information are retrieved from board.txt and used while the building process.

Building process in Arduino (for AVR architecture for instance) consists of the following stages in order ( to double check that, enable the output messages during compilation):

arduino preferences compilation

  • The IDE detects the used board and core according to user choice and retrieves the corresponding information form board.txt.
  • A temp directory is created in the following path Users\{username}\AppData\Local\Temp\arduino_build_x to store all compiled and temporary files in it. Object files for all Arduino core source code files and used libraries files will be there beside the output files of compilation like HEX file which will be uploaded later to the MCU by the programmer.

Arduino temp folder

  • Compiling sketch.
  • Compiling the libraries source files.
  • Compiling Arduino core source files with the following steps:
    • Compiling core C files.
    • Compiling core C++ files.
    • Compiling core assembly files.
  • Combining everything in an archive file (.a) called core.a (it’s like putting all core object files in one file). What is archive file? Check the answer on Stackoverflow.
  • Linking everything together to generate the output files like HEX file.

In a nutshell, what Arduino allows you to do with platform.txt is:

  • Defining  your own compilation/building flags/variables to add them later to building commands.
  • Changing the “pattern” of each stage mentioned previously.
  • Changing building commands.
  • Adding hooks to the building process.

Have a look how Arduino building patterns look like:

These patterns are converted to repeated building commands

Arduino building pattern

Thankfully, Arduino team has added since IDE 1.6.5 a way to add building steps as a hook to the predefined building process called pre and post build hooks.

recipe.hooks.sketch.prebuild.NUMBER.pattern (called before sketch compilation) recipe.hooks.sketch.postbuild.NUMBER.pattern (called after sketch compilation) recipe.hooks.libraries.prebuild.NUMBER.pattern (called before libraries compilation) recipe.hooks.libraries.postbuild.NUMBER.pattern (called after libraries compilation) recipe.hooks.core.prebuild.NUMBER.pattern (called before core compilation) recipe.hooks.core.postbuild.NUMBER.pattern (called after core compilation) recipe.hooks.linking.prelink.NUMBER.pattern (called before linking) recipe.hooks.linking.postlink.NUMBER.pattern (called after linking) recipe.hooks.objcopy.preobjcopy.NUMBER.pattern (called before objcopy recipes execution) recipe.hooks.objcopy.postobjcopy.NUMBER.pattern (called after objcopy recipes execution) recipe.hooks.savehex.presavehex.NUMBER.pattern (called before savehex recipe execution) recipe.hooks.savehex.postsavehex.NUMBER.pattern (called after savehex recipe execution)

As an example we will add a hook after core compilation. First, we will add some variables (variables name can be anything else):

The hook here is to list symbols inside an object file using a tool from GCC called nm. Second, we will add the hook:

Note: Multiple hooks can be added to the same stage by increasing the hook number.

Arduino building hook

Finally, If the developer needs even more control to the building process, then projects like Arduino-makefile should be considered. Arduino-makefile defines entire workflows for compiling code that can be customized.

Read More

Yahya Tawil

Embedded Hardware Engineer interested in open hardware and was born in the same year as Linux. Yahya is the editor-in-chief of Atadiat and believes in the importance of sharing free, practical, spam-free and high quality written content with others. His experience with Embedded Systems includes developing firmware with bare-metal C and Arduino, designing PCB&schematic and content creation.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *