Personal notes from Elixir Conf 2019

Last week I attended Elixir Conf 2019 in Aurora, Colorado. It was my fourth Elixir Conf and by far the one that I engaged more with other people that I've never talked before, the hallway track was great and I could meet again some conference buddies from other editions. Another highlight was the conference hotel, a brand-new resort close to Denver, who knows me well understands how I like anything Colorado so I am glad the next edition will be in the same location.

The conference format was similar to previous years, it was three tracks, two days, with two extra days for an optional training, that I didn’t attend this time. The conference had 4 keynotes, José Valim (Elixir creator), Chris McCord (Phoenix creator) and Justin Schneck (Nerves co-creator), Dockyard team introducing Lumen, and great talks.

All the talks and keynotes are available in the Elixir Conf YouTube channel. Each keynote was focused in some areas, Elixir language and future, LiveView, Nerves and the brand-new Lumen, a project from Dockyard that uses Elixir in the client (browser).

As I always like to take notes when attending conferences and these are my highlights for Elixir Conf 2019. Please be advised that those notes are written like reminders for things I considered relevant during the talks and they are not a summary of them by any means. As the conference had many tracks, of course I couldn't attend all the talks, so feel free to comment with your favorite talk and notes:

Keynote: José Valim

Before anything, thank you very much José for the incredible language, it is a pleasure working full-time with Elixir and share the same community with great people, and I extend that to all Elixir core team as well.

Goal for 2019 is streamline Elixir in production

The main goal is enhance the ability to put applications in production easier, and to ensure what is running in production can be inspected.

The two main pieces are releases and telemetry.

Releases

• Elixir 1.9 introduced release commands as part of the language, with lots of lessons and concepts from Distillery;
• shellcheck to test shell scripts;

Telemetry

The library is composed by many pieces and helps snapshot what is going on inside running applications through metrics from the BEAM and custom ones.

• telemetry, telemetry_poller, telemetry_metrics, and telemetry_reports;
• Phoenix 1.5 will come with a Telemetry module to help define application specific metrics;

What is next to Elixir?

The next release, 1.10, will come with a different set up for its CI, now using Cirrus CI. It will also contain compiler tracing and ExUnit pattern diffing.

What is next to José Valim?

Now that the language enhancements are getting to a very stable point, and the community projects are being built and showcasing the power of the language, the efforts will be more directed to lower level (HiPE, Lumen and Hastega) and code analysis.

Keynote: Chris McCord

At Lonestar Elixir Conf 2019, Chris McCord presented the still-private project LiveView, which enables rich, real-time user experiences with server-rendered HTML. Right after the conference, the project was made available to the public and since then, a collection of projects is showcasing how interesting and powerful LiveView is. That also includes a Phoenix Phrenzy, a contest for LiveView projects.

In this keynote, he presents a few interesting points of LiveView and also what is coming next and the reasons behind.

LiveView library

• template engine;
• tests;
• navigation: live_redirect, live_link and handle_params for pagination URL changes;
• prepend, append, ignore and replace updates (phx-update);
• JS hooks when you need just a piece of extra Javascript;
• IE11 support;

Coming soon

• phx-debounce;
• file uploads;
• exception translation;
• stash for client state;
• automatic from recovery;

Keynote: Justin Schneck

In his keynote, Justin Schneck, again, gave a really nice demo to show the improvements Nerves is getting.

Resilient

• operating system: Erlang VM and Linux kernel;
• application domain: Nerves runtime allows other than Elixir, and resilience from Erlang VM;
• application data for specific data storage;

Reproducible

Read-only filesystem, immutable;

Reasonable

Whitelist approach (build up) with minimal output;

Nerves Hub

• CryptoAuthentication chip;
• NervesKey: write once;
• delegated auth;
• certificate;
• remote console;
• Terraform scripts to host your own Nerves Hub;

Keynote: Brian Cardarella, Paul Schoenfelder and Luke Imhoff

Introducing Lumen

Lumen is a new project from Dockyard that uses Elixir in the client. It uses WebAssembly and Rust as well in the compiler.

Compiler

The compiler has some unique constratins such as code size, load time and concurrency model.

Why not BEAM?

• Runtime: unavailable APIs, incompatible scheduler, JS managed types;
• Code size: BEAM bytecode is expensive, weak dead-code elimination;
• Performance: VM on a VM, JS engine unable to reason about bytecode;

New Compiler

• Restrictions: no hot-code, allow dead-code elimination;
• Ahead of time vs VM: only pay for what you use, no interpretation overhead;
• Build on existing tools: LLVM, Rust, wasm-bindgem;
• Key challenges: recursion and tail-call optimization, non-local returns, green threads, webassembly specific limitations;
• Continuations: represent jumps as calls to a continuation, never return, always moving forward;

Frontend
Accepts source in Erlang and Core Erlang, including mix task.

Middle tier
AST lowered to EIR.

Backend
Lowers from EIR to LLVM IR, generate object files.

Future goals
Push more data to LLVM, bootstrapping, MLIR, and embedded targets.

Runtime

Memory management
BEAM and Rust, with property-based testing.

Memory model
Process heap and pre-process garbage collection.

Processes
Very similar with what we have in Elixir. Code stack, mailbox, pid,
memory, links, and so on.

Schedulers
One per thread, main thread and webworkers.

WebAssembly main thread
Calls are blocking, scheduler needs to run timers.

Interacting with web
JS calling Lumen, and Lumen calling JS.

Why Lumen?

• Joe's paper about GUI in functional programming;
• Optimize front-end development;
• GUI is concurrent by nature, we could have a supervisor taking care
  of a DOM tree;

Phoenix LiveView Demystified: Alex Garibay

In this talk Alex showed some LiveView internals, how it works and how it works so well.

LiveView EEx

From template with sigils to AST

• static/vars and dynamic code;
• %Phoenix.LiveView.Rendered{} with static, dynamic and fingerprint;
• %Phoenix.LiveView.Comprehension{} to optmize data sent to the client;

Mounting the DOM

• rounter, controller or template with live and live_render macros;
• rendered template has few ids for channels and sessions;
• container <div> that receives the template can be configured to be any HTML tag;

Phoenix Channels

• uses Phoenix.LiveView.Socket;
• uses a specific channel "lv:*";
• socket receives session data and potentially user data;
• client triggers an event that is sent to the socket using %Phoenix.Socket.Message{};
• channel handles the event with callbacks;

Javascript

• import LiveView library;
• instantiate and connect LiveSocket;
• a JS view is created with LiveView container, socket and so on;
• each JS object in the view has static and dynamic data that is constructed for every change;
• uses Morphdom.js to re-render the changes in the DOM;

WebRTC from start to finish: Scott Hamilton

WebRTC (Web Real-Time Communication) is a free, open-source project that provides web browsers and mobile applications with real-time communication (RTC) via simple application programming interfaces (APIs). It allows audio and video communication to work inside web pages by allowing direct peer-to-peer communication, eliminating the need to install plugins or download native apps.

Janus is a general purpose WebRTC server that has an Elixir client available.

WebRTC

• spec and project;
• basic implementation in P2P;
• terminology: STUN, TURN and SDP;

Janus

• general purpose WebRTC gateway;
• JS, websocket;
• 101: session, plugin, peer connection, handle;
• resources to learn it are easy to find;

Elixir Phoenix Janus

Phoenix as middleman between client and Janus.

What could go wrong?

• Janus configuration - ice trickle;
• Janus on Docker;
• deployment;
• translation from Janus response to Elixir terms;
• mixing HTTP and WS calls;

Elixir + CQRS - Architecting for availability, operability, and maintainability at PagerDuty: Jon Grieman

PagerDuty has a feature that records all incident user logs and they use CQRS pattern to design that piece of functionality.

They use composite keys for tracing that can be order-able. Other benefits of the approach is having separation for monitoring and scaling.

Overall incident log system

• upstream systems;
• Kafka;
• creator;
• DB cluster;
• querier;
• client systems;

DB incident recovery

• whole stack that could be duplicated in region;
• replace the DB engine entirely after back to normal;
• operational benefits coming from ES and CQRS;

Date, Time, and Time Zones in Elixir 1.9: Lau Taarnskov

Handling date and time is a challenge in any language, in this talk we see the details behind Calendar in Elixir that had an upgrade in the version 1.9.

Differences between UTC, TAI and the concept of layers of wall time.

Elixir Calendar, Date and Time specifics

• sigils: ~D, ~T, ~N, ~U;
• chose the type by the meaning, not convenience;
• NaiveDateTime vs DateTime date/time comparison;
  - no data is better than fake/assumed data (JS and Ruby, for example);
  - correct data is better than assumptions;
• TimeZoneDatabase behaviour, as example tzdata;
• check date time stored to verify they are appending Z;

Mint, disrupting HTTP clients: Andrea Leopardi

Mint is a reasonable new low-level HTTP client that aims to provide a small and functional core that others can build on top.

Story and features

The initial need was caused by a potential removal of httpc from Erlang standard library. Mint is a processless client, that defines a wrapper as data structure for the connection, on top of gen_tcp. Mint knows how to handle raw bits and also HTTP protocol.

Streaming by default
Responses will arrive async (status, headers, body and so on).

Security

• httpc is not safe by default;
• hackney is safe by default but can be overridden if you work with `ssl_options`;
• mint is safe by default;
• SSL certificate store with castore;

Proxying
Mint has proxying support for request and tunnel proxy types.

HTTP2

• multiplexed streams;
• server push;
• backpressure;

Use cases

GenServer, GenStage, GenStatem with connection data. Also, it can be used as one process with many connections.

Challenges and future planes

• immutability;
• low level but usable;
• increase adoption;
• pooling;
• websockets;
• gRPC;

BEAM extreme: Miriam Pena

In her talk, Miriam showed us things to consider to improve performance, also alerted us that any tuning in the VM should be done only when needed, for very specific cases. Besides that, performance measuring should be done for a long time, and not using IEx.

One of the things she mentioned is that memory copy, something that happens a lot in the BEAM brings CPU overhead.

Pool bottleneck

• use process directly instead of GenServer, 80% faster;
• leverage process priority level;
• as a con, hard code readability;

Key-value storage

• process dictionary: hard to debug, tuple performance as an example;
• code generation;
• persistent term: access in constant time, write once read many, no copy to memory heap, tuple performance as an example;

NIFs

• for when all the rest fails;
• extreme memory usage;
• no VM guarantees;

Other suggestion is to keep OTP version updated as possible as new releases are always improving performance.

Contracts for building robust systems: Chris Keathley

In this talk, Chris presents some insight why Design by Contract should be considered and how his new library Norm can help projects in the data specification and generation aspect.

Breaking changes require coordination

• refactor;
• requirement (requires no breaking changes);
• technology;

Contracts

• enforcing callers are correct (pre-conditions);
• ensure the function is correct (post-conditions);
• ExContract;

Data specification

• Norm;
• using in combination with ExContract;
• supports schema, optionality;

Testing strategy

• Norm uses stream data to generate data for testing;
• can be used with property-based test;

Writing an Ecto adapter, introducing MyXQL: Wojtek Mach

Even not personally interested in MySQL as prefer and use only Postgres, I was willing to know more about internals of Ecto and how Ecto uses its adapters to connect and interact with databases.

Driver

• :gen_tcp for database connection;
• library binpp;
• Wireshark app;

MySQL packet

• payload length;
• sequence id;
• packet payload;
• leverages Elixir binary pattern matching;
• initial handshake package;
• handshake response;

Building the adapter

• encoding and decoding: function for every data type, OK_Packet with defrecord;
• DBConnection behaviour: maintain a connection pool, does not overload the database, reconnect, support to common database features, and it needs to be fast;

Connection

• start N connections using DBConnection based on the pool configuration;
• fetching results preparing and executing the query;
• other functions as disconnect, checkout, ping, and so on;

Ecto integration

• all the features Postgres adapter has;
• implements Ecto.Adapter, Ecto.Adapter.Queryable, Ecto.Adapter.Schema, Ecto.Adapter.Storage, and Ecto.Adapter.Transaction;
• constraints;
• data normalization: boolean as an example, as in MySQL it is set as 1 and 0;
• integration tests;

Kubernetes at small scale: Phil Toland

Kubernetes has great benefits, even being a not so easy implementation. Some of the benefits are improved resource efficiency and reduced cost, and operational scalability. In this talk Phil described his process to implement Kubernetes at Hippware.

Main components

• control plane (leave it alone);
• workers;
• workload:
  - pod;
  - deployment (how your pod runs in the cluster);
  - service (expose a pod to outside, expose a port);
  - config map;

Lessons learned

• outsource as much as possible;
• automate all the things;
• pods are ephemeral;
• automatic clustering: via libraries as libcluster and peerage;
• deploying: via libraries as kubernetes-deploy and swarm;
• one instance per node: anti-affinity specification;

ETS Versus ElasticSearch for Queryable Caching: David Schainks

In this talk David compares the characteristics of ElasticSearch that is well-known as a great solution for search and cached data, with Elixir/Erlang out-of-box solutions such as ETS and Persistent Term, listing the pros and cons of each option.

ElasticSearch

• filtering, auto completion and full text search;
• performant;
• queryable cache;
• operational cost: another DSL, integration time, expensive, and configuration gotchas;

ETS

• no garbage collection;
• use cases: preset configuration, cached data, message buffer;
• filtering with match/2, match_object/1 and fun2ms/1;
• auto completion with fun2ms/1;
• full text search: using Joe Armstrong's elib1 library;

Real world concerns

• performance with large data sets;
• data ranking;

Operational concerns

• high availability;
• synchronization;
• index changes;

Persistent term

• no filtering;
• much faster than ETS;
• garbage collection on update;

UI Testing is Ruff; Hound Can Help: Vanessa Lee

Whether you call it UI testing, end-to-end testing, end-to-user testing, or acceptance testing, it is often an intensely manual and time-consuming process. An Elixir library, Hound, can carry some of the load through browser automation.

Hound takes screenshots of tests and stores in the test folder.

Library test helpers

• check page elements;
• fill, execute actions and submit form;
• manage sessions;

Property-based testing

• combining Hound with property-based libraries is very straightforward;
• using custom StreamData generator using bind/2 helper;

While handy, there are some gotchas when elements are not ready or if the application is heavily dependent in Javascript events/side effects;