kōtiri

Scaleconf 2018 notes

2018-08-07T00:00:00+12:00

Progressive Web Apps workshop

Fairly broad support as of iOS 11.3
No support for native push notifications
Still tricky to install, particularly on iOS
Course material

Kubernetes workshop

Worth considering if we want to be independent of the AWS ecosystem
Strong support on multiple providers (AWS, GCP, Azure, Catalyst)
Need to consider other related services along with compute/block/object, e.g. functions, managed SQL, NoSQL, queues
Course material

Scaling the clouds - a design approach

stratamap.io
Companies have 72 hours to notify affected users of a breach under GDPR. Is your tooling good enough to find affected users and what was exposed?
Archimate - an open successor of UML and BPMN
Useful for all kinds of mapping at multiple levels of detail - cloud infrastructure, customer journeys
Mandated by EU, NZ govt, etc
Archi tool

The Blockchain, a beginners guide to all the important bits

An introduction to blockchain, bitcoin, ethereum, etc

A token walks into a SPA

“Auth0 ambassador”
An explanation of JWTs and how they are useful
An analogy of JWTs as driver licenses
JWT payload does not have to be JSON - could be xml, protobuf
JWT algorithm overview
JWT handbook
How do you rotate private keys?
Token expiry should be measured in minutes

My service runs at 99.999%, tweets about outages are fake: it’s our competition trying to malign us!

Detect anomalous changes in traffic volumes
Your service may be fine during an outage if the problem is at an ISP or CDN - will you know? Users don’t care where the problem lies
Measured metrics can be gamed. Post-load time should be included in performance measurements.
Metrics drive behaviour and may have perverse incentives
Hanoi rat massacre of 1902
Your metrics may miss the point - you might have great uptime but with incorrect results. The fastest page on Bing is an error message
Another Bing example: a results page that was entirely ads: great click rate, great revenue, wrong data
Another Bing example measuring click rate: if they can answer a question inline on the page (e.g. population of Wellington), no need for a click
Metrics should be high quality, regularly reviewed
Consider “dialtone availability” - you should always pick up a phone and get a dialtone. You should always be able to get to Bing home page and do a search. Other features can have lower SLAs
Combine outside-in (outside your network) metrics with anomaly detection

Lessons learnt building Heptio Contour

Contour which maps Kubernetes to Envoy sidecar proxy
Ingress controller benefits:
- traffic consolidation: combining multiple LBs into one
- TLS management
- configuration abstraction (over haproxy/nginx/apache etc)
- path-based routing
overview of golang dependency awfulness and their workarounds
Keep docker build/push out of dev lifecycle
Try and keep integration tests to a minimum

Architecting for performance in the cloud

Uh, don’t do table scans?
Consider whole stack - code, network, infra, DB
track mean time to detection and mean time to recovery
Even Amazon doesn’t always get it right
Have a recovery plan

High Availability Microservices: Load Balancing in the World of Ingress and Service Mesh

A promo for Nginx Plus - can consolidate some functions
Cloud Native Landscape
Istio

Making test automation observable

Applying the five W’s to testing
- What are we testing?
- When would be find out about an issue?
- Where are our tests running?
- Why did something go wrong?
- Who should know about these tests?
Seemed not to be directly relevant to a CI-on-PR workflow, where the answers to all five are right there?
Built up a integration test coverage map by tracking User-Agent on web server logs against API paths
Try and reduce noise in your test failures - retries, cleaning up and consolidating errors
Apply the same test for your users. Their test coverage was much different from user behaviour
- What are our users doing?
- When would be find out if they have a problem

When two platforms become one: the complexities of merging 6 years of data

Focus on customers
Small, iterative changes
Having two brands travis-ci.{com,org} hurt acquisition, diluted
Tested new features on .org; paid customers felt left out they had to wait
Make changes user-facing ASAP

DevOps vs SRE - Get off my lawn

Fundamental conflict between agility and stability, traditional dev vs ops/sysadmin
Devs typically “won” as they were closest to business needs
c.a. 2009, resolved by putting dev+ops in the same room, talking
c.a. 2018, also means breaking down barriers in security, marketing, etc
Accept partial failure as normal, necessary for rapid progress
Use outages as opportunity for leaning (blameless postmortems)
Eliminate toil
Measure everything - MTTR, deploy times, deploy frequency, toil
Paying attention to your measurements is another story
“class SRE implements DevOps”
Reduce cost of failure (blue/green deployments and rollbacks)
“Automate this year’s job”
Books free until August 23
Avoid “superheroes” - might be useful early in a small company, bad as it scales

Hacking Web Performance

“CSS as appetizer, JavaScript as dessert” - slows load times
Nokia 1, brand new, $140 smartphone than runs Android 9, good example of a low-end phone
Try and fit your first load in 14.6KiB
Can fit more in that 14K with gzip
Can fit even more with zopfli, brotli
Embed as much as possible (styles, js, data urls)
Speed up HTTPS first load with HSTS
Consider HSTS preload too
Speed up first byte with QUIC
New cache headers: immutable, stale-while-revalidate
Prefetch, preconnect, and preload with <link> elements or Link: headers
React to bad connections a la Netflix dynamic quality changes
Use client hints, save-data, and device-memory
Books

How we automated our test database restores

All devs, managers, testers at FMG can restore a test database at any time
Had problems with polluted test data, slow restores, etc
Used a snapshot tool for SQL Server
Built a self-service tool for people to use

Serverless in Practise

Used Lambda, Firebase, Algolia
API GW -> λ GraphQL -> λ -> DynamoDB -> λ -> Firebase
λ cost $700/mo, compared to cloudfront $5100, cloudwatch $3100
netlify, cypress, jest
DLQ for failed λ
Can set λ max concurrency to avoid slamming SQL DB. Can set to 0 as a kill switch
Each function:
- should perform 1 task
- should perform 0 or 1 data transformations
- should have no state and be idempotent
- should have minimal permissions
- should avoid recursion
Use step functions or durable functions to manage state
Serverless Code Patterns
acloud.guru
API Proxy, fanout, inline stream transformation patterns
Serverless Architectures on AWS
Serverless Conf [I stumbled across this by accident NY ‘16’]
How do you avoid vendor lockin? Not so much a problem for code, much more for surrouding ecosystem: SQS, SNS, DynamoDB, step functions

Move legacy apps to Windows Containers to take advantage of modern infrastructure and orchestration

3 platform options: windows server core, nano server, .net core
No GUI
[some tips to do silent installs]
Install prerequisites first to avoid multi-step installs
ServiceMonitor.exe
[example Dockerfile]
cmd.exe /c start analogous to macOS open or Linux xdg-open
Can’t join activedirectory domains inside containers yet
Can almost run Linux containers in Windows

Data patterns catalogue: simplifying the journey from monolith to modern data architectures

Monoliths have been working for decades
Enterprise data warehouses have also accumulated data monoliths
Focus on asynchronous events, immutable log streaming, and microservices
Strangler pattern
“Collection of info available “in a few weeks”
Event Gateway: identify event seams, publish events when a monolith changes state
Batch to event adapter
Event to batch adapter
Change data capture
Data Catalogue and Schema Registry with avro, parquet, protobuf, thrift
Map event types to topics + streams: can validate events are going where they supposed to, support introspection

A comprehensive mapping of PostgreSQL, Diesel, and Rust types

2018-01-31T00:00:00+13:00

Over the last year or so I’ve been learning Rust as time permits. With the release of Diesel 1.0, I’ve recently kicked off a pet project that makes heavy use of PostgreSQL types.

I couldn’t find a mapping between the PostgreSQL type, the Diesel schema type, and the Rust type, so I’ve compiled a mapping for what I could find.

Happily all the types I use have been implemented, it seems there’s only XML and PostGIS types remaining.

Updated as at Postgresql 10.1, Diesel 1.1.

PostgreSQL Type	PostgreSQL Size	Description	Range	Diesel Type	Rust Type
Nullable Types		nullable		`Nullable<T>`	`Option<T>`
Numeric Types
`smallint`, `int2`	2 bytes	signed integer	-32768 to +32767	`SmallInt`	`i16`
`integer`, `int`, `int4`	4 bytes	signed integer	-2147483648 to +2147483647	`Integer`	`i32`
`bigint`, `int8`	8 bytes	signed integer	-9223372036854775808 to +9223372036854775807	`BigInt`	`i64`
`numeric(p, s)`, `decimal(p, s)`	2 bytes per 4 digits + 3 to 8 bytes	exact numeric of selectable precision	131072.16383 digits	`Numeric`	`bigdecimal::BigDecimal`
`real`, `float4`	4 bytes	single precision floating-point number	6 digits precision	`Float`	`f32`
`double precision`, `float8`	8 bytes	double precision floating-point number	15 digits precision	`Double`	`f64`
`smallserial`, `serial2`	2 bytes	autoincrementing integer	1 to 32767	`SmallInt`	`i16`
`serial`, `serial4`	4 bytes	autoincrementing integer	1 to 2147483647	`Integer`	`i32`
`bigserial`, `serial8`	8 bytes	autoincrementing integer	1 to 9223372036854775807	`BigInt`	`i64`
Monetary Types
`money`	8 bytes	currency amount	-92233720368547758.08 to +92233720368547758.07	`Money`	`Cents`
Character Types
`character varying(n)`, `varchar(n)`	n+1 or n+4 bytes	variable-length character string		`Text`	`String`, `&str`
`character(n)`, `char(n)`	n+1 or n+4 bytes	fixed-length character string		`Text`	`String`, `&str`
`text`	n+1 or n+4 bytes	variable-length character string		`Text`	`String`, `&str`
Binary Data Types
`bytea`	n+1 to n+4 bytes	binary data (“byte array”)		`Binary`	`Vec<u8>`, `&u8`
Date/Time Types
`timestamp`, `timestamp(p) without time zone`	8 bytes	date and time of day	4713 BC to 294276 AD, 1 microsecond	`Timestamp`	`chrono::NaiveDateTime`
`timestamptz`, `timestamp(p) with time zone`	8 bytes	date and time of day, with time zone	4713 BC to 294276 AD, 1 microsecond	`Timestamptz`	`chrono::DateTime`
`date`	4 bytes	calendar date (year, month, day)	4713 BC to 5874897 AD, 1 day	`Date`	`chrono::NaiveDate`
`time`, `time(p) without time zone`	8 bytes	time of day (no date)	00:00:00 to 24:00:00, 1 microsecond	`Time`	`chrono::NaiveTime`
`timetz`, `time(p) with time zone`	12 bytes	time of day (no date), with time zone	00:00:00+1459 to 24:00:00-1459, 1 microsecond
`interval(fields)(p)`	16 bytes	time span	-178000000 years to 178000000 years, 1 microsecond	`Interval`	`PgInterval`
Boolean Type
`boolean`, `bool`	1 byte	logical Boolean (true/false)		`Bool`	`bool`
Geometric Types
`point` `(x,y)`	16 bytes	geometric point on a plane
`line` `{A,B,C}`	32 bytes	infinite line on a plane
`lseg` `((x1,y1),(x2,y2))`	32 bytes	finite line segment on a plane
`box` `((x1,y1),(x2,y2))`	32 bytes	rectangular box on a plane
`path` `((x1,y1),...)`	16+16n bytes	closed geometric path on a plane
`path` `[(x1,y1),...]`	16+16n bytes	open geometric path on a plane
`polygon` `((x1,y1),...)`	40+16n bytes	closed geometric path on a plane
`circle` `<(x,y),r\>`	24 bytes	circle on a plane
Network Address Types
`cidr`	7 or 19 bytes	IPv4 or IPv6 network address		`Cidr`	`ipnetwork::IpNetwork`
`inet`	7 or 19 bytes	IPv4 or IPv6 host address		`Inet`	`ipnetwork::IpNetwork`
`macaddr`	6 bytes	MAC address		`MacAddr`	`[u8; 6]`
`macaddr8`	8 bytes	MAC address (EUI-64 format)
Enumerated Types
`enum`	4 bytes	enumerated value		(user-defined)	`String`, `enum`
Bit String Types
`bit(n)`	1 byte per 8 bits + 5 or 8 bytes	fixed-length bit string
`bit varying(n)`, `varbit`	1 byte per 8 bits + 5 or 8 bytes	variable-length bit string
Text Search Types
`tsvector`		text search document		`TsVector`
`tsquery`		text search query		`TsQuery`
UUID Type
`uuid`	16 bytes	universally unique identifier		`Uuid`	`uuid::Uuid`
XML Type
`xml`		XML data
JSON Types
`json`		textual JSON data		`Json`	`serde_json::Value`
`jsonb`		binary JSON data, decomposed		`Jsonb`	`serde_json::Value`
`Arrays`
`t[]`		array of values		`Array<T>`	`Vec<T>`, `Vec<Option<T>>`, `&[T]`, `&[Option<T>]`
Range Types
`int4range`		range of integer		`Range<Integer>`	`(Bound<i32>,` `Bound<i32>)`
`int8range`		range of bigint		`Range<BigInt>`	`(Bound<i64>,` `Bound<i64>)`
`numrange`		range of numeric		`Range<Numeric>`	`(Bound<bigdecimal::BigDecimal>,` `Bound<bigdecimal::BigDecimal>)`
`tsrange`		range of timestamp		`Range<Timestamp>`	`(Bound<chrono::NaiveDateTime>,` `Bound<chrono::NaiveDateTime>)`
`tstzrange`		range of timestamptz		`Range<Timestamptz>`	`(Bound<chrono::DateTime>,` `Bound<chrono::DateTime>)`
`daterange`		range of date		`Range<Date>`	`(Bound<chrono::NaiveDate>,` `Bound<chrono::NaiveDate>)`

Rails 4.0.0-beta1 STDOUT logger

2013-03-19T00:00:00+13:00

I use Unicorn and Heroku for one Rails application I maintain. I also use this fragment of code from the Heroku website:

config.logger = Logger.new(STDOUT)

This worked fine on Heroku, but locally I got an error after upgrading to Rails 4.0.0-beta1:

ArgumentError
wrong number of arguments (0 for 1)
activesupport (4.0.0.beta1) lib/active_support/core_ext/kernel/reporting.rb:82:in `capture'
activerecord (4.0.0.beta1) lib/active_record/migration.rb:345:in `call'

I stumbled across the solution after being sidetracked by a couple of Rails commits, it just needs to be scoped differently:

config.logger = ActiveSupport::Logger.new STDOUT

Rails 3.2.13 / Mail 2.5.3 OpenSSL error

2013-03-19T00:00:00+13:00

If you get an error like this:

OpenSSL::SSL::SSLError:
SSL_connect returned=1 errno=0 state=SSLv2/v3
read server hello A: unknown protocol

It might be because you are sending emails on localhost and don’t have SSL set up on your mail server.

The behaviour between Mail 2.4.4 (Rails 3.2.12) and Mail 2.5.3 (Rails 3.2.13) changed, and I needed to explicitly disable enable_starttls_auto:

config.action_mailer.smtp_settings = {
  :enable_starttls_auto => false
}

PHPUnit - Serialization of Closure is not allowed in Symfony 2

2013-03-18T00:00:00+13:00

If you get an error like this:

PHPUnit_Framework_Exception: PHP Fatal error:
Uncaught exception 'Exception' with message
'Serialization of 'Closure' is not allowed'
in vendor/symfony/src/Symfony/Component/HttpFoundation/Session.php:383

It might be because your unit test references an uninitialized variable or array index.

No, really.

(Thanks @ethos49)