Self-host Spinifex

What's Changed

DNS (Route 53)

• Route 53 public hosted zones: manage zones and records with the AWS CLI, SDKs, Terraform or the console
• A zone goes live once its parent domain delegates to the cluster's name servers, returned by CreateHostedZone
• GetChange reports INSYNC only when every DNS server is serving the change
• IAM authorises each call per hosted zone, with AmazonRoute53FullAccess and AmazonRoute53ReadOnlyAccess managed policies
• New DNS console guides you through adding a domain and edits A, AAAA, CNAME, MX, TXT, SRV and CAA records
• New spx admin dns commands verify, activate and release zones

Security

• System AMIs and the load balancer microVM now ship patched guest kernels
• The load balancer microVM moves off the end-of-life Alpine 3.20 to Alpine 3.24
• AddRoleToInstanceProfile now requires iam:PassRole on the role
• IMDSv2 token responses now honour HttpPutResponseHopLimit
• Per-instance IMDS servers now limit connections and enforce timeouts
• RunInstances caps instance counts, and vCPU quota checks can no longer overflow
• Key pair names that try to escape the account's key storage are rejected
• spx admin cert renew keeps the internal service names and management IP on the reissued certificate
• Signing out or in ends the console session in every open tab

Identity (IAM / STS / IMDS)

• GetUser without UserName returns the calling user
• Presigned GetCallerIdentity with GetSessionToken credentials resolves to the IAM user
• DeleteUser succeeds even when one of the user's access keys is already gone
• Updating a role no longer re-attaches a policy that was detached at the same time
• IMDS public-ipv4 follows Elastic IP associations
• IMDS serves the new SSH key after a key pair is re-imported under the same name

Compute (EC2)

• DescribeVolumes supports the snapshot-id filter
• DeleteSnapshot is refused while a registered AMI uses the snapshot
• The DeleteSnapshot in-use error names the volume that blocks the delete
• DescribeInstanceStatus no longer lists platform-managed instances
• RunInstances returns InsufficientInstanceCapacity instead of an internal messaging error when a node can't take the launch
• Lifecycle calls on unknown or malformed instance IDs return the correct NotFound or Malformed error
• ModifyLaunchTemplate returns the template NotFound error when the template is deleted at the same time
• Deleting a launch template removes every version
• Deleting a launch template or placement group also clears its tags

Load Balancing (ELBv2)

• Target groups now return IpAddressType and LoadBalancerArns, which removes Terraform drift
• AddTags and RemoveTags check every ARN before tagging anything, so a bad ARN no longer causes a partial update

Certificates (ACM)

• Changing a certificate's tags no longer drops its private key

Reliability

• IMDS stays reachable after an Open vSwitch restart
• Concurrent EC2 tag changes no longer overwrite each other
• Deleting a launch template version is safe while the default version is being changed at the same time
• Changing an instance's type is refused while a node is unresponsive
• DeleteVolume and ForceDetachVolume return an error, not NotFound, when a volume record can't be read
• Cluster shutdown counts each node's acknowledgement only once, so a duplicate can't hide a silent node
• Account termination stops if the instance inventory is incomplete
• Account termination confirms no node still holds a volume before force-deleting it
• A stalled client connection can no longer block the console's TLS listener

Operator Tooling

• New spx admin storage status shows S3 metadata node raft state
• New spx admin init --external-dns sets DNS servers for the external IP pool
• Service Metrics reads single-host S3 storage health correctly
• OCI topology validation can resume after a failure, without reinstalling or repeating completed setup

What's Changed

Oracle Cloud Infrastructure (OCI) support

• Spinifex now deploys on Oracle Cloud (OCI) on both VM and bare-metal shapes
• OCI deployments can use an instance principal or IAM for authentication to provide VNIC, public IP and private IP support to Spinifex resources (EC2, EKS, ALB, NLB, etc)
• Spinifex provided Terraform workbooks (EC2, S3, RDS, EKS, ECS, VPC) validated on OCI as a deployment
• New E2E testing path to validate Spinifex on OCI (single-vm, multi-VM, bare-metal)
• Oracle Linux 9.8 and Oracle Linux 10.1 support for EC2 guests
• Install Guide available, create a Spinifex deployment on OCI in under 10 minutes.

Deployment

• aws s3 works out of the box with profiles created by admin init and account create
• spx prints plain output when stdout is not a terminal, or when NO_COLOR or TERM=dumb is set

Identity (IAM / STS / IMDS)

• Identity policies now accept NotAction and NotResource
• Policy conditions now accept negated and case-insensitive string operators, plus NotIpAddress, Null and ...IfExists
• Date and numeric condition operators now work, with aws:CurrentTime, aws:EpochTime and s3:max-keys supplied at evaluation time
• iam:PassRole checks now supply iam:PassedToService
• EKS cluster and node roles now require iam:PassRole, as AWS does
• Per-type policy size quotas are now enforced, and policies with Version 2008-10-17 are accepted
• New UpdateRoleDescription and ListAccountAliases APIs
• A pathed IAM user's caller ARN now includes its path
• Policy and instance-profile ARNs now resolve correctly when the user, role or policy has a path
• ListAttached*Policies now fails with an error instead of silently omitting a policy it cannot read
• ListInstanceProfiles now skips roles that were deleted instead of failing
• DeleteInstanceProfile is now safe against concurrent modification
• IAM and STS request validation (including DurationSeconds and MFA parameters) now returns AWS's error codes and messages

Networking

• Network Describe calls now support MaxResults/NextToken paging and tag-key/tag-value filters
• DescribeTags now applies tag:<key> filters and lists tags added when a key pair is created
• Repeating an identical CreateRoute now succeeds, as it does on AWS
• Security group rules with no protocol, missing TCP/UDP ports or inverted port ranges are now rejected
• All-protocol security group rules no longer report ports, and they match an existing rule regardless of the ports given
• CreateSubnet and DescribeSubnets now return SubnetArn and AWS's default-subnet fields
• CreateInternetGateway and DescribeInternetGateways now return OwnerId
• DescribeAccountAttributes now reports the account's default VPC
• Invalid network requests now fail with AWS's error codes and messages, covering bad CIDRs, missing gateways and routes, duplicates, dependencies, malformed IDs, sg- group names, aws: tag keys, and Modify*Attribute calls that set more than one attribute

Compute

• DescribeInstanceTypes now supports paging and filters, and rejects unknown instance type names
• DescribeRegions and DescribeAvailabilityZones now validate and apply filters and names
• DescribeKeyPairs now returns PublicKey when IncludePublicKey is set
• CreateKeyPair now rejects KeyFormat=ppk instead of returning a PEM key
• Placement groups now return GroupArn
• CopyImage now copies the distro metadata from the source AMI
• DeleteLaunchTemplateVersions now refuses to delete the default version and reports missing versions as AWS does
• ModifyInstanceAttribute now returns InvalidParameterCombination when given no attribute or several attributes
• CreateVolume errors now list the accepted volume types
• MaxResults ranges, UnauthorizedOperation responses and tag resource types now match AWS

Certificates (ACM)

• DescribeCertificate now returns the same fields and formats as AWS for imported certificates
• RSA key algorithms are now reported as RSA-<bits>
• GetCertificate now returns the certificate itself as the chain when no chain was imported
• Re-importing a certificate with Tags is now rejected, and the certificate's existing tags are kept
• RequestCertificate now returns ValidationException for a malformed domain

Containers (ECR)

• New DescribeRegistry API, which returns the caller's registry
• ECR responses and error messages now follow AWS's format
• ECR validation failures now return InvalidParameterException, and tag errors return InvalidTagParameterException

Databases (RDS)

• New ModifyDBSubnetGroup API
• Omitting DBParameterGroupFamily is now rejected, and an unknown family returns AWS's error message
• Describe filter and DbiResourceId errors now return AWS's codes and messages

Security

• S3 batch deletes now authorize each key individually
• S3 now rejects attached policy ARNs that do not match the stored ARN
• S3 credential caches now refresh as soon as a user, policy or group changes
• Peer and storage health checks now verify certificates against the cluster CA

Reliability

• Volumes now survive storage-layer outages of up to 30 seconds, so a short host I/O stall no longer takes a guest's disk away
• S3 placement read errors now return InternalError instead of a misleading NoSuchKey
• spx image imports now extract to disk-backed scratch space instead of memory-backed temp storage

What's Changed

GPU Compute

• Multi-GPU instance types, with up to 8 GPUs passed through to a single instance
• Multi-GPU sizes such as g5.48xlarge and p4d.24xlarge get the same GPU count as on AWS, up to 8
• New gpu.<count>x<vcpu>c family for edge hosts too small for AWS's multi-GPU sizes, such as gpu.8x4c with eight GPUs on 32 vCPU
• gpu.* types come with 1 to 8 GPUs at 2, 4 or 8 vCPU per GPU, with 4 GiB of memory per vCPU
• DescribeInstanceTypes reports the GPU model and VRAM each host actually has
• Hosts with mixed GPU models offer single-GPU types only, so an instance never gets mismatched cards

Compute (EC2)

• RunInstances creates the extra EBS data volumes named in the block device mapping, so Terraform ebs_block_device works
• Instances report the block device names you asked for, so a no-op plan no longer destroys and recreates the instance
• A requested private IP address is validated and honoured at launch
• DryRun returns DryRunOperation for every EC2 action that supports it
• Tags applied at creation show up in DescribeTags and are cleared when the resource is deleted
• Concurrent CreateTags calls on one resource no longer lose tags
• DescribeInstanceAttribute omits the user data value when an instance has none, clearing a permanent user_data diff
• DescribeImages reports CreationDate in UTC, so most_recent picks the newest AMI whatever the node's timezone
• Local image imports can be verified with --checksum, and AMIs record their source digest
• Single-file .zst, .bz2, .xz and .gz disk images are decompressed on import
• The console shows how to SSH into an instance and which user to log in as

Reliability

• Instances on a node that stops heartbeating are automatically relaunched on a surviving node
• Cluster state is replicated across every node, so losing one node no longer causes a cluster-wide outage
• A brief control-plane stall no longer cuts running guests off from their volumes
• The first launch from an AMI after a cluster restart no longer times out and terminates the instance
• Instances whose recovery failed are retried, and a dead instance is no longer reported as running
• Instances in an error state can now be terminated, and a refused termination is no longer reported as success
• A stuck DNS reconcile can no longer wedge the cluster
• Dropped connections no longer count toward the authentication lockout

Networking

• Auto-assigned public IPs are released on stop and a new one is assigned on start, as in AWS
• AWS service endpoint names such as ec2.<region>.<domain> now resolve in cluster DNS
• Gateway TLS certificates cover the published AWS service endpoint names
• New ModifySecurityGroupRules
• Security group rules accept IPv6 ranges, so the default ::/0 egress no longer leaves a group with no egress at all
• Security group rule tags persist, clearing a permanent diff
• DescribeRegions returns endpoints on the gateway's advertised host

Identity (IAM / STS / IMDS)

• New CreatePolicyVersion, SetDefaultPolicyVersion and DeletePolicyVersion
• Changing a policy's default version immediately changes what every attached user and role can do
• Editing an aws_iam_policy document in Terraform now updates the policy in place
• A policy holds up to five versions, and one more returns LimitExceeded, as in AWS
• IAM list actions honour MaxItems and Marker
• ListPolicies honours PathPrefix, Scope and OnlyAttached
• IAM role, policy, user and account summary responses return the fields AWS returns for each action
• IAM policy documents are URL-encoded as AWS returns them, clearing drift on documents containing + or %
• Trust policies evaluate conditions, including aws:SourceAccount
• The aws:PrincipalType condition key is supported for EC2 and S3 requests
• STS rejects session policies and tags as unsupported instead of silently ignoring them
• OIDC providers return their URL with the scheme they were created with, so Terraform no longer replaces the provider on every plan
• IMDS returns the cluster's services domain, so an SDK missing its endpoint override fails closed instead of reaching real AWS
• IMDS serves block-device-mapping, so the aws-ebs-csi-driver starts

Kubernetes (EKS)

• AssociateAccessPolicy now grants the matching Kubernetes RBAC in the cluster
• The API_AND_CONFIG_MAP authentication mode is accepted

Containers (ECS)

• DescribeServices returns service events and deployment timestamps
• DescribeServices returns availabilityZoneRebalancing, clearing a permanent diff from every plan
• ListTasks and ListContainerInstances honour their filters
• awsvpc ENI capacity is used in task placement, so tasks aren't placed on instances with no ENIs left
• Stopped awsvpc tasks release their ENIs reliably, so DeleteSecurityGroup no longer fails on a leaked ENI
• A deployment that cannot start is halted instead of retrying indefinitely
• Unsupported container runtime fields are refused instead of silently dropped

Storage (S3)

• New S3 bucket versioning
• Content-Type and x-amz-meta-* user metadata are stored and returned
• Object keys with special characters are stored and listed by their decoded name
• Tags supplied at bucket creation are applied
• S3 verifies SigV4 requests that sign transfer-encoding
• Unsupported bucket and object sub-resources are refused instead of being misrouted

Databases (RDS)

• Every RDS Describe* call paginates with MaxRecords and Marker
• DB subnet groups report SupportedNetworkTypes and SubnetOutpost

Installation & Deployment

• Spinifex can now be deployed on Oracle Cloud Infrastructure (OCI), in beta

Errors & Diagnostics

• Validation errors name the failing parameter
• Invalid IAM names, paths and enum values return ValidationError
• A request for a service Spinifex doesn't serve returns InvalidAction instead of blaming your credentials

What's Changed

Compute (EC2)

• DescribeInstances reports the public IP association on the primary network interface, so a no-op plan no longer proposes destroying and recreating a working instance
• Instances return EnclaveOptions, clearing a permanent diff from every plan
• A launch spread across several nodes comes back as one reservation, not several
• An incomplete DescribeInstances sweep returns a retryable ServiceUnavailable instead of an empty result that reads as authoritative
• DescribeVpcs accepts the AWS filter spellings cidr and isDefault, so the aws_vpc data source works
• describe-images --owners self resolves owner aliases before filtering, instead of matching nothing
• New DescribeInstanceTypeOfferings and GetSecurityGroupsForVpc
• New UpdateSecurityGroupRuleDescriptionsIngress and UpdateSecurityGroupRuleDescriptionsEgress
• RebootInstances asks the guest to shut down cleanly first, so a write made seconds before a reboot survives it

Kubernetes (EKS)

• DescribeCluster returns the service CIDR, cluster security group, logging and authentication mode, so the terraform-aws-modules/eks module applies clean and re-plans with no changes
• resourcesVpcConfig echoes your own security groups instead of overwriting them with platform groups you cannot see
• UpdateNodegroupConfig applies UpdateConfig and taints instead of reporting success and ignoring them

Containers (ECR / ECS)

• ECR CreateRepository stores and returns encryption, image scanning configuration and tags
• New ECR TagResource and UntagResource
• ECR rejects encryptionType: KMS rather than echoing a customer-managed key that nothing uses
• ECS DescribeTaskDefinition, DescribeServices and DescribeClusters return the fields you set at create time, plus cluster instance, task and service counts
• ECS UpdateService honours networkConfiguration and enableECSManagedTags, so a changed value converges

Identity (IAM / STS)

• IAM policy tags are persisted and projected by CreatePolicy, GetPolicy and ListPolicies
• Policy attachment counts include roles and groups, so an attached policy can no longer be deleted out from under a principal
• New ListEntitiesForPolicy, honouring EntityFilter and PathPrefix
• New sts:GetAccessKeyInfo
• STS session credentials are bound to immutable principal IDs

Load Balancing (ELBv2) & Certificates (ACM)

• Concurrent RegisterTargets no longer lose members, so a parallel Terraform apply puts the whole fleet behind the load balancer
• New DescribeAccountLimits
• RequestCertificate fails fast with a terminal error naming the missing tenant CA and how to create one, instead of retrying for minutes
• ListCertificates returns an empty list on an empty account rather than omitting the field

Networking

• An account gets exactly one set of default VPC resources, even when several nodes build them at once
• Releasing a management IP invalidates the host ARP entry, so the address is reusable immediately
• A stopped instance's Elastic IP datapath is left alone rather than probed and nudged

Storage (S3)

• A ranged GET reads only the stripes the range touches, so a small range on a large object costs a small read
• Shard repair waits for the cluster to be ready and reads placement through the metadata leader, so a restart no longer rebuilds shards that were never owed
• New standalone deployment path with systemd integration and a packageable make install

Reliability

• A StopInstances or TerminateInstances already accepted survives a daemon restart that finds the guest still running
• A guest drain-stopped by a cluster shutdown retries its relaunch instead of being recorded as permanently failed
• Multi-node formation discards stale JetStream data by default, so replicas cannot diverge permanently
• New spx admin kv compare verifies every node's data agrees after formation
• A node reset now stops guests and services before wiping state, and clears the JetStream store completely

Errors & Diagnostics

• A daemon's own refusal message reaches the client instead of a generic blurb for the error code, with internal context stripped
• Rejected filters name the offending filter
• REST-XML and JSON clients receive errors in the envelope their protocol expects, with X-Amzn-Errortype set for JSON SDKs
• A request for an unserved service says which service was refused instead of blaming your credentials
• Per-action throttling limits now apply to JSON-protocol requests

What's Changed

⚠️ Breaking Change

• Predastore's on-disk format changed, so a store written by v1.18.0 or earlier is rejected on read and no migration is provided
• EBS volume and snapshot metadata is now partitioned by owning account, and metadata written by an earlier release is not readable under the new layout
• Re-initialise affected clusters with spx admin init --force, which discards stored objects, export anything you need first
• This is the last breaking change: every future format or layout change ships with a migration

Storage (S3 & EBS)

• CopyObject and UploadPartCopy are implemented, so server-side and multipart copies work
• Batch DeleteObjects removes many keys in a single call
• ListObjectsV2 paginates correctly, and v1 listings honour the marker parameter
• Object ETags are derived from content, so client-side integrity checks match
• aws-chunked upload bodies are decoded and authenticated, unblocking streaming SigV4 uploads
• Unsupported S3 sub-resources are refused instead of being silently ignored
• EBS metadata is partitioned by account, cutting describe latency on clusters with many tenants
• S3 Service Metrics derive cluster health from raft quorum

Identity (IAM / STS / IMDS)

• sts:AssumeRole is gated on the caller's identity policy
• IAM policy matching supports the ? single-character wildcard
• A policy variable that cannot be resolved fails closed instead of matching
• aws:userid works as both a condition key and a policy variable across the EC2, S3 and gateway paths
• PassRole is authorized against the stored role ARN
• Caller-supplied role and policy ARNs are verified against the stored canonical ARN
• A policy ARN supplied by a caller is re-anchored onto the caller's own account
• PermissionsBoundary is rejected on CreateRole and CreateUser rather than silently dropped
• An IpAddress condition that cannot parse the request address fails closed
• IMDS credentials resolve locally, cutting a network hop off every instance credential fetch
• A credential refresh fails outright instead of extending a dead key's expiry

Compute

• Instances on a node that has stopped answering now report impaired status
• DescribeInstances judges completeness by responder identity, so a partial fan-out no longer looks like a full answer
• Seven EC2 create calls honour ClientToken, so a retry no longer duplicates the resource
• RunTask and CreateLaunchTemplateVersion honour ClientToken
• InsufficientInstanceCapacity is returned with a 5xx status, so SDK retries kick in
• DescribeImages reports a conformant BootMode

Networking

• Outbound SMTP to ports 25, 465 and 587 is blocked for public destinations by default, matching AWS, with an operator-held per-VPC exemption
• Unsupported security group protocols are rejected instead of emitting an allow-all rule
• Security group default-deny rules now match every ethertype
• Unscoped firewall accepts are family-qualified to IPv4
• An internet gateway reports itself attached only once the underlying gateway is up
• ENIs and their Elastic IP associations are released when the owning instance is gone
• A launch that fails part-way releases its ENI and external address
• DeleteSecurityGroup names the blocking resource in its DependencyViolation
• DNS converges on change instead of on a timer, so records appear promptly after a lifecycle event
• Stale EC2 DNS records are pruned against the live instance set
• A node's resolver no longer answers recursive queries for arbitrary clients
• Cluster-wide IPsec is enabled only once every chassis is configured

Kubernetes (EKS)

• Control-plane egress is tunnelled through konnectivity
• Internal control-plane routes bind to the calling control-plane VM
• The cluster reconciler wakes on signals it already receives, so clusters converge faster
• A transient reconcile failure retries immediately rather than waiting for the next resync
• Control-plane health diagnostics report CPU and memory pressure alongside etcd disk state, and an unanswered probe reads as unreachable rather than healthy

Load Balancing (ELBv2)

• ModifyLoadBalancerAttributes and ModifyTargetGroupAttributes accept the full documented attribute set, so Terraform and the AWS Load Balancer Controller no longer fail an apply on one missing key

Reliability

• A guest is paused on a backend I/O error instead of being handed EIO, turning a transient storage fault into a recoverable stall
• A write-ahead log flush is reported durable only once the fsync covering it completes
• A node upgrade refuses to proceed when a volume seal fails, instead of reporting success and leaving an unbootable root volume
• A volume falls back to its local state copy when the backend copy is torn
• A failed shard write can no longer wedge an object put
• Storage readiness is gated on the write floor, so a node takes traffic only when it can serve writes
• Caller cancellations no longer evict healthy connections from the pool
• The message store keeps recovery headroom, so a full disk no longer takes it down on restart
• Console assets are content-hashed, so a deploy cannot serve a mix of two builds

Performance

• Boot volumes spread virtio-blk queues across an IOThread pool sized to the instance's vCPU count
• Volume write-ahead logs can be placed on a dedicated device via wal_base_dir, keeping fsync tail latency off the shared device queue
• Streaming shard read and write path with hedged repair
• Object shards spread evenly across nodes, so losing one node no longer degrades the whole keyspace
• NBD request latency and guest write backpressure are measured at the guest boundary, so slow-disk complaints are now attributable

Security

• Session tokens are redacted from canonical requests written to logs
• Object listing no longer logs the raw query string
• A streamed payload is verified at its declared length, not only at EOF
• Inbound object bodies are bounded, and the CreateBucket body payload check is enforced