Fossil SCM and Concourse v2
Three days ago, I wrote a post about my workflow with Fossil and Concourse, which involved polling the RSS feed and pulling the repo manually with the CLI: it worked, but it was not the best way to do it. Concourse allows you to easily implement “resources”, so you can conveniently separate the polling, pushing and pulling from your pipelines. So, I thought on creating one.
A Concourse resource consists of three scripts, check
, in
and out
: check
listens for changes on something, in
pulls something using that something, and out
pushes a something somewhere. A change will make check
trigger in
, whose output will be passed to a task. The output of that task will put some output in some folder, so that out
can push its contents somewhere or do something with it. I think I’m really good with explanations!
Fossil, in addition to the built-in RSS feed, also provides a JSON API, which I found very convenient for this purpose. So, I started writing the scripts in CHICKEN Scheme. Why CHICKEN? Because I think it’s cool, and also because it can run as a script; but mainly because I wanted to practice. Functional programming is fun! I think I’m getting better at it.
When I tested the script, I discovered that the API wasn’t working, so I reported the issue on the Fossil forum, and almost immediately got a response from the person who wrote the API. After giving him some extra information and logs, he was able to find the problem and fixed it almost immediately. I compiled the new version, and everything was working, except for POST data handling, so I reported the issue, and it was fixed quickly as well.
Once the API was working, I continued writing and testing my scripts, but my limited Scheme skills made it harder than it should. It was a lot of trial and error, until I eventually managed to make it work (mostly). Now, it was time to (actually) test with Concourse, so I pushed the Docker image to Docker Hub, and wrote a very simple pipeline. I still had to fix some bugs with IO, so I did. I finished some hours ago, and wrote a simple pipeline to automatically build and publish the image of the resource, using the resource itself.
The resource can be found here, it is free software, under GPLv3 license. I think it still needs some improvements, but feel free to use it for any project! I really hope more people get into the Fossil ecosystem with tools like this.
My (new) automated workflow
Warning: this may (probably) not be the best and most secure configuration, because I’m still kind of a newbie in devops. I would appreciate (more) feedback and suggestions!
I’m running the Fossil server, reverse-proxied with NGINX, secured with Certbot and kept up by Supervisor. The command I use to run the server is the following:
fossil server /var/www/fossil/ --https --port 8079
Then, I have the same Concourse instance running inside a Docker container (reversed-proxied as well). The YAML pipeline for this blog is deployed with fly
. This time, instead of just throwing up a long YAML with ugly ellipses (for brevity), I will explain the full pipeline in parts. Here’s the full pipeline.yml
that is explained below.
The following part contains the resource_type
definition and the resource
based on that definition. The “official” Docker image of the resource is avalos/fossil-concourse-resource
, which Concourse will use to perform the check
and in
steps. The config params are passed to the resource in source
, in this case, the url
where the Fossil repo is located, and the branch
we want to poll and pull from.
resource_types:
- name: fossil-resource
type: docker-image
source:
repository: avalos/fossil-concourse-resource
tag: latest
resources:
- name: fossil
type: fossil-resource
source:
url: https://fossil.avalos.me/avalos-indie-blog
branch: trunk
Here, deploy-to-server
is the only job. The job will be triggered every time fossil
detects a new commit in trunk, as defined in the get
step. In the build
step, the output of fossil
will be mounted to the working directory (.
) of the container, as defined in inputs
. In this case, the directory will contain the files of this blog, so compile the files, and the public
folder automatically generated by hugo
will be the output of the task, as defined in outputs
.
jobs:
- name: deploy-to-server
serial: true
plan:
- get: fossil
trigger: true
- task: build
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine, tag: edge}
inputs:
- name: fossil
path: .
outputs:
- name: blog
path: public
run:
path: /bin/sh
args:
- -c
- apk update && apk add hugo && hugo
And finally, we have the deploy
task, which will obviously deploy this blog. The blog
output will be used as the input for this task, it will be mounted in the working directory. Because this blog, along with all my other websites and services, is hosted in a AWS EC2 instance, I can use EC2 Instance Connect to securely SSH into it and push the files. Instead of insecurely using a single SSH key as a secret all the time, I generate a new temporary SSH key pair on each deployment, and push the public key to my instance over EC2 Instance Connect: it will expire after 60 seconds, so there’s enough time. Once I have access, I proceed to rsync
into the public directory. That’s it!
Notice how the variables are enclosed with (())
. Those variables in specific are sensitive, so I load them from a separate file instead of hardcoding them into the pipeline itself.
Note: AWS EC2 Instance Connect needs to be configured on the instance in order for this to work, here’s a nice post on how to do it.
- task: deploy
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine, tag: edge}
inputs:
- name: blog
params:
AWS_ACCESS_KEY_ID: ((AWS_ACCESS_KEY_ID))
AWS_SECRET_ACCESS_KEY: ((AWS_SECRET_ACCESS_KEY))
AWS_DEFAULT_REGION: ((AWS_DEFAULT_REGION))
AWS_REGION: ((AWS_REGION))
AWS_INSTANCE_ID: ((AWS_INSTANCE_ID))
AWS_AVAILABILITY_ZONE: ((AWS_AVAILABILITY_ZONE))
AWS_INSTANCE_OS_USER: ((AWS_INSTANCE_OS_USER))
run:
path: /bin/sh
args:
- -c
- |
apk update &&
apk add rsync openssh-client aws-cli &&
ssh-keygen -t rsa -f aws_key -q -N "" &&
chmod 600 aws_key &&
aws ec2-instance-connect send-ssh-public-key \
--region $AWS_REGION \
--instance-id $AWS_INSTANCE_ID \
--availability-zone $AWS_AVAILABILITY_ZONE \
--instance-os-user $AWS_INSTANCE_OS_USER \
--ssh-public-key file://aws_key.pub &&
rsync --progress -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i aws_key" \
-arvz --delete ./blog/ ec2-user@avalos.me:/var/www/blog.avalos.me/public_html/
So, it’s time to deploy the pipeline.yml
to Concourse! We authenticate with fly login
and then, run the following command:
fly set-pipeline \
-t ci.avalos.me \
-p blog.avalos.me \
-c pipeline.yml \
-l vars.yml
The variables will be loaded from vars.yml
, written as YAML key: value
pairs. If encryption is enabled, Concourse will encrypt all the variables and pipeline definitions, and only decrypt them for a short time when needed, so, with a solid and secure setup (you don’t need to trust), there’s no need to worry. In this case, I can easily revoke the (restricted) IAM user if the access keys are stolen.