Page MenuHomePhabricator

Make Teamcity agent setup easier
Changes PlannedPublic

Authored by Fabien on Jun 4 2019, 18:03.

Details

Reviewers
deadalnix
jasonbcox
Group Reviewers
Restricted Project
Summary

This diff adds a setup-agent.sh script that turns a fresh Debian 9
server into a functional Teamcity agent.

It updates the gitian.sh script to allow for single shot gitian
installation, when the agent is setup with the setup-agent.sh script.

Test Plan

Install an agent, run builds, gitian builds and ibd.

After installation, check that root login is disabled and teamcity password login as well.

Check the help:

./setup-agent.sh
./setup-agent.sh -h
./setup-agent.sh --help
./setup-agent.sh this call has way too much arguments

Diff Detail

Repository
rABC Bitcoin ABC
Branch
teamcity_setup_with_keys
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 6901
Build 11849: Bitcoin ABC Buildbot (legacy)
Build 11848: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
deadalnix requested changes to this revision.Jun 5 2019, 12:07
deadalnix added inline comments.
contrib/teamcity/setup-agent.sh
136 ↗(On Diff #9147)

Always verify intergrity of what you get from the internet.

177 ↗(On Diff #9147)
sudo -u teamcity command
195 ↗(On Diff #9147)

Having a machine setup with a user with a default known password that is also a sudoer is probably not the best security practice.

The machine can be presetup with pubkeys for login.

This revision now requires changes to proceed.Jun 5 2019, 12:07

Address feedback + minor improvements.
No SSH setup yet.

Fabien planned changes to this revision.Jun 10 2019, 14:37

The script now runs from the user machine, and connect to the
remote using ssh.
At the end, root login as well as password login are disabled,
and only the teamcity user is availble for login using key
authentication.

Fabien planned changes to this revision.Jul 7 2019, 20:21
Fabien edited the test plan for this revision. (Show Details)
Fabien planned changes to this revision.Jul 7 2019, 20:29

Fix argument count and check for remote to be set.

jasonbcox requested changes to this revision.Jul 8 2019, 18:45
jasonbcox added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

This should use the newly authorized PUBKEY instead of the PASSWORD to test that the pubkey authorization is working as expected. Should the pubkey authorization fail for any reason, the following steps will lock you out of the machine.

This revision now requires changes to proceed.Jul 8 2019, 18:45
Fabien requested review of this revision.Jul 8 2019, 18:58
Fabien added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

This is what the previous line is about. It ensures that you can connect using the key as the teamcity user.
Intended that we disabling root login just after, this doesn't really make any difference to log with password or file at this point.

jasonbcox added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

I guess that's fair. The two lines could have been combined to test and login with the pubkey, but then you wouldn't have the nice error message.

Pick libevent from buster.
Add mirror selection.
Use apt-get instead of apt to get rid of the immature cli interface warning.

Don't fail on host key verification when ssh'ing the first time.
Print help before warning for sshpass.

Did a few test runs setting up new VMs and got a passing build with the gitian build on this diff.

jasonbcox requested changes to this revision.Jul 11 2019, 19:05

Came across these in another round of review.

contrib/teamcity/setup-agent.sh
52 ↗(On Diff #10186)

PUBKEY should have quotes

123 ↗(On Diff #10186)

Personal preference, but some useful tools could be installed for debugging and maintenance:
vim
less
bash-completion

163 ↗(On Diff #10186)

git is listed twice. it should probably be moved under # Server management anyway since it's practically ubiquitous.

This revision now requires changes to proceed.Jul 11 2019, 19:05
deadalnix requested changes to this revision.Jul 22 2019, 18:36

My understanding is that this is an agent that will run the build raw, as this. This is a security disaster pending, in addition to having very little flexibility. If some package changes, it's all the agents that need to be reconstructed. In addition, the agent seems to have various credential for various things, and we'll run arbitrary code on it. Even with permission, this is a tremendously bad idea.

contrib/teamcity/setup-agent.sh
202

These stable/testing do not correspond to debian stable and testing. Plus you probably don't want to depend on testing anyways, because it's a rolling release.

314

Wouln't it be preferable that the agent had no password and be connected using some pubkey ?

This revision now requires changes to proceed.Jul 22 2019, 18:36
Fabien planned changes to this revision.Jul 22 2019, 19:55

I will separate the gitian script update and rework the CI setup according to feedback.

contrib/teamcity/setup-agent.sh
134

I know this is still WIP, but don't forget to add clang to this list.

contrib/teamcity/setup-agent.sh
135

Note to self: add cmake
Note to self: don't install the headless version as cmake's FindJNI also require AWT