Page MenuHomePhabricator
Differential D3197

Make Teamcity agent setup easier
Changes PlannedPublic
Actions

Authored by Fabien on Jun 4 2019, 18:03.

Details

Reviewers
deadalnix
jasonbcox
Group Reviewers
Restricted Project
Summary

This diff adds a setup-agent.sh script that turns a fresh Debian 9
server into a functional Teamcity agent.

It updates the gitian.sh script to allow for single shot gitian
installation, when the agent is setup with the setup-agent.sh script.

Test Plan

Install an agent, run builds, gitian builds and ibd.

After installation, check that root login is disabled and teamcity password login as well.

Check the help:

./setup-agent.sh
./setup-agent.sh -h
./setup-agent.sh --help
./setup-agent.sh this call has way too much arguments

Diff Detail

Repository
rABC Bitcoin ABC
Branch
teamcity_setup
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 6780
Build 11607: Bitcoin ABC Buildbot (legacy)
Build 11606: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
deadalnix requested changes to this revision.Jun 5 2019, 12:07
deadalnix added inline comments.
contrib/teamcity/setup-agent.sh
136 ↗(On Diff #9147)

Always verify intergrity of what you get from the internet.

177 ↗(On Diff #9147)
sudo -u teamcity command
195 ↗(On Diff #9147)

Having a machine setup with a user with a default known password that is also a sudoer is probably not the best security practice.

The machine can be presetup with pubkeys for login.

This revision now requires changes to proceed.Jun 5 2019, 12:07
Fabien updated this revision to Diff 9291.Jun 10 2019, 14:37

Address feedback + minor improvements.
No SSH setup yet.

teamcity added a comment.Jun 10 2019, 14:37

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Fabien planned changes to this revision.Jun 10 2019, 14:37
Harbormaster completed remote builds in B6249: Diff 9291.Jun 10 2019, 14:47
teamcity added a comment.Jun 10 2019, 14:47

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

teamcity added a comment.Jun 11 2019, 18:14

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jun 11 2019, 18:15

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing failed.

teamcity added a comment.Jun 11 2019, 19:01

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jun 11 2019, 19:01

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing failed.

teamcity added a comment.Jun 11 2019, 19:02

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jun 11 2019, 19:16

Build Bitcoin-ABC / Master / Nightly Builds / Bitcoin-ABC Gitian Diff Testing passed.

Fabien updated this revision to Diff 10056.Jul 7 2019, 20:17

The script now runs from the user machine, and connect to the
remote using ssh.
At the end, root login as well as password login are disabled,
and only the teamcity user is availble for login using key
authentication.

teamcity added a comment.Jul 7 2019, 20:17

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Fabien planned changes to this revision.Jul 7 2019, 20:21
Fabien edited the test plan for this revision. (Show Details)
Harbormaster completed remote builds in B6711: Diff 10056.Jul 7 2019, 20:27
teamcity added a comment.Jul 7 2019, 20:27

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Fabien updated this revision to Diff 10059.Jul 7 2019, 20:28

Fix help.

teamcity added a comment.Jul 7 2019, 20:28

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Fabien planned changes to this revision.Jul 7 2019, 20:29
Fabien edited the test plan for this revision. (Show Details)Jul 7 2019, 20:33
teamcity added a comment.Jul 7 2019, 20:34

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Harbormaster completed remote builds in B6714: Diff 10059.Jul 7 2019, 20:34
Fabien updated this revision to Diff 10060.Jul 7 2019, 20:38

Fix argument count and check for remote to be set.

teamcity added a comment.Jul 7 2019, 20:39

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Harbormaster completed remote builds in B6715: Diff 10060.Jul 7 2019, 20:45
teamcity added a comment.Jul 7 2019, 20:45

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

jasonbcox requested changes to this revision.Jul 8 2019, 18:45
jasonbcox added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

This should use the newly authorized PUBKEY instead of the PASSWORD to test that the pubkey authorization is working as expected. Should the pubkey authorization fail for any reason, the following steps will lock you out of the machine.

This revision now requires changes to proceed.Jul 8 2019, 18:45
Fabien requested review of this revision.Jul 8 2019, 18:58
Fabien added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

This is what the previous line is about. It ensures that you can connect using the key as the teamcity user.
Intended that we disabling root login just after, this doesn't really make any difference to log with password or file at this point.

jasonbcox accepted this revision.Jul 8 2019, 19:02
jasonbcox added inline comments.
contrib/teamcity/setup-agent.sh
268 ↗(On Diff #10060)

I guess that's fair. The two lines could have been combined to test and login with the pubkey, but then you wouldn't have the nice error message.

Fabien updated this revision to Diff 10125.Jul 9 2019, 08:45

Pick libevent from buster.
Add mirror selection.
Use apt-get instead of apt to get rid of the immature cli interface warning.

teamcity added a comment.Jul 9 2019, 08:45

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

teamcity added a comment.Jul 9 2019, 08:51

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Harbormaster completed remote builds in B6753: Diff 10125.Jul 9 2019, 08:51
teamcity added a comment.Jul 9 2019, 20:23

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jul 9 2019, 20:23

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing failed.

teamcity added a comment.Jul 9 2019, 22:39

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jul 9 2019, 22:39

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing failed.

teamcity added a comment.Jul 9 2019, 23:23

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jul 9 2019, 23:24

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing failed.

Fabien updated this revision to Diff 10167.Jul 9 2019, 23:52

Update COMMIT setting

teamcity added a comment.Jul 9 2019, 23:52

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

teamcity added a comment.Jul 9 2019, 23:54

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

Harbormaster completed remote builds in B6780: Diff 10167.Jul 10 2019, 00:01
teamcity added a comment.Jul 10 2019, 00:01

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

teamcity added a comment.Jul 10 2019, 00:06

Build Bitcoin-ABC / Master / Gitian Builds / Bitcoin-ABC Gitian Diff Testing passed.

Fabien updated this revision to Diff 10186.Jul 10 2019, 06:23

Don't fail on host key verification when ssh'ing the first time.
Print help before warning for sshpass.

teamcity added a comment.Jul 10 2019, 06:23

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Harbormaster completed remote builds in B6791: Diff 10186.Jul 10 2019, 06:32
teamcity added a comment.Jul 10 2019, 06:32

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

jasonbcox accepted this revision.Jul 10 2019, 15:44

Did a few test runs setting up new VMs and got a passing build with the gitian build on this diff.

jasonbcox requested changes to this revision.Jul 11 2019, 19:05

Came across these in another round of review.

contrib/teamcity/setup-agent.sh
52 ↗(On Diff #10186)

PUBKEY should have quotes

123 ↗(On Diff #10186)

Personal preference, but some useful tools could be installed for debugging and maintenance:
vim
less
bash-completion

163 ↗(On Diff #10186)

git is listed twice. it should probably be moved under # Server management anyway since it's practically ubiquitous.

This revision now requires changes to proceed.Jul 11 2019, 19:05
teamcity added a comment.Jul 11 2019, 21:55

Build Bitcoin-ABC / Resource Intensive Builds / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jul 11 2019, 21:55

Build Bitcoin-ABC / Resource Intensive Builds / Gitian Builds / Bitcoin-ABC Gitian Diff Testing failed.

teamcity added a comment.Jul 11 2019, 21:58

Build Bitcoin-ABC / Resource Intensive Builds / Gitian Builds / Bitcoin-ABC Gitian Diff Testing started.

teamcity added a comment.Jul 11 2019, 22:41

Build Bitcoin-ABC / Resource Intensive Builds / Gitian Builds / Bitcoin-ABC Gitian Diff Testing passed.

Fabien updated this revision to Diff 10372.Jul 22 2019, 16:13

Address feedbacks.

teamcity added a comment.Jul 22 2019, 16:13

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

jasonbcox accepted this revision.Jul 22 2019, 16:17
teamcity added a comment.Jul 22 2019, 16:24

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Harbormaster completed remote builds in B6901: Diff 10372.Jul 22 2019, 16:25
deadalnix requested changes to this revision.Jul 22 2019, 18:36

My understanding is that this is an agent that will run the build raw, as this. This is a security disaster pending, in addition to having very little flexibility. If some package changes, it's all the agents that need to be reconstructed. In addition, the agent seems to have various credential for various things, and we'll run arbitrary code on it. Even with permission, this is a tremendously bad idea.

contrib/teamcity/setup-agent.sh
202 ↗(On Diff #10372)

These stable/testing do not correspond to debian stable and testing. Plus you probably don't want to depend on testing anyways, because it's a rolling release.

314 ↗(On Diff #10372)

Wouln't it be preferable that the agent had no password and be connected using some pubkey ?

This revision now requires changes to proceed.Jul 22 2019, 18:36
Fabien planned changes to this revision.Jul 22 2019, 19:55

I will separate the gitian script update and rework the CI setup according to feedback.

jasonbcox added inline comments.Aug 5 2019, 15:44
contrib/teamcity/setup-agent.sh
134 ↗(On Diff #10372)

I know this is still WIP, but don't forget to add clang to this list.

Fabien added inline comments.Aug 14 2019, 09:30
contrib/teamcity/setup-agent.sh
135 ↗(On Diff #10372)

Note to self: add cmake
Note to self: don't install the headless version as cmake's FindJNI also require AWT

Revision Contents
Changeset List

PathSize
contrib/
teamcity/
18 lines
205 lines

Diff 10167

View Options

contrib/teamcity/gitian.sh

Loading...
View Options

contrib/teamcity/setup-agent.sh

Loading...