diff --git a/doc/README_osx.md b/doc/README_osx.md index 2a4460478c..975be4be9e 100644 --- a/doc/README_osx.md +++ b/doc/README_osx.md @@ -1,97 +1,97 @@ -Deterministic OS X Dmg Notes. +Deterministic OS X DMG Notes. Working OS X DMGs are created in Linux by combining a recent clang, the Apple binutils (ld, ar, etc) and DMG authoring tools. Apple uses clang extensively for development and has upstreamed the necessary functionality so that a vanilla clang can take advantage. It supports the use of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary when building for OS X. Apple's version of binutils (called cctools) contains lots of functionality missing in the FSF's binutils. In addition to extra linker options for frameworks and sysroots, several other tools are needed as well such as install_name_tool, lipo, and nmedit. These do not build under linux, so they have been patched to do so. The work here was used as a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). In order to build a working toolchain, the following source packages are needed from Apple: cctools, dyld, and ld64. These tools inject timestamps by default, which produce non-deterministic binaries. The ZERO_AR_DATE environment variable is used to disable that. This version of cctools has been patched to use the current version of clang's headers and its libLTO.so rather than those from llvmgcc, as it was originally done in toolchain4. To complicate things further, all builds must target an Apple SDK. These SDKs are free to download, but not redistributable. To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). This file is several gigabytes in size, but only a single directory inside is needed: ``` Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk ``` Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. To create a tarball suitable for Gitian input, there are two options: Using Mac OS X, you can mount the dmg, and then create it with: ``` $ hdiutil attach Xcode_7.3.1.dmg $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk ``` Alternatively, you can use 7zip and SleuthKit to extract the files one by one. The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure the dmg file is in the current directory, and then run the script. You may wish to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when you've confirmed the extraction succeeded. ```bash apt-get install p7zip-full sleuthkit contrib/macdeploy/extract-osx-sdk.sh rm -rf 5.hfs MacOSX10.11.sdk ``` The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries which are created using these tools. The build process has been designed to avoid including the SDK's files in Gitian's outputs. All interim tarballs are fully deterministic and may be freely redistributed. genisoimage is used to create the initial DMG. It is not deterministic as-is, so it has been patched. A system genisoimage will work fine, but it will not be deterministic because the file-order will change between invocations. The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). No effort was made to fix this cleanly, so it likely leaks memory badly. But it's only used for a single invocation, so that's no real concern. genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the libdmg-hfsplus project is used to compress it. There are several bugs in this tool and its maintainer has seemingly abandoned the project. It has been forked and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). The 'dmg' tool has the ability to create DMGs from scratch as well, but this functionality is broken. Only the compression feature is currently used. Ideally, the creation could be fixed and genisoimage would no longer be necessary. Background images and other features can be added to DMG files by inserting a .DS_Store before creation. This is generated by the script contrib/macdeploy/custom_dsstore.py. As of OS X Mavericks (10.9), using an Apple-blessed key to sign binaries is a requirement in order to satisfy the new Gatekeeper requirements. Because this private key cannot be shared, we'll have to be a bit creative in order for the build process to remain somewhat deterministic. Here's how it works: - Builders use Gitian to create an unsigned release. This outputs an unsigned dmg which users may choose to bless and run. It also outputs an unsigned app structure in the form of a tarball, which also contains all of the tools that have been previously (deterministically) built in order to create a final dmg. - The Apple keyholder uses this unsigned app to create a detached signature, using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). - Builders feed the unsigned app + detached signature back into Gitian. It uses the pre-built tools to recombine the pieces into a deterministic dmg. diff --git a/doc/REST-interface.md b/doc/REST-interface.md index 7fbb174030..0cf5481a80 100644 --- a/doc/REST-interface.md +++ b/doc/REST-interface.md @@ -1,98 +1,98 @@ Unauthenticated REST Interface ============================== The REST API can be enabled with the `-rest` option. The interface runs on the same port as the JSON-RPC interface, by default port 8332 for mainnet and port 18332 for testnet. Supported API ------------- ####Transactions `GET /rest/tx/.` Given a transaction hash: returns a transaction in binary, hex-encoded binary, or JSON formats. For full TX query capability, one must enable the transaction index via "txindex=1" command line / configuration option. ####Blocks `GET /rest/block/.` `GET /rest/block/notxdetails/.` Given a block hash: returns a block, in binary, hex-encoded binary or JSON formats. The HTTP request and response are both handled entirely in-memory, thus making maximum memory usage at least 2.66MB (1 MB max block, plus hex encoding) per request. With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response. ####Blockheaders `GET /rest/headers//.` Given a block hash: returns amount of blockheaders in upward direction. ####Chaininfos `GET /rest/chaininfo.json` Returns various state info regarding block chain processing. Only supports JSON as output format. * chain : (string) current network name as defined in BIP70 (main, test, regtest) * blocks : (numeric) the current number of blocks processed in the server * headers : (numeric) the current number of headers we have validated * bestblockhash : (string) the hash of the currently best block * difficulty : (numeric) the current difficulty * verificationprogress : (numeric) estimate of verification progress [0..1] * chainwork : (string) total amount of work in active chain, in hexadecimal * pruned : (boolean) if the blocks are subject to pruning -* pruneheight : (numeric) heighest block available +* pruneheight : (numeric) highest block available * softforks : (array) status of softforks in progress ####Query UTXO set `GET /rest/getutxos//-/-/.../-.` The getutxo command allows querying of the UTXO set given a set of outpoints. See BIP64 for input and output serialisation: https://github.com/bitcoin/bips/blob/master/bip-0064.mediawiki Example: ``` $ curl localhost:18332/rest/getutxos/checkmempool/b2cdfd7b89def827ff8af7cd9bff7627ff72e5e8b0f71210f92ea7a4000c5d75-0.json 2>/dev/null | json_pp { "chaintipHash" : "00000000fb01a7f3745a717f8caebee056c484e6e0bfe4a9591c235bb70506fb", "chainHeight" : 325347, "utxos" : [ { "scriptPubKey" : { "addresses" : [ "mi7as51dvLJsizWnTMurtRmrP8hG2m1XvD" ], "type" : "pubkeyhash", "hex" : "76a9141c7cebb529b86a04c683dfa87be49de35bcf589e88ac", "reqSigs" : 1, "asm" : "OP_DUP OP_HASH160 1c7cebb529b86a04c683dfa87be49de35bcf589e OP_EQUALVERIFY OP_CHECKSIG" }, "value" : 8.8687, "height" : 2147483647, "txvers" : 1 } ], "bitmap" : "1" } ``` ####Memory pool `GET /rest/mempool/info.json` Returns various information about the TX mempool. Only supports JSON as output format. * size : (numeric) the number of transactions in the TX mempool * bytes : (numeric) size of the TX mempool in bytes * usage : (numeric) total TX mempool memory usage `GET /rest/mempool/contents.json` Returns transactions in the TX mempool. Only supports JSON as output format. Risks ------------- Running a web browser on the same node with a REST enabled bitcoind can be a risk. Accessing prepared XSS websites could read out tx/block data of your node by placing links like `