This is a public version of the main Kite repo
The main Kite repo (originally
kiteco/kiteco) was intended for private use. It has been lightly adapted for publication here by replacing private information with
XXXXXXX. As a result many components here may not work out of the box.
Getting started with the codebase
Our codebase is primarily located at github.com/kiteco/kiteco (http://github.com/kiteco/kiteco). There are a few auxiliary repositories that host very experimental code, but the goal is to make the “kiteco” repository the point of truth for all of our services.
- Our codebase is primarily Go. (
- Infrastructure uses Terraform (for AWS) provisioning, and Fabric/shell scripts for deployment and management of remote hosts (
- You need VPN credentials to access any of our remote AWS (or Azure) hosts.
- Platform-specific logic & instructions live in subdirectories
linux. You probably don't need these.
We use Git LFS to store our various
bindata.go files. You will need to install the command line tool to get the contents of those files when you pull the repository. Installation instructions are on their website, but for MacOS you can install it by running (from inside the
brew update brew install git-lfs git lfs install
Then do a
git pull to get the bindata.go files. If they do not download from LFS, try running
git lfs pull (you should only need to do this once - subsequent
git pulls should update the bindata correctly).
Optional: Improving Performance
git lfs install installs a smudge filter that automatically downloads and replaces the contents of newly checked out "pointer files" with their content. By default smudge filters operate on checked out blobs in sequence, so cannot download in batch as would typically happen when running
git lfs pull. Furthermore, by default, git checkouts will block on downloading the new LFS files which can be annoying. You might prefer to disable the smudge filter (this can be run even if you've already run the regular
git lfs install):
git lfs install --skip-smudge git lfs pull
Then, when building after a new checkout, you may see an error of the form "expected package got ident." This occurs because
go reads some Go files and sees the Git LFS pointers instead of the actual data file. At this point, you can download the latest files with
git lfs pull and rebuilding should work.
Nothing needs to be done when pushing LFS blobs. That will still happen automatically.
The bulk of our code is currently in Go. This can be found at github.com/kiteco/kiteco/kite-go (http://github.com/kiteco/kiteco/kite-go). To get started working in this part of the codebase, first make sure you have your Go environment setup correctly (i.e Go is installed, $GOPATH is set, etc.).
Locally, however, you will need to install Go 1.15.3. The following steps will get you going.
$GOPATH in your .profile / .bashrc/ .bash_profile / .zshrc, e.g:
export GOROOT=/usr/local/go export GOPATH=$HOME/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
Make sure to create these directories as well:
mkdir $HOME/go mkdir $HOME/go/src $HOME/go/bin $HOME/go/pkg
If you are on a Mac and set the above in either .bashrc or .zshrc, make sure to load it in either your .profile or .bash_profile. See this for an explanation.
It would be useful to become familiar with how
go code is organized. Check out https://golang.org/doc/code.html for more on this topic.
Navigate to where the
kiteco repo will live in your
GOPATH, and clone the repo.
# Create kiteco directory within GOPATH, and clone the repo there mkdir -p ~/go/src/github.com/kiteco cd ~/go/src/github.com/kiteco git clone [email protected]:kiteco/kiteco
To install the latest version of Go that's compatible with our codebase, run:
cd ~/go/src/github.com/kiteco/kiteco cd devops/scripts ./install-golang.sh
From here, just run
make install-deps from the root of the
kiteco repo to get basic utilities installed.
# Install dependencies make install-deps
./scripts/update-golang-version.sh if you'd like to make Kite require a newer version of Golang.
For development builds (see below), you may need to have Tensorflow installed globally on your system.
You're now ready to build Kite! First, build the sidebar for your platform
./osx/build_electron.sh force # ./linux/build_electron.sh force # ./windows/build_electron.sh force
This process is asynchronous to the Kite daemon build, so you must manually rebuild the sidebar as needed.
Now build and run Kite:
Note that this is not a full Kite build, but is the recommended approach for development, as it is much faster. Some functionality is disabled in the development build (depending on the platform):
- Kite system tray icon
- Updater service
You should be able to develop, build, and test Kite entirely on your local machine. However, we do have cloud instances & VMs available for running larger jobs and for testing our cloud services
Dependency Management with Go Modules
We use the Go Modules system for dependency management.
- make sure in
~/go/src/github.com/kiteco/kitecoand not a symlink
- make sure deps are updated to the versions in
go mod download
$GOPRIVATEin your .profile / .bashrc/ .bash_profile / .zshrc, e.g:
To add or update a dependency, all you need to do is
go get it, which will automatically update the
go.sum files. To remove a dependency, remove references to it in the code and run
go mod tidy. In general, make sure to run
go mod tidy to make sure all new dependencies have been added and unused ones have been removed before committing any dependency changes.
The process for updating a dependency is:
go get -u github.com/foo/bar
- (optional) run any
gocommand, such as
go mod tidy
git add go.mod go.sum
git commit ...
The process for adding a dependency is:
go get github.com/foo/bar
- edit code to import "github.com/foo/bar"
go mod tidy
git add go.mod go.sum
git commit ...
godep may attempt to clone private repositories via HTTPS, requiring manual authentication. Instead, you can add the following section to your
~/.gitconfig in order to force SSH authentication:
[url "[email protected]:"] insteadOf = https://github.com/
We bundle a lot of pre-computed datasets & machine learning models into the Kite app through the use of a custom filemap & encoding on top of go-bindata. The data, located in
kite-go/client/datadeps, is kept in Git-LFS.
All needed data files is first stored on S3. There are pointers at various places in our codebase to S3 URIs. After updating references to these datasets, the datadeps file must be manually rebuilt:
This will bundle all data that is loaded at Kite initialization time. You must ensure the needed data is loaded at initialization, otherwise it will not be included!
Some logs are displayed in Xcode, but most are written to a log file:
tail -F ~/.kite/logs/client.log
Testing and Continuous Integration
Your Go code should pass several quality criteria before being allowed into the master branch. Travis CI (https://travis-ci.org/) acts as the gatekeeper between pull requests and merging. You can test your code before pushing to a pull request to speed up the process by navigating to the
kite-go directory and running
make * commands directly (any of
You will need access to our VPN to connect to our backend hosts.
- Get VPN credentials (*.ovpn file) from @tarak (You will need to type in a password IRL - don't IM/chat it)
- Install Tunnelblick for OS X (https://code.google.com/p/tunnelblick/)
- Double click on the “.ovpn” file that contains your credentials.
- Tunnelblick should automatically apply the configuration.. look for the icon on the OS X status bar
- Click on the Tunnelblick icon, select your config, and enter your VPN password. (NOTE: Tunnelblick will complain saying the IP hasn't changed. Check the box to disable the message and continue.)
- Ping 'test-0.kite.com' and make sure it resolves. It's okay if the pings timeout; ICMP is disabled by default on aws instances.
Kite's Dropbox has ssh credentials for all the machines on AWS and Azure under Shared > Engineering > keys > kite-dev.pem and Shared > Engineering > keys > kite-dev-azure. Place both of these in your .ssh directory, i.e. ~/.ssh/kite-dev.pem. As a convenience, you should add the following to your
Host *.kite.com ForwardAgent yes IdentityFile ~/.ssh/kite-dev.pem User ubuntu # Test instances are on Azure Host test-*.kite.com User ubuntu IdentityFile ~/.ssh/kite-dev-azure
Don't forget to set appropriate permissions on the credential files (e.g. 700)